README-design

OpenTelemetry filter -- design patterns
==========================================================================

This document describes the cross-cutting design patterns used throughout
the OTel filter implementation.  It complements README-implementation
(component-by-component architecture) with a pattern-oriented view of the
mechanisms that span multiple source files.


1  X-Macro Code Generation
----------------------------------------------------------------------

The filter uses X-macro lists extensively to generate enums, keyword tables,
event metadata and configuration init/free functions from a single definition.
Each X-macro list is a preprocessor define containing repeated invocations of
a helper macro whose name is supplied by the expansion context.

1.1  Event Definitions

FLT_OTEL_EVENT_DEFINES (event.h) drives the event system.  Each entry has the
form:

  FLT_OTEL_EVENT_DEF(NAME, CHANNEL, SMP_FE, SMP_BE, INJECT, EXTRACT,
                     HTTP_ONLY, CONTEXT, "string")

The same list is expanded twice:

  - In enum FLT_OTEL_EVENT_enum (event.h) to produce symbolic constants built
    by pasting the channel and the name (e.g. FLT_OTEL_EVENT_REQ_INSPECT_FE,
    or FLT_OTEL_EVENT__STREAM_START with a double underscore for a lifecycle
    event whose channel argument is empty), followed by the FLT_OTEL_EVENT_MAX
    sentinel.

  - In the flt_otel_event_data[] initializer (event.c) to produce a const table
    of struct flt_otel_event_data entries carrying the analyzer bit and name,
    the sample fetch direction, the SMP_VAL_FE_* / SMP_VAL_BE_* fetch-validity
    bits, the HTTP context-injection and context-extraction flags, the HTTP-only
    flag, the readable-context flag and the event name string.

Adding a new event requires a single line in the X-macro list.

1.2  Parser Keyword Definitions

The configuration parser is driven by these X-macro lists:

  FLT_OTEL_PARSE_INSTR_DEFINES   (parser.h)
  FLT_OTEL_PARSE_GROUP_DEFINES   (parser.h)
  FLT_OTEL_PARSE_SCOPE_DEFINES   (parser.h)

Each entry has the form:

  FLT_OTEL_PARSE_DEF(ENUM, flag, check, min, max, "name", "usage")

Each list is expanded to:

  - An enum for keyword indices (FLT_OTEL_PARSE_INSTR_ID, etc.).
  - A static parse_data[] table of struct flt_otel_parse_data entries carrying
    the keyword index, flag_check_id, check_name type, argument count bounds,
    keyword name string and usage string.

The section parsers (flt_otel_parse_cfg_instr, _group, _scope) look up args[0]
in their parse_data table via flt_otel_parse_cfg_check(), which validates
argument counts and character constraints before dispatching to
the keyword-specific handler.

Additional X-macro lists generate:

  FLT_OTEL_PARSE_SCOPE_STATUS_DEFINES      Span status codes.
  FLT_OTEL_PARSE_SCOPE_INSTRUMENT_DEFINES  Instrument type keywords.
  FLT_OTEL_HTTP_METH_DEFINES               HTTP method name table.
  FLT_OTEL_GROUP_DEFINES                   Group action metadata.

1.3  Configuration Init/Free Generation

Macros in conf_funcs.h generate init and free functions for all configuration
structure types:

  FLT_OTEL_CONF_FUNC_INIT(_type_, _id_, _max_, _func_)
  FLT_OTEL_CONF_FUNC_FREE(_type_, _id_, _func_)

The _type_ parameter names the structure (e.g. "span"), _id_ names the
identifier field within the FLT_OTEL_CONF_HDR (e.g. "id", "key", "str"), _max_
is the length the identifier must stay below, and _func_ is a brace-enclosed
code block executed after the boilerplate allocation (for init) or before the
boilerplate teardown (for free).

The generated init function:
  1. Validates the identifier is non-NULL and non-empty.
  2. Handles auto-generated IDs (FLT_OTEL_CONF_HDR_SPECIAL prefix), so that the
     generated name is what the next two steps measure and compare.
  3. Checks the length against _max_, FLT_OTEL_ID_MAXLEN (128) for a name and
     FLT_OTEL_LEN_UNLIMITED for the text of a sample expression.
  4. Detects duplicates in the target list.
  5. Allocates with OTELC_CALLOC, duplicates the ID with OTELC_STRDUP.
  6. Appends to the parent list via LIST_APPEND.
  7. Executes the custom _func_ body.

The generated free function:
  1. Executes the custom _func_ body (typically list destruction).
  2. Frees the identifier string.
  3. Removes the node from its list.
  4. Frees the structure with OTELC_SFREE_CLEAR.

conf.c instantiates these macros for every configuration type: hdr, str, ph,
sample_expr, sample, link, context, span, exception, scope, instrument,
log_record, set_var_ctx, unset_var, stop, group, instr.  The more complex types
(span, scope, instr) carry non-trivial _func_ bodies that initialize sub-lists
or set default values.


2  Type-Safe Generic Macros
----------------------------------------------------------------------

define.h provides utility macros -- several built on typeof() and statement
expressions -- that centralize common expression patterns without repetitive
boilerplate.

2.1  Safe Pointer Dereferencing

  FLT_OTEL_DEREF(p, m, v)
    Dereferences p->m and returns 'v' if 'p' is NULL -- a plain conditional
    expression, usable wherever an expression is expected.

The macro eliminates repetitive NULL checks at member access sites.

2.2  String Comparison

  FLT_OTEL_STR_CMP(S, s)
    Compares a runtime string 's' against a compile-time string literal 'S':
    the cached s_len field is compared with the literal's compile-time size
    first and memcmp() only runs on a match, so no strlen() is ever repeated
    on known constants.

  FLT_OTEL_CONF_STR_CMP(s, S)  (defined in conf.h)
    Compares two configuration strings using their cached length fields (from
    FLT_OTEL_CONF_STR / FLT_OTEL_CONF_HDR).  Falls through to memcmp() only when
    lengths match, providing an early-out fast path.

2.3  List Operations

  FLT_OTEL_LIST_ISVALID(a)
    Checks whether a list head has been initialized (both prev and next are
    non-NULL).

  FLT_OTEL_LIST_DEL(a)
    Safely deletes a list element only if the list head is valid.

  FLT_OTEL_LIST_DESTROY(t, h)
    Destroys all elements in a typed configuration list by iterating with
    list_for_each_entry_safe and calling flt_otel_conf_<t>_free() for each
    entry.  The type 't' is used to derive both the structure type and the
    free function name.

2.4  Thread-Local Rotating Buffers

  FLT_OTEL_BUFFER_THR(b, m, n, p)
    Declares a thread-local pool of 'm' string buffers, each of size 'n' bytes.
    The pool index rotates on each invocation, avoiding the need for explicit
    allocation in debug formatting functions.  Each call returns a pointer to
    the next buffer via the output parameter 'p'.

    Used by the debug list formatter flt_otel_list_dump(), whose returned string
    must survive until the caller finishes with it.


3  Error Handling Architecture
----------------------------------------------------------------------

3.1  Error Accumulation Macros

Macros in define.h manage error messages:

  FLT_OTEL_ERR(f, ...)
    Formats an error message via memprintf() into the caller's char **err
    pointer, but only if *err is still NULL.  Prevents overwriting the first
    error with a cascading secondary error.

  FLT_OTEL_ERR_APPEND(f, ...)
    Formats the error message unconditionally, regardless of prior content.
    Used to wrap an existing message with additional context by passing the
    old *err among the format arguments.

  FLT_OTEL_ERR_FREE(p)
    Emits the accumulated error message as a debug-build trace (OTELC_DBG at
    ERROR level), then frees the string and NULLs the pointer; a release build
    only discards it.

All source functions that can fail accept a char **err parameter.  The
convention is: set *err on error, leave it alone on success.  Callers check
both the return value and *err to detect problems.

An error message string has two possible origins: HAProxy's memprintf(), or
the wrapper's own allocator, used by otelc_init() and the tracer, meter and
logger create calls for their diagnostics.  FLT_OTEL_ERR_FREE() releases the
string through OTELC_SFREE_CLEAR() so that one macro covers both origins: in
a debug build the wrapper's free first probes for its metadata header below
the payload and transparently plain-frees a foreign (memprintf-allocated) block.
The reverse pairing is not safe: a wrapper-allocated string must never reach
free(), ha_free() or an occupied memprintf() argument, whose internal free()
would abort a debug build.  Where such a string crosses over to HAProxy -- the
otelc_cfg_validate() diagnostic at parse time -- it is copied via memprintf()
first and the original is freed with OTELC_SFREE().

3.2  Parse-Time Error Reporting

Configuration parsing (parser.c) uses additional macros:

  FLT_OTEL_PARSE_ERR(e, f, ...)
    Sets the error string and ORs ERR_ABORT | ERR_ALERT into the return value.
    Like FLT_OTEL_ERR, it fills *e only while it is still NULL, so the first
    error wins.  Used inside the section parsers' keyword dispatch chains.

  FLT_OTEL_PARSE_ALERT(f, ...)
    Issues a ha_alert() with the current file and line, then sets the error
    flags.  Used when the error message should appear immediately.

  FLT_OTEL_POST_PARSE_ALERT(f, ...)
    Same as PARSE_ALERT but uses cfg_file instead of file, for post-parse
    validation that runs after the parser has moved on.

  FLT_OTEL_PARSE_IFERR_ALERT()
    Checks whether *err was set by a called function, and if so, issues an
    alert and clears the error.  This bridges between functions that use the
    char **err convention and the parser's own alert mechanism.

3.3  Runtime Error Modes

Helper functions in filter.c implement the runtime error policy:

  flt_otel_return_int(f, err, retval)
  flt_otel_return_void(f, err)

If an error is detected (retval == FLT_OTEL_RET_ERROR or *err != NULL):

  Hard-error mode (rt_ctx->flag_harderr):
    Sets rt_ctx->flag_disabled = 1, disabling the filter for the remainder of
    the stream.  Atomically increments the n_harderr tally (and the disabled
    counter in a counters build), and logs the error message with a " (filter
    disabled)" suffix at level err.

  Soft-error mode (default):
    Increments the n_softerr tally and logs the error message at level warning;
    processing continues.  The tracing data for the current event may then be
    incomplete but the stream is not affected.

Both emissions go through FLT_OTEL_LOG_LIM (debug.h): one per-mode latch bit
makes each error episode log only once -- a clean callback return re-arms it
-- and a sliding frequency counter caps the output at FLT_OTEL_LOG_RATE_MAX
lines per FLT_OTEL_LOG_RATE_PERIOD per instance (config.h), with the held-back
lines tallied in log.sup_pending and reported as a suffix on the next emitted
line, and counted in log.sup_total, a lifetime total that is never reset.

A message may quote a sample value, so whatever bytes the peer sent would reach
the log as sent.  Both helpers therefore escape the text before logging it,
through flt_otel_str_escape() (util.c): the non-printable bytes are rewritten
in HAProxy's '#XX' form and the length is bounded by FLT_OTEL_LOG_MSG_SIZE.

In both modes, FLT_OTEL_RET_OK is always returned to HAProxy.  A tracing failure
never interrupts stream processing.

3.4  Disabled State Checking

flt_otel_is_disabled() (filter.c) is called at the top of every filter callback
that processes events.  It reports whether the stream's runtime context has its
flag_disabled field set, which a hard error (in hard-error mode) or a fired
'otel-stop' directive does.  The runtime context is per-stream and therefore
single-threaded, so the field is read directly, without atomic access; when it
is set, the callback returns immediately without processing.

The global "flt-otel disable" CLI command uses a separate instrumentation field,
conf->instr->flag_disabled, which flt_otel_ops_attach() checks atomically when
the filter attaches to a stream (section 6.3), not through this per-stream test.


4  Thread Safety Model
----------------------------------------------------------------------

The filter runs within HAProxy's multi-threaded worker model.  Each stream is
bound to a single thread, so per-stream state (the runtime context, spans and
contexts) is accessed without locking.  Cross-stream shared state uses atomic
operations.

4.1  Atomic Fields

The following fields are accessed atomically across threads:

  conf->instr->flag_disabled  Toggled by CLI "flt-otel enable/disable".
                              Checked in flt_otel_ops_attach().

  conf->instr->flag_harderr   Toggled by CLI "flt-otel hard-errors/soft-errors".
                              Copied to rt_ctx at attach time.

  conf->instr->flag_noflush   Toggled by CLI "flt-otel noflush".
                              Read in flt_otel_ops_deinit().

  conf->instr->rate_limit     Set by CLI "flt-otel rate".
                              Read in flt_otel_ops_attach().

  conf->instr->log.type       Set by CLI "flt-otel logging".
                              Read at each FLT_OTEL_LOG emission.

  conf->instr->log.latch      Edge-trigger bits of the rate-limited runtime
  conf->instr->log.sup_pending log and its held-back line tally (section 3.3).

  conf->instr->log.sup_total  Lifetime total of held-back log lines; never
                              cleared (section 3.3).

  conf->instr->n_harderr      Runtime-error tallies shown by "flt-otel status"
  conf->instr->n_softerr      and cleared by "flt-otel reset-errors".

  flt_otel_drop_cnt           Incremented in flt_otel_log_handler_cb().
                              Read by CLI "flt-otel status".

  conf->cnt.disabled[]        Incremented in flt_otel_session_disable() and
                              flt_otel_return_int().
  conf->cnt.attached[]        Incremented in flt_otel_ops_attach().

All use _HA_ATOMIC_LOAD(), _HA_ATOMIC_STORE(), _HA_ATOMIC_INC() or
_HA_ATOMIC_ADD() from HAProxy's atomic primitives.

4.2  Metric Instrument Creation (CAS Pattern)

Metric instruments are shared across all threads but created lazily on first
use.  The creation uses a compare-and-swap pattern to ensure exactly one thread
performs the creation:

  int64_t expected = OTELC_METRIC_INSTRUMENT_UNSET;  /* -1 */
  if (HA_ATOMIC_CAS(&instr->idx, &expected, OTELC_METRIC_INSTRUMENT_PENDING)) {
      /* This thread won the race -- create the instrument. */
      idx = meter->create_instrument(...);
      HA_ATOMIC_STORE(&instr->idx, idx);
  }

The states are:
  UNSET   (-1)  Not yet created.  The CAS target.
  PENDING (-2)  Creation in progress by another thread.
  FAILED  (-3)  Creation failed too often; the instrument is given up.
  >= 0          Valid meter index.  Ready for recording.

Threads that lose the CAS skip creation.  A failure stores UNSET again, so a
transient fault is retried by a later scope execution, though not more than
FLT_OTEL_INSTR_FAIL_MAX times; the idx then becomes FAILED and the creation is
never tried again, as every attempt takes a process-wide lock.  Update-form
instruments check the referenced create-form's idx before recording: an idx
that is still PENDING is waited for until the creating thread resolves it, then
recorded; a negative idx (never created, failed, or given up) is skipped.

4.3  Per-Thread Initialization Guard

flt_otel_ops_init_per_thread() starts the tracer, meter and logger background
threads on the first call only.  An atomic claim on conf->instr->flag_started
(HA_ATOMIC_BTS) guarantees this: the thread that wins the claim runs the start
sequence, while the others -- including later calls from other proxies sharing
the same instrumentation -- return success without repeating it.

The separate FLT_CFG_FL_HTX flag is unrelated to this guard.  It is set once in
flt_otel_ops_init(), and only for an HTTP-mode proxy, to declare HTX filtering;
HAProxy then binds the filter to the proxy's HTX streams.  A TCP-mode proxy
leaves the flag unset, so the filter runs on the raw stream, which is what lets
the OTel filter trace TCP-mode proxies.

4.4  CLI Update Propagation

CLI set commands (cli.c) iterate all OTel filter instances across all proxies
using the FLT_OTEL_PROXIES_LIST_START / FLT_OTEL_PROXIES_LIST_END macros
(util.h) and atomically update the target field on each instance.  This ensures
that "flt-otel disable" takes effect globally by default, while the optional
"@<filter>" target token restricts a command to the one matching instance.


5  Sample Evaluation Pipeline
----------------------------------------------------------------------

Sample expressions bridge HAProxy's runtime data (headers, variables, connection
properties) into OTel attributes, events, baggage, status, metric values and log
record bodies.

5.1  Two Evaluation Paths

The parser detects which path to use based on whether the sample value argument
begins with the "%[" sequence.  Bare sample expressions never start with '%',
so this is a positional check.  An explicit '%[ ... ]' wrapper around the whole
argument marks the inner string as a log-format string that may contain HAProxy
aliases such as %ci, %t or %ft; the outer '%[' and trailing ']' are stripped
before parse_logformat_string() is called:

  Log-format path (sample->lf_used == true):
    The value is parsed via parse_logformat_string() and stored in
    sample->lf_expr.  At runtime, build_logline() evaluates the expression into
    a temporary buffer of global.tune.bufsize bytes.  The result is always a
    string.

  Bare sample path (sample->lf_used == false):
    Each whitespace-separated token is parsed via sample_parse_expr() and
    stored as an flt_otel_conf_sample_expr in sample->exprs.  At runtime, each
    expression is evaluated via sample_process().  If there is exactly one
    expression, the native HAProxy sample type is preserved (bool, int, IP
    address, string).  If there are multiple expressions, their string
    representations are concatenated.

5.2  Type Conversion Chain

flt_otel_sample_to_value() (util.c) converts HAProxy sample types to OTel value
types:

  SMP_T_BOOL  ->  OTELC_VALUE_BOOL
  SMP_T_SINT  ->  OTELC_VALUE_INT64
  Other       ->  OTELC_VALUE_DATA (string, via flt_otel_sample_to_str)

flt_otel_sample_to_str() handles the string conversion for non-trivial types:

  SMP_T_BOOL     "0" or "1"
  SMP_T_SINT     snprintf decimal
  SMP_T_IPV4     inet_ntop
  SMP_T_IPV6     inet_ntop
  SMP_T_STR      direct copy
  SMP_T_METH     lookup from static HTTP method table, or raw string for
                 HTTP_METH_OTHER

For metric instruments the value is always int64.  A string value -- for example
a value read from a variable, which the filter stores as a string -- is coerced
with otelc_value_strtonum(); only a value that is not a valid integer is
rejected.

5.3  Dispatch to Data Structures

flt_otel_sample_add() (util.c) is the top-level evaluator.  After converting the
sample to an otelc_value, it dispatches based on type:

  FLT_OTEL_EVENT_SAMPLE_ATTRIBUTE
    -> flt_otel_sample_add_kv(&data->attributes, key, &value)

  FLT_OTEL_EVENT_SAMPLE_BAGGAGE
    -> flt_otel_sample_add_kv(&data->baggage, key, &value)

  FLT_OTEL_EVENT_SAMPLE_EVENT
    -> flt_otel_sample_add_event(&data->events, sample, &value)
       Groups attributes by event name; creates a new flt_otel_scope_data_event
       node if the event name is not yet present.  A sample's 'time' clause
       fills the node's ts field and sets ts_set, overriding the span timestamp
       at emission.

  FLT_OTEL_EVENT_SAMPLE_STATUS
    -> flt_otel_sample_set_status(&data->status, sample, &value, err)
       Sets the status code from the int32 carried in the sample's extra data
       and the description from the evaluated value.  Status lines are tried
       in configuration order and only the first whose condition holds runs;
       a guard against setting the status twice remains as a safeguard.

5.4  Growable Key-Value Arrays

Attribute and baggage arrays use a lazy-allocation / grow-on-demand pattern via
flt_otel_sample_add_kv() (util.c):

  - Initial allocation: FLT_OTEL_ATTR_INIT_SIZE (8) elements via otelc_kv_new().
  - Growth: FLT_OTEL_ATTR_INC_SIZE (4) additional elements via OTELC_REALLOC()
    when the array is full.
  - The cnt field tracks used elements; size tracks total capacity.

Event attribute arrays follow the same pattern within each
flt_otel_scope_data_event node.


6  Rate Limiting and Filtering
----------------------------------------------------------------------

6.1  Rate Limit Representation

The rate limit is configured as a floating-point percentage (0.0 to 100.0) but
stored internally as a uint32_t for lock-free comparison:

  FLT_OTEL_FLOAT_U32(a)  Converts a percentage to a uint32 value:
                          (uint32_t)((a) / 100.0 * UINT32_MAX + 0.5)

  FLT_OTEL_U32_FLOAT(a)  Converts back for display:
                          ((double)(a) / UINT32_MAX) * 100.0

At attach time, the filter compares a fresh ha_random32() value against the
stored rate_limit.  If the random value exceeds the threshold, the filter
returns FLT_OTEL_RET_IGNORE and does not attach to the stream.  This provides
uniform sampling without floating-point arithmetic on the hot path.

6.2  ACL-Based Filtering

Each scope may carry an ACL condition (if/unless) evaluated at scope
execution time via acl_exec_cond().  flt_otel_parse_acl() (parser.c) resolves
an ACL reference by trying the lists its callers supply, in order, until
build_acl_cond() succeeds.  The callers pass the lists in this priority order:

  1. Scope-local ACLs (declared within the otel-scope section).
  2. Instrumentation ACLs (declared in the otel-instrumentation section).
  3. Proxy ACLs (declared in the HAProxy frontend/backend section).

If the condition fails at runtime, the scope is simply skipped.

6.3  Global Disable

The filter can be disabled globally via the CLI ("flt-otel disable") or the
configuration ("option disabled").  The flag_disabled field on the
instrumentation configuration is checked atomically in flt_otel_ops_attach()
before any per-stream allocation occurs.

6.4  Per-Connection Stop

The 'otel-stop' scope keyword stops tracing for a single connection.  When the
keyword fires, flt_otel_scope_run() marks every open span and context with the
"*" wildcard, finishes them via flt_otel_scope_finish_marked(), and then sets
rt_ctx->flag_disabled = 1.  A guard at the top of flt_otel_scope_run() skips
any further scopes for that stream, so that sibling scopes on the same event or
in the same otel-group no longer run.

The directive is evaluated after the scope's own spans, log records and metric
instruments, so a stopping scope still emits its telemetry before tracing ends.
The keyword may be written on several lines, one per condition, held in the
scope's stops list of struct flt_otel_conf_stop and kept apart from the scope's
own condition; the lines are tried in configuration order and the stop takes
effect as soon as one of them fires.  A bare line always fires.  This is the
per-stream counterpart to the global "flt-otel disable" of section 6.3.


7  Memory Management Strategy
----------------------------------------------------------------------

7.1  Pool-Based Allocation

HAProxy memory pools are registered for frequently allocated structures
(pool.c):

  pool_head_otel_scope_span       Per-stream span entries.
  pool_head_otel_scope_context    Per-stream context entries.
  pool_head_otel_runtime_context  Per-stream runtime context.
  pool_head_otel_span_context     OTel SDK objects (via C wrapper allocator
                                  callback).

Pool registration is conditional on compile-time flags (USE_POOL_OTEL_* in
config.h).  flt_otel_pool_alloc() allocates from the pool when one is present,
and from the heap (OTELC_MALLOC) when the pool is NULL.

The C wrapper library's memory allocations are redirected through
flt_otel_mem_malloc() / flt_otel_mem_free() (filter.c), which use the
otel_span_context pool via otelc_ext_init().  This keeps OTel SDK objects in
HAProxy's pool allocator for cache-friendly allocation.

7.2  Trash Buffers

flt_otel_trash_alloc() (pool.c) provides temporary scratch buffers of
global.tune.bufsize bytes.  When USE_TRASH_CHUNK is defined, it uses HAProxy's
alloc_trash_chunk(); otherwise it manually allocates a buffer structure and data
area.  Trash buffers serve a single user, HTTP header name construction in
flt_otel_http_header_set().  Metric recording, log-record emission and sample
evaluation do not use them: those paths allocate their scratch buffers from
the heap instead (OTELC_CALLOC / OTELC_MALLOC of global.tune.bufsize bytes).

7.3  Scope Data Lifecycle

flt_otel_scope_data (scope.h) is initialized and freed within a single span
processing block in flt_otel_scope_run().  It is stack-conceptual data:
allocated at the start of each span's processing, populated during sample
evaluation, consumed by flt_otel_scope_run_span(), and freed immediately after.
The key-value arrays within it are heap-allocated via otelc_kv_new() and freed
via otelc_kv_destroy().


8  Context Propagation Mechanism
----------------------------------------------------------------------

8.1  Carrier Abstraction

The OTel C wrapper uses a carrier pattern for context propagation.  Two carrier
types exist, each with writer and reader variants:

  otelc_text_map_writer / otelc_text_map_reader
  otelc_http_headers_writer / otelc_http_headers_reader

otelc.c provides callback functions that the C wrapper invokes during
inject/extract operations:

  Injection (writer callbacks):
    flt_otel_text_map_writer_set_cb()
    flt_otel_http_headers_writer_set_cb()
    Both append key-value pairs to the carrier's text_map via
    OTELC_TEXT_MAP_ADD.

  Extraction (reader callbacks):
    flt_otel_text_map_reader_foreach_key_cb()
    flt_otel_http_headers_reader_foreach_key_cb()
    Both iterate the text_map's key-value pairs, invoking a handler function
    for each entry.  Iteration stops early if the handler returns -1.

8.2  HTTP Header Storage

flt_otel_http_headers_get() (http.c) reads headers from the HTX buffer with
prefix matching.  The prefix is stripped from header names in the returned
text_map.  A header whose name is only the prefix is skipped: the stripped key
would be empty, and the text map reads a zero-length key as a NUL-terminated
string.  A special prefix character ('-') matches all headers without prefix
filtering.

flt_otel_http_header_set() constructs a "prefix-name" header, removes all
existing occurrences via an http_find_header / http_remove_header loop, then
adds the new value via http_add_header.  A prefixed name that does not fit the
build buffer is rejected with a runtime error instead of being truncated.

8.3  Variable Storage

When USE_OTEL_VARS is enabled, span context can also be stored in HAProxy
transaction-scoped variables.  Variable names are normalized
(flt_otel_normalize_name in vars.c):

  Dashes    -> 'D'  (FLT_OTEL_VAR_CHAR_DASH)
  Spaces    -> 'S'  (FLT_OTEL_VAR_CHAR_SPACE)
  Uppercase -> lowercase

Full variable names are constructed as "scope.prefix.name" with dots as
component separators.

Two implementation paths exist based on the USE_OTEL_VARS_NAME compile flag:

  With USE_OTEL_VARS_NAME:
    Direct CEB tree traversal on the HAProxy variable store, matching on the
    prefix of the name that struct var carries in this HAProxy version.  This
    is the simpler path.

  Without USE_OTEL_VARS_NAME:
    A binary tracking buffer (stored as a HAProxy variable) records the names
    of all context variables.  flt_otel_ctx_loop() iterates length-prefixed
    entries in this buffer, calling a callback for each entry.  This enables
    enumeration although the stored variables carry no scannable names.

The choice is auto-detected at build time by checking whether struct var has
a 'name' member.

On extraction, both paths add an empty variable value as an empty string, as
the text map would read a zero-length buffer as a NUL-terminated string.


9  Debug Infrastructure
----------------------------------------------------------------------

9.1  Conditional Compilation

When OTEL_DEBUG=1 is set at build time, -DDEBUG_OTEL is added to the compiler
flags.  This enables:

  - Additional flt_ops callbacks: deinit_per_thread, http_payload and
    http_reset.  In non-debug builds these are NULL.

  - FLT_OTEL_USE_COUNTERS (config.h), which activates the attach and disable
    outcome counters in flt_otel_counters, ie the attached[] and disabled[]
    arrays.  The per-event dispatch counters (cnt.event[]) are switched directly
    by DEBUG_OTEL.

  - Debug-only functions throughout the codebase, marked with [D] in
    README-func: flt_otel_scope_data_dump, flt_otel_http_headers_dump,
    flt_otel_vars_dump, flt_otel_args_dump, flt_otel_filters_dump, etc.

  - The OTELC_DBG() macro for level-filtered debug output.

9.2  Debug Level Bitmask

The debug level is a uint stored in conf->instr and controllable at runtime via
"flt-otel debug <level>".  The default value is FLT_OTEL_DEBUG_LEVEL
(0b11101111111 in config.h).  Each bit enables a category of debug output.

9.3  Logging Integration

The FLT_OTEL_LOG macro (debug.h) integrates with HAProxy's send_log() system.
Its behavior depends on the logging flags:

  FLT_OTEL_LOGGING_OFF (0):
    No log messages are emitted.

  FLT_OTEL_LOGGING_ON (1 << 0):
    Log messages are sent to the log servers configured in the instrumentation
    block via parse_logger().

  FLT_OTEL_LOGGING_NOLOGNORM (1 << 1):
    Combined with ON, suppresses normal-level messages (only warnings and above
    are emitted).

These flags are set via the "option dontlog-normal" configuration keyword or the
"flt-otel logging" CLI command.

Runtime data-path errors do not call FLT_OTEL_LOG directly: they go through
FLT_OTEL_LOG_LIM, which adds the per-episode edge trigger and the sliding rate
limit of section 3.3, so a recurring fault cannot flood the log servers.

9.4  Counter System

When FLT_OTEL_USE_COUNTERS is defined, the flt_otel_counters structure (conf.h)
maintains:

  attached[4]   Counters for attach outcomes, incremented atomically in
                flt_otel_ops_attach().
  disabled[2]   Counters for the per-stream disables, incremented atomically:
                the first by flt_otel_session_disable() for an 'otel-stop' or
                a 'require-context' miss, the second by flt_otel_return_int()
                for a hard error.

The counters are reported by the "flt-otel status" CLI command and dumped at
deinit time in debug builds.


10  Idle Timeout Mechanism
----------------------------------------------------------------------

The idle timeout fires periodic events on idle streams, enabling heartbeat-style
span updates or metric recordings.

10.1  Configuration

Each otel-scope bound to the "on-idle-timeout" event must declare an
idle-timeout interval.  At check time (flt_otel_ops_check), the minimum idle
timeout across all scopes is stored in conf->instr->idle_timeout.

10.2  Initialization

In flt_otel_ops_attach() the runtime context's idle_timeout and idle_exp fields
are initialized from the precomputed minimum.  The attach callback runs for a
filter of a backend section as well, which the stream-start callback does not.
The first wake-up is then scheduled by flooring the stream task's expiry via
flt_otel_idle_expire_set().  The channel's analyse_exp is deliberately left
untouched: the analysers own that field and freely overwrite it (the tarpit
analyser assigns its timeout there and the HTTP analysers reset it on
completion), so an idle tick stored in it would be lost and the idle event would
starve.  Driving the wake-up via the stream task keeps the idle timer firing
alongside analyser-owned timeouts such as the tarpit.

10.3  Firing

flt_otel_ops_check_timeouts() checks tick_is_expired(rt_ctx->idle_exp).
When expired:
  - The on-idle-timeout event fires via flt_otel_event_run().
  - The timer is rescheduled for the next interval, unless the event itself
    disabled the filter (e.g. through 'otel-stop'), in which case it is
    disarmed.
  - STRM_EVT_MSG is set on stream->pending_events to ensure the stream is
    re-evaluated.

On every invocation, expired or not, the idle wake-up is re-asserted on the
stream task expiry: process_stream() recomputes the task expiry from scratch
after a foreign timer fires and the idle tick is stored nowhere else.  When
the filter has been disabled for the stream, the callback disarms the timer
instead, so a stale tick cannot keep waking the stream task.


11  Group Action Integration Pattern
----------------------------------------------------------------------

The "otel-group" action integrates OTel scopes with HAProxy's rule system
(http-request, http-response, http-after-response, tcp-request, tcp-response).

11.1  Registration

group.c registers action keywords via INITCALL1 macros for all action contexts.
Each registration points to flt_otel_group_parse() as the parse function.

11.2  Parse-Check-Execute Cycle

  Parse (flt_otel_group_parse):
    Stores the filter ID and group ID as string duplicates in rule->arg.act.p[].
    Sets the check, action and release callbacks.

  Check (flt_otel_group_check):
    Resolves the string IDs to configuration pointers by scanning the proxy's
    filter list.  Replaces the string pointers with resolved flt_conf,
    flt_otel_conf and flt_otel_conf_ph pointers.  Frees the original string
    duplicates.

  Execute (flt_otel_group_action):
    Finds the filter instance in the stream's filter list.  Validates rule->from
    against the flt_otel_group_data[] table to determine the sample fetch
    direction.  Iterates all scopes in the group and calls flt_otel_scope_run()
    for each.  Always returns ACT_RET_CONT -- a group action never interrupts
    rule processing.

11.3  Group Data Table

The flt_otel_group_data[] table (group.c) maps each HAProxy action context
(ACT_F_*) to:

  act_from     The action context enum value.
  smp_val      SMP_VAL_* fetch-validity bit for the action's processing
               point; flt_otel_group_check() matches it against each fetch
               and condition of the group's scopes.
  smp_opt_dir  The sample fetch direction (REQ or RES).

This table is generated from the FLT_OTEL_GROUP_DEFINES X-macro list (group.h).


12  CLI Runtime Control Architecture
----------------------------------------------------------------------

cli.c registers commands under the "flt-otel" prefix.  The command table maps
keyword sequences to handler functions with access level requirements.

12.1  Command Table

  "flt-otel status"       No access restriction.  Read-only status report.
  "flt-otel enable"       ACCESS_LVL_ADMIN.  Clears flag_disabled.
  "flt-otel disable"      ACCESS_LVL_ADMIN.  Sets flag_disabled.
  "flt-otel hard-errors"  ACCESS_LVL_ADMIN.  Sets flag_harderr.
  "flt-otel soft-errors"  ACCESS_LVL_ADMIN.  Clears flag_harderr.
  "flt-otel reset-errors" ACCESS_LVL_ADMIN.  Clears the runtime-error tallies
                          and the log rate-limiter state; the lifetime
                          suppressed total is retained.
  "flt-otel logging"      ACCESS_LVL_ADMIN for setting; any for reading.
  "flt-otel noflush"      ACCESS_LVL_ADMIN for setting; any for reading.
                          Sets or reads flag_noflush.
  "flt-otel rate"         ACCESS_LVL_ADMIN for setting; any for reading.
  "flt-otel debug"        ACCESS_LVL_ADMIN for setting; any for reading.
                          DEBUG_OTEL only.
  "flt-otel flush"        ACCESS_LVL_ADMIN.  Forces a telemetry export.
  "flt-otel instruments"  No access restriction.  Lists metric instruments.
  "flt-otel scopes"       No access restriction.  Lists scopes and groups.

The "flt-otel enable" and "flt-otel disable" commands share the same handler
(flt_otel_cli_parse_disabled) with the private argument distinguishing the
operation (1 for disable, 0 for enable).  The same pattern is used for
hard-errors / soft-errors.

12.2  Global Propagation

All set operations iterate every OTel filter instance across all proxies using
FLT_OTEL_PROXIES_LIST_START / FLT_OTEL_PROXIES_LIST_END macros and atomically
update the target field.  A single "flt-otel disable" command disables the
filter in every proxy that has it configured; only an "@<filter>" target token
limits it to the one matching instance.

12.3  Status Report

The "flt-otel status" command builds its report iteratively in an io_handler
that emits through the applet output buffer, rendering the tabular parts with
the column-oriented table builder (util.c).  The report contains:

  - Library versions (C++ SDK and C wrapper).
  - Debug level (DEBUG_OTEL only).
  - The per-signal export pipeline table -- the queued, dropped, exported and
    failed counts and the age of the last successful export -- filled in by
    otelc_pipeline_status_get().
  - Dropped diagnostic message count ('sdk diagnostics').
  - Per-proxy: config file, group/scope counts, instrumentation details, rate
    limit, error mode, disabled state, logging state, runtime-error tallies
    (hard, soft), suppressed-line counts (pending, total), the minimum idle
    timeout, analyzer bitmask, and counters (if FLT_OTEL_USE_COUNTERS).

12.4  Introspection and Flush

The "flt-otel flush" command iterates every configured filter instance with
the FLT_OTEL_PROXIES_LIST_START / FLT_OTEL_PROXIES_LIST_END macros and, rather
than changing state, forces output: it calls force_flush on each instance's
tracer, meter and logger handles, and skips any handle not yet initialized.
As force_flush blocks the calling thread, all of these calls share the single
FLT_OTEL_FLUSH_CLI_S budget that flt_otel_flush_budget() (util.c) measures;
a call that finds it spent is skipped, because the SDK reads a non-positive
timeout as no timeout at all.  An exporter that cannot be reached therefore
stalls the command for the budget, not for the sum of the per-call timeouts.
The "@<filter>" target token limits the flush to the one matching instance.

The "flt-otel instruments" command reports, for every filter instance, the
instruments declared in each scope.  A create-form instrument is shown with
its type, unit and lazy-creation state, an update-form instrument by name with
the 'update' type, its create-only columns left blank.

The "flt-otel scopes" command likewise builds a per-instance report, printing
each scope with its bound event and used flag, each group with its scope count
and, in a debug build, the per-event dispatch counts held in conf->cnt.event[].