Skip to the content.

MMT-DPI Threading Model

This document describes the threading contract for MMT-DPI: what state is global, what state is per-handler, and the rules a multi-threaded application must follow to use the library safely.

Summary of the contract

  1. Initialization before workers. init_extraction() and all protocol / plugin (un)registration must complete on a single thread before any worker thread starts processing packets.
  2. One handler per worker. Each worker thread owns its own mmt_handler_t (created with mmt_init_handler()). A handler must not be shared or used concurrently by more than one thread. Creating and destroying handlers (mmt_init_handler() / mmt_close_handler()) concurrently from multiple threads is supported — the global handler-bookkeeping map they mutate is mutex-guarded (see below), and the analysis/classification switches each mmt_init_handler() enables are the new handler’s own, accessed atomically (see the note under Locking), so even the very first handlers may be created concurrently from cold.
  3. No registration once workers are running. Registering or unregistering protocols/plugins while workers are processing packets is discouraged; if a defensive caller does so, the mutations are mutex-guarded (see below) but the per-handler snapshot taken at mmt_init_handler() time will not reflect the change.

Following these rules makes the per-packet hot path completely lock-free.

Global (process-wide) state

Three pieces of state are global and shared across all handlers/threads:

These are written only at these moments:

Phase Functions
Init init_extraction() (allocates configured_protocols[] and mmt_configured_handlers_map)
Plugin load load_plugin() / load_plugins() (append to plugin_handlers_list)
(Un)registration register_protocol(), unregister_protocol_by_id(), unregister_protocol_by_name()
Handler create/destroy mmt_init_handler() (insert), mmt_close_handler() (delete) — mutate mmt_configured_handlers_map
Teardown close_extraction() (internally close_plugins(), free_registered_protocols())

Locking

Each global has a dedicated pthread_mutex_t:

The locks are never held nested, so there is no lock-ordering hazard (classify_freeze_mutex is released before mmt_init_handler() takes configured_handlers_map_mutex): load_plugin() releases the plugin-list lock around init_proto_fct(), which is the call that may take configured_protocols_mutex via register_protocol(); configured_handlers_map_mutex is held only around the single map insert/delete call and takes no other lock while held.

The single-threaded teardown iteration in close_extraction() walks the map via iterate_through_mmt_handlers() (an unlocked read) and force-closes each handler through mmt_close_handler() (which takes the lock). This is safe only because teardown runs on a single thread per the init-before-workers contract; taking the lock in the iterator would self-deadlock against that nested mmt_close_handler() delete. For the same reason, iterate_through_mmt_handlers() (a public symbol) is not safe to call concurrently with mmt_init_handler() / mmt_close_handler() — it is an unlocked reader of the map.

Note: protocol analysis/classification switches are per-handler atomics

mmt_init_handler() also enables analysis and classification for every protocol — enable_protocol_analysis() / enable_protocol_classification() set analysis_enabled / classify_enabled in the handler’s own mmt_handler->configured_protocols[proto_id] snapshot (issue #549; before it they wrote data_analyser.status / classify_next.status on the shared global protocol descriptor, so disabling a protocol on one handler disabled it on every handler, and the next mmt_init_handler() re-enabled it for all). The disable_* calls likewise affect only the handler they are given. The flags are read lock-free on the per-packet hot path (proto_packet_analyze(), proto_packet_classify_next()), so they are not mutex-guarded.

Every access to these flags is atomic (issue #69): the hot-path reads and the enable_*/disable_* writes go through proto_status_load() / proto_status_store() in src/mmt_core/private_include/packet_processing.h, which use the compiler __atomic_* builtins with __ATOMIC_RELAXED ordering. A relaxed atomic load of an int compiles to a plain load on common targets, so the hot path stays lock-free and just as cheap, and a caller toggling a handler’s switch from a thread other than the one feeding it packets stays race-free. Relaxed ordering is sufficient because the flag only gates whether analysis/classification runs: the function pointers and analyse/classify lists it guards are populated during single-threaded init_extraction() / registration and are stable before any worker observes the flag, so no other data is published through it.

Consequence: creating the very first handlers from multiple threads simultaneously (with no prior single-threaded mmt_init_handler()) is race-free — each mmt_init_handler() writes only its own handler’s switches, so concurrent creations share no flag at all. TSan is clean on this cold concurrent-create path. Before #69 this was a deliberately accepted, currently-benign data race that required a prior single-threaded mmt_init_handler() (e.g. the warmup the handler-churn harness used) to finalize the then-shared flags.

These mutexes are taken only on the (un)registration/teardown mutation paths and in mmt_init_handler() / mmt_close_handler(). They are deliberately not taken on the per-packet processing hot path. That is safe because:

So under the documented init-before-workers ordering, the global registries are read-only while workers run, and concurrent calls to mmt_init_handler() only perform concurrent reads of the global table - which is safe.

Exception: S1AP decoder state

The S1AP dissector (libmmt_tmobile) keeps two process-wide tables that the per-packet path updates, so S1AP packets — unlike the rest of the hot path — take a mutex:

The lock order is fixed: attribute extraction (proto_s1ap.c) holds the entity mutex while it decodes the packet, and the decoder takes _nas_ciphering_mutex inside it — mutex → _nas_ciphering_mutex, never the reverse (_free_entities_list() releases mutex before it resets the ciphering table), so the two cannot deadlock. Both tables are shared by every handler, and the S1AP protocol-cleanup callback (run when a handler is closed) and s1ap_entities_reset() clear both of them — for all handlers, not just the one being closed.

Attribute extraction parses each S1AP packet once into a per-thread cache (issue #507, _extraction_att() in src/mmt_mobile/proto_s1ap.c). The cache is keyed by the handler, the per-handler packet_id and a generation counter the store bumps each time it is cleared, and it holds copies of the packet’s entities taken under mutex, never pointers into the shared list — so a thread driving several handlers, or a handler outliving another one’s close, never reads another packet’s data or a freed entity.

The NAS decoders under src/mmt_mobile/nas/ keep no global state: each one returns its error code (issue #507 removed the process-global errorCodeDecoder every decoding thread wrote; see src/mmt_mobile/nas/util/decoder.h).

The security rule engine — one engine per handler

The optional rule engine (libmmt_security, built with ENABLESEC=1) keeps its state per engine, and an engine belongs to one handler (issue #551; until then the rule list top_rule, the options op and the loader’s father stack were process globals and #512 allowed one engine per process). The state is a mmt_sec_engine_t (src/mmt_security/tips_internal.h): the options, the rule list with its rule instances, the packet counter, the XML loader’s stacks and the corrupted-message count. Every engine entry point binds its engine to the calling thread for its own duration through the thread-local sec_cur (sec_bind() in src/mmt_security/tips.c) — the load in init_sec_lib(), the packet handler analyse_incoming_packet(), close_sec_lib() and the summaries — so two engines never see each other’s state:

Shared by every engine of the process, by design: the detection_<n>.data counter (numbers files in one reaction folder; claimed atomically, open_detection_file() in src/mmt_security/tips_report.c) and the embedded-function library handle (opened once under a mutex, embedded_funct_resolve() in src/mmt_security/tips_extract.c).

Per-handler state (not shared)

The following live inside mmt_handler_t and are private to the owning thread, so they need no locking as long as the “one handler per worker” rule holds:

Checklist for multi-threaded use

  1. On the main thread: call init_extraction() (which loads plugins and registers protocols). Wait for it to finish.
  2. Create one mmt_handler_t per worker thread with mmt_init_handler(). Creating them on the main thread before spawning is always safe. Creating them inside the workers concurrently is safe too — the handler-bookkeeping map insert is guarded by configured_handlers_map_mutex, the global protocol table is read-only by then, and the analysis/classification switches are per-handler and accessed atomically (see the note under Locking), so even the very first handlers may be created concurrently from cold.
  3. Each worker feeds packets only to its own handler.
  4. On shutdown: stop all workers first, free each handler with mmt_close_handler(), then call the public global teardown close_extraction() once, on a single thread. It force-closes any handler still registered and runs the internal close_plugins() / free_registered_protocols() steps itself (src/mmt_core/src/packet_registry.c:677-689); a handler must not be used or closed after it returns. The single-threaded embedding sequence is in USER_GUIDE.md §3.

Verification (ThreadSanitizer harness)

The contract above is mechanically verified by a multi-threaded harness (tools/phase0/tests/mt_tsan_harness.c) built and run under ThreadSanitizer. Because TSan only detects races in code compiled with -fsanitize=thread, the SDK itself is built with the BUILD=tsan profile (see rules/common.mk) and the harness is compiled and linked against that instrumented build — including the dlopen’d libmmt_tcpip.so protocol plugin.

The harness has three modes:

Run it locally:

bash tools/phase0/tests/run_mt_tsan_test.sh

The script builds + installs the BUILD=tsan SDK into a temporary prefix, synthesizes the RADIUS pcap, compiles the harness, and runs it under TSan (-fno-sanitize-recover=all, so any detected race aborts). CI runs the same script in the harness-tsan leg of .github/workflows/phase0-baseline.yml, through tools/phase0/run_all_harnesses.sh --group tsan (issue #655).