MMT-DPI Threading Model
This document describes the threading contract for MMT-DPI: what state is global, what state is per-handler, and the rules a multi-threaded application must follow to use the library safely.
Summary of the contract
- Initialization before workers.
init_extraction()and all protocol / plugin (un)registration must complete on a single thread before any worker thread starts processing packets. - One handler per worker. Each worker thread owns its own
mmt_handler_t(created withmmt_init_handler()). A handler must not be shared or used concurrently by more than one thread. Creating and destroying handlers (mmt_init_handler()/mmt_close_handler()) concurrently from multiple threads is supported — the global handler-bookkeeping map they mutate is mutex-guarded (see below), and the analysis/classification switches eachmmt_init_handler()enables are the new handler’s own, accessed atomically (see the note under Locking), so even the very first handlers may be created concurrently from cold. - No registration once workers are running. Registering or unregistering
protocols/plugins while workers are processing packets is discouraged; if a
defensive caller does so, the mutations are mutex-guarded (see below) but the
per-handler snapshot taken at
mmt_init_handler()time will not reflect the change.
Following these rules makes the per-packet hot path completely lock-free.
Global (process-wide) state
Three pieces of state are global and shared across all handlers/threads:
plugin_handlers_list(src/mmt_core/src/plugins_engine.c) - linked list of loaded plugin handles.configured_protocols[PROTO_MAX_IDENTIFIER]andconfigured_protocols_names_map(src/mmt_core/src/packet_registry.c) - the table of protocol descriptors and the name->descriptor index.mmt_configured_handlers_map(src/mmt_core/src/packet_registry_handler.c) - handler-lifecycle bookkeeping: the set of livemmt_handler_tinstances, used byiterate_through_mmt_handlers()and force-closed at teardown. It is not touched on the per-packet hot path.
These are written only at these moments:
| Phase | Functions |
|---|---|
| Init | init_extraction() (allocates configured_protocols[] and mmt_configured_handlers_map) |
| Plugin load | load_plugin() / load_plugins() (append to plugin_handlers_list) |
| (Un)registration | register_protocol(), unregister_protocol_by_id(), unregister_protocol_by_name() |
| Handler create/destroy | mmt_init_handler() (insert), mmt_close_handler() (delete) — mutate mmt_configured_handlers_map |
| Teardown | close_extraction() (internally close_plugins(), free_registered_protocols()) |
Locking
Each global has a dedicated pthread_mutex_t:
plugin_handlers_list_mutexguards the list mutations inload_plugin()andclose_plugins(), and the class-provider registry’s writes inmmt_register_class_provider()andclose_plugins(); the class lookups read it without the lock, like the other load-time registries.configured_protocols_mutexguards the registry mutations inregister_protocol(),unregister_protocol_by_id()andunregister_protocol_by_name().configured_handlers_map_mutexguards themmt_configured_handlers_mapmutations inmmt_init_handler()(insert) andmmt_close_handler()(delete), making concurrent handler create/destroy race-free.classify_freeze_mutex(src/mmt_core/src/packet_registry_classify.c, issue #560) serialises the classification-chain freezemmt_init_handler()runs after allocating the handler: each protocol’s weight-sorted checker list is copied, in list order, into one contiguous array that the walk inproto_packet_classify_next()reads. Unlike the descriptor snapshot below, the chains are shared by every handler through the globalprotocol_t. The array is published with a release store and read with an acquire load, so handlers created concurrently, from cold, freeze it once and walk it race-free. A registration (or ammt_classifier_mark_legacy()/mmt_classifier_set_guard()/mmt_classifier_set_guard_view()call, issue #561) after the freeze moves the array onto a retired list, freed only at teardown, and the walk uses the list until the nextmmt_init_handler()freezes it again — so a registration made after handlers exist but while no worker runs (rule 3) is walked by every handler from its next packet. A registration while workers run stays unsupported: the list insert itself is not synchronised against a walk.
The locks are never held nested, so there is no lock-ordering hazard
(classify_freeze_mutex is released before mmt_init_handler() takes
configured_handlers_map_mutex):
load_plugin() releases the plugin-list lock around init_proto_fct(), which is
the call that may take configured_protocols_mutex via register_protocol();
configured_handlers_map_mutex is held only around the single map insert/delete
call and takes no other lock while held.
The single-threaded teardown iteration in close_extraction() walks the map via
iterate_through_mmt_handlers() (an unlocked read) and force-closes each handler
through mmt_close_handler() (which takes the lock). This is safe only because
teardown runs on a single thread per the init-before-workers contract; taking
the lock in the iterator would self-deadlock against that nested
mmt_close_handler() delete. For the same reason, iterate_through_mmt_handlers()
(a public symbol) is not safe to call concurrently with
mmt_init_handler() / mmt_close_handler() — it is an unlocked reader of the map.
Note: protocol analysis/classification switches are per-handler atomics
mmt_init_handler() also enables analysis and classification for every
protocol — enable_protocol_analysis() / enable_protocol_classification()
set analysis_enabled / classify_enabled in the handler’s own
mmt_handler->configured_protocols[proto_id] snapshot (issue #549; before it
they wrote data_analyser.status / classify_next.status on the shared global
protocol descriptor, so disabling a protocol on one handler disabled it on
every handler, and the next mmt_init_handler() re-enabled it for all). The
disable_* calls likewise affect only the handler they are given. The flags are
read lock-free on the per-packet hot path (proto_packet_analyze(),
proto_packet_classify_next()), so they are not mutex-guarded.
Every access to these flags is atomic (issue #69): the hot-path reads and
the enable_*/disable_* writes go through proto_status_load() /
proto_status_store() in src/mmt_core/private_include/packet_processing.h,
which use the compiler __atomic_* builtins with __ATOMIC_RELAXED ordering. A
relaxed atomic load of an int compiles to a plain load on common targets, so
the hot path stays lock-free and just as cheap, and a caller toggling a
handler’s switch from a thread other than the one feeding it packets stays
race-free. Relaxed ordering is sufficient because the flag only gates whether
analysis/classification runs: the function pointers and analyse/classify lists
it guards are populated during single-threaded init_extraction() /
registration and are stable before any worker observes the flag, so no other
data is published through it.
Consequence: creating the very first handlers from multiple threads
simultaneously (with no prior single-threaded mmt_init_handler()) is
race-free — each mmt_init_handler() writes only its own handler’s
switches, so concurrent creations share no flag at all. TSan is clean on this
cold concurrent-create path. Before #69 this was a deliberately accepted,
currently-benign data race that required a prior single-threaded
mmt_init_handler() (e.g. the warmup the handler-churn harness used) to
finalize the then-shared flags.
These mutexes are taken only on the (un)registration/teardown mutation
paths and in mmt_init_handler() / mmt_close_handler(). They are
deliberately not taken on the per-packet processing hot path. That is safe because:
init_extraction()allocates the descriptor table once, before workers start.mmt_init_handler()copies (snapshots) the descriptor pointers from the globalconfigured_protocols[]into the handler’s ownmmt_handler->configured_protocols[]array.- Packet processing reads only that per-handler snapshot, never the global table.
So under the documented init-before-workers ordering, the global registries are
read-only while workers run, and concurrent calls to mmt_init_handler() only
perform concurrent reads of the global table - which is safe.
Exception: S1AP decoder state
The S1AP dissector (libmmt_tmobile) keeps two process-wide tables that the
per-packet path updates, so S1AP packets — unlike the rest of the hot path —
take a mutex:
- the entity list (
src/mmt_mobile/proto_s1ap.c), guarded by its ownmutex; - the per-S1-connection NAS ciphering algorithm table (issue #452,
src/mmt_mobile/s1ap/s1ap_common.c): a fixed array ofS1AP_NAS_CIPHERING_SLOTS(1024) entries, direct-mapped by MME-UE-S1AP-ID, guarded by_nas_ciphering_mutex. It is written from a DownlinkNASTransport Security Mode Command, freed per connection by a UEContextReleaseCommand, and read on the InitialContextSetupRequest Attach Accept path. A colliding connection overwrites the slot, and a lookup matches both S1AP IDs, so an evicted connection reads as “unknown”, never as another UE’s algorithm.
The lock order is fixed: attribute extraction (proto_s1ap.c) holds the
entity mutex while it decodes the packet, and the decoder takes
_nas_ciphering_mutex inside it — mutex → _nas_ciphering_mutex, never
the reverse (_free_entities_list() releases mutex before it resets the
ciphering table), so the two cannot deadlock. Both
tables are shared by every handler, and the S1AP protocol-cleanup callback
(run when a handler is closed) and s1ap_entities_reset() clear both of them
— for all handlers, not just the one being closed.
Attribute extraction parses each S1AP packet once into a per-thread cache
(issue #507, _extraction_att() in src/mmt_mobile/proto_s1ap.c). The cache is keyed by the handler, the per-handler packet_id
and a generation counter the store bumps each time it is cleared, and it holds
copies of the packet’s entities taken under mutex, never pointers into the
shared list — so a thread driving several handlers, or a handler outliving
another one’s close, never reads another packet’s data or a freed entity.
The NAS decoders under src/mmt_mobile/nas/ keep no global state: each one
returns its error code (issue #507 removed the process-global
errorCodeDecoder every decoding thread wrote; see src/mmt_mobile/nas/util/decoder.h).
The security rule engine — one engine per handler
The optional rule engine (libmmt_security, built with ENABLESEC=1) keeps
its state per engine, and an engine belongs to one handler (issue #551; until
then the rule list top_rule, the options op and the loader’s father stack
were process globals and #512 allowed one engine per process). The state is a
mmt_sec_engine_t (src/mmt_security/tips_internal.h): the options, the rule
list with its rule instances, the packet counter, the XML loader’s stacks and
the corrupted-message count. Every engine entry point binds its engine to the
calling thread for its own duration through the thread-local sec_cur
(sec_bind() in src/mmt_security/tips.c) — the load in init_sec_lib(),
the packet handler analyse_incoming_packet(), close_sec_lib() and the
summaries — so two engines never see each other’s state:
init_sec_lib(mmt, …)loads the rule file into a new engine and registers its packet handler onmmt. Several handlers of one process may each load their own engine, with different rule files; each one reports only its own rules’ verdicts, to its own callback anduser_args(tests/rule_engine/test_rule_eval.c,two-engines).- A handler runs at most one engine: a second
init_sec_lib()on the same handler returns-1(“Error 105”) without touching the loaded rules (the engine it allocated for the attempt is freed). The handler is claimed under the engine registry’s mutex (sec_engines_lock), so of two threads that race to load an engine on one handler exactly one wins. - A load that fails — unreadable or malformed rule file, missing embedded
function — returns
-1, frees everything it built and releases the handler, soinit_sec_lib()may be called on it again. - The engine is the handler’s packet handler 1:
init_sec_lib()refuses a handler whose packet handler 1 is already registered (“Error 42”). close_sec_lib(mmt)unregisters the engine’s packet handler and frees its rule list, live rule instances and options; call it beforemmt_close_handler(mmt), from the thread that feedsmmt— not from the engine’s own result callback, where it returns-1. Afterwards the handler may load a new engine. An engine whose handler was closed without it stays allocated until a later handler gets the same address; it never answers for, or blocks, that handler (find_engine_locked()insrc/mmt_security/tips.cdrops it when the live handler’s packet handler 1 is not the engine’s).- Packets reach an engine only through its handler, so the “one handler per worker” rule makes each engine single-threaded: feed each handler from one thread. Engines on different handlers may run on different threads at once.
print_summary()andxml_summary()take no engine argument: they report the engine bound on the calling thread (from inside a result callback), otherwise the most recently loaded live engine.
Shared by every engine of the process, by design: the detection_<n>.data
counter (numbers files in one reaction folder; claimed atomically,
open_detection_file() in src/mmt_security/tips_report.c) and the
embedded-function library handle (opened once under a mutex,
embedded_funct_resolve() in src/mmt_security/tips_extract.c).
Per-handler state (not shared)
The following live inside mmt_handler_t and are private to the owning thread,
so they need no locking as long as the “one handler per worker” rule holds:
mmt_handler->configured_protocols[]- the per-handler snapshot of protocol descriptors plus that handler’s session/protocol contexts.sessions_map(per protocol instance) - session tables.ip_streams- IP reassembly/stream state.timeout_milestones_map, packet counters, current-packet scratch state, etc.
Checklist for multi-threaded use
- On the main thread: call
init_extraction()(which loads plugins and registers protocols). Wait for it to finish. - Create one
mmt_handler_tper worker thread withmmt_init_handler(). Creating them on the main thread before spawning is always safe. Creating them inside the workers concurrently is safe too — the handler-bookkeeping map insert is guarded byconfigured_handlers_map_mutex, the global protocol table is read-only by then, and the analysis/classification switches are per-handler and accessed atomically (see the note under Locking), so even the very first handlers may be created concurrently from cold. - Each worker feeds packets only to its own handler.
- On shutdown: stop all workers first, free each handler with
mmt_close_handler(), then call the public global teardownclose_extraction()once, on a single thread. It force-closes any handler still registered and runs the internalclose_plugins()/free_registered_protocols()steps itself (src/mmt_core/src/packet_registry.c:677-689); a handler must not be used or closed after it returns. The single-threaded embedding sequence is in USER_GUIDE.md §3.
Verification (ThreadSanitizer harness)
The contract above is mechanically verified by a multi-threaded harness
(tools/phase0/tests/mt_tsan_harness.c) built and run under ThreadSanitizer.
Because TSan only detects races in code compiled with -fsanitize=thread, the
SDK itself is built with the BUILD=tsan profile (see rules/common.mk) and the
harness is compiled and linked against that instrumented build — including the
dlopen’d libmmt_tcpip.so protocol plugin.
The harness has three modes:
replay— follows the documented init-before-workers contract:init_extraction(), all registration, and one ownmmt_handler_tper worker are all created on the main thread; then N worker threads run concurrently, each replaying the same read-only packet array through its own handler (the lock-free per-packet hot path). Each worker builds an order-independent classification fingerprint; the harness asserts every worker’s fingerprint equals a single-thread baseline. Replaying a synthetic multi-flow RADIUS pcap exercises the per-session RADIUS parser state (issue #23) concurrently across threads.registry-stress— spawns N threads that concurrently drive the global registry mutation API (unregister_protocol_by_id/unregister_protocol_by_name), hammeringconfigured_protocols_mutexdirectly (issue #22). This is intentionally not the normal runtime pattern; it drives the lock so TSan can confirm the mutex serialises the registry mutations.handler-churn— spawns N threads that concurrently create and destroy their ownmmt_handler_tin a loop (mmt_init_handler()/mmt_close_handler()), hammering the global handler bookkeeping map (mmt_configured_handlers_map) guarded byconfigured_handlers_map_mutex(issue #67). Without that lock the concurrentinsert_key_value/delete_key_valuemap mutations are a data race that TSan flags; with it the run is clean. The workers are the first callers ofmmt_init_handler()afterinit_extraction()— there is no main-thread warmup handler — so this mode also exercises the cold concurrent-create path that enables every protocol’s analysis/classification switch (issue #69; per-handler since issue #549); those flag accesses are atomic, so TSan stays clean.init_extraction()/close_extraction()still bracket the run on the main thread, so the map alloc/free stays single-threaded.
Run it locally:
bash tools/phase0/tests/run_mt_tsan_test.sh
The script builds + installs the BUILD=tsan SDK into a temporary prefix,
synthesizes the RADIUS pcap, compiles the harness, and runs it under TSan
(-fno-sanitize-recover=all, so any detected race aborts). CI runs the same
script in the harness-tsan leg of .github/workflows/phase0-baseline.yml,
through tools/phase0/run_all_harnesses.sh --group tsan (issue #655).