nestepc intro

Please see presentation of nextepc developer Sukchan Lee at OsmoDevCon 2019

This talk is about a "behind the scenes" look at the nextepc codebase through the eyes of an Osmocom developer.

Goal is to understand the high-level software architecture, and whether there is any chance for sharing code/infrastructure between the two projects

linked lists

lib/core/include/core_list.h
double-linked lists like our linuxlist.h
not thread safe
list_insert_sorted() has no libosmo* equivalent
- only user is timer.c, which appears not used?

⇒ looks very much compatible to libosmocore logging

logging

static/global number of log targets
- stdout, console, syslog, network, file
- each target has independent log level
message types (defines formatting)
- RAW, TRACE, LOG, ASSERT
non-RAW logging happens via snprintf to 8k sized stack buffer

⇒ looks very much compatible to libosmocore logging

FSM

nextepc FSM abstraction in lib/core/src/fsm.c
fsm_init(), fsm_dispatch() and fsm_final() are only API functions
states are expressed by switching to a different function pointer for the event handler function
rather simplistic when compared to osmo_fsm
- no onenter/onleave
- no constraints on permitted events / state transitions
- no integrated logging
- no introspection (like VTY or CTRL)
- no concept of FSM classes / instances
- no FSM hierarchy
- no extensive logging by FSM infrastructure

FSM usage

The only point where many instances of FSMs are used in parallel is inside the MME context.c (EMM and ESM).
There don’t appear to be multiple threads in the EMM and ESM FSMs, AFAICT.
overall surprisingly low number of FSMs

⇒ migration to osmo_fsm seems feasible

Events

lib/core/src/event.c
- asynchronous event delivery based on message queue
- event_{create,delete,send,recv,timedrecv}()
timed events use dynamically-allocated timer to send event via queue at timer expiration
- event_timer_create(), timer_create(), periodic_timer_create()
SGW and PGW combine event queue with FSM, where main thread receives events from queue to dispatch them to FSM

⇒ nice idea. libosmo* signals and FSM input events are entirely synchronous. This ensures that all related data structures exist while the event is being processed. Having queued events would require very careful code design and possibly refcounting for pretty much all objects :/

packet buffers

nextepc packet buffers in lib/core/src/unix/pkbuf.c
management structure is kept separate from packet payload
- similar to Linux sk_buff, where you can skb_clone() whcih just duplicates the struct sk_buff but not the actual packet data
single payload pointer to distinguish headers from payload
pools of different buffer sizes (127/256/512/1024/2048/8192)
- index into pool is used as TEID :/
allocations are thread-safe
buffers have reference count (to avoid deep copy?)
- only one user currently: S1AP paging message copying

memory allocator

lib/core/src/unix/malloc.c
core_{malloc,free,calloc,realloc}()
internally uses pkbuf as backing storage (not other way around)

message queue

lib/core/include/core_msgq.h
msgbq_{init,final,create,delete,send,recv,timedrecv}()
contains mutexes, condition variable
blocking and non-blocking receive semantics

⇒ may be candidate for osmo_it_msgq which si currently WIP. Signaling happens via osmo-select compatible eventfd, not condition variable

string utilities

core_strdup, core_strndup
dynamically allocate their result using core_malloc

timers

lib/core/include/core_timer.h
maintains separate list of active and idle timers
- no rbtree, linear list iteration
supports both one-shot and periodic timers (libosmocore only one-shot)
six (!) arguments for timer expiration function call-back function
doesn’t seem to have any direct users, only indirectly via lib/core/src/event.c

TLV

lib/core/include/core_tlv.h
hierarchical TLV parser using dynamically-allocated objects for each TLV
can express repeated tags
can express nested tags
supports only a sub-set of the TLV types (TLV, TL16V, T16L16)
used heavily throughout [generated] GTP

freeDiameter

C-language DIAMETER protocol library
development seems mostly discontinued during past 5 years
- 0 commits during past 12 months
- 11 commits during past 24 months
projects typically use "fork" of freeDiameter copied into their repo
internally uses plenty of threads
applications register callback functions whenever related DIAMETER message is received
- callbacks executed in context of whichever freeDiameter thread

MME

has the following threads:
- sm_thread / sm_main()
  - inbound event queue, dispatched into mme_sm FSM
- net_thread / net_main()
  - endless sock_select_loop()
- whatever threads freeDiameter creates

SGW

has the following threads:
- sgw_thread / sgw_main()
  - inbound event queue, dispatched into sgw_sm FSM
  - handles all of the SGW functionality (S1U to eNB, S11 to MME, S8 to PGW)

PGW

has the following threads:
- pgw_thread / pgw_main()
- inbound event queue, dispatched into pgw_sm FSM
- whatever threads freeDiameter creates

HSS

FIXME
web UI using node.js
- usees tons of dependencies (npm nightmare)
- doesn’t rely on older/distibution-packaged versions
- should IMHO be an optional part, not mandatory

GTPv2C code generation

use 3GPP TS 29.274 word documen
convert .doc to .docx (Office 2007+)
use python script to parse tables in .docx
generate C source code for encode/decoder from python script
lib/gtp/support/gtp_tlv.py

⇒ I love it :)

Same approach also used for generating NAS encoder/decoder
- /lib/nas/support/nas_message.py

S1-AP / ASN.1 PER / asn1c

Tests

C-language testsuite sending/receiving messages onvarious interfaces
resembles the kind of tests we usually do in TTCN-3, but without TTCN-3

Conclusions

nextepc uses threads, but is not heavily multithreaded
- this actually would make libosmocore integration more feasible than originally expected
nextepc seems a much more heavyweight heap user
- e.g. every event or every timer is dynamically allocated vs. osmocom struct embedding, heavy stack use and synchronous event delivery
there are some interesting ideas in nextepc, such as queued event, and timers that generate them
most difficult problems probably around refcounting of packet buffers
GTPv2 code generation could be adopted for OsmoGGSN GTPv2 support

Conclusions

I would love to bring some of our powerful features to nextepc
- talloc with related ability for memory leak debugging
- osmo_fsm with all of its power
- VTY for state introspection and runtime config changes

If I had a dream…

I would
- osmo-ify nextepc MME (logging, FSMs, VTY)
- extend OsmoGGSN with GTPv2 support via the nextepc code generation approach
- implement a simple DIAMENTER→GSUP translator to use OsmoHLR with nextepc
  - use the kernel-side GTP-U user plane to focus on control plane only in GGSN/G-GW
- keep S-GW as-is for now, but think about kernel user plane there, too.
- convert test suite TTCN-3

EOF

End of File

nextepc from Osmocom point of view

nestepc intro

linked lists

logging

FSM

FSM usage

Events

packet buffers

memory allocator

message queue

string utilities

timers

TLV

freeDiameter

MME

SGW

PGW

HSS

GTPv2C code generation

S1-AP / ASN.1 PER / asn1c

Tests

Conclusions

Conclusions

If I had a dream…

EOF