TTCN-3 and Eclipse TITAN for testing protocol stacks ==================================================== :author: Harald Welte :copyright: 2017-2018 by Harald Welte (License: CC-BY-SA) :backend: slidy :max-width: 45em == Protocol Testing Important for: * conformance to specification * ensuring interoperability * network security * regression testing * performance == Protocol Testing No standard methodology, language, approach, tool * testing implementation against itself ** works only for symmetric protocols ** wouldn't cover lots of problems * testing against wireshark ** wireshark often way more tolerant than spec * custom implementation ** in Python (e.g. using scapy) ** in Erlang (good binary encoder/decoder) or other languages * specific tools like packetdrill == Protocol Testing Personal story: During past years, * I implemented tons of [telecom] protocols / stacks at Osmocom.org * I was looking for better tools to help [automatic] testing ** primarily functional testing (correctness / conformance) ** not so much performance testing * I figured Ideal test tool would... ** allow very productive and expressive way to describe encoding/decoding ** allow very convenient pattern matching on incoming messages ** allow exchange of messages asynchronously with implementation under test * I stumbled on TTCN-3 occasionally and investigated == The TTCN-3 Language * domain-specific language *just* for protocol conformance tests * TTCN history back to 1983 (!), TTCN-3 since 2000 * used extensively in classic telecom sector (Ericsson, Nokia, etc.) * ETSI developed and published abstract test suites in TTCN-3 for ** IPv6, SIP, DIAMETER, ePassports, Digital Mobiel Radio, 6LoWPAN * Other bodies published test suites for ** CoAP, MQTT, MOST, AUTOSAR But: Until 2015, only proprietary tools / compilers :( == Eclipse TITAN * After TTCN-3 specification in 2000, Ericsson internally develops TTCN-3 toolchain * adopted for many Ericsson-internal testing of all kinds of products * proprietary software with commercial licenses * 300,000 lines of Java + 1.6 Million lines of C++ * Released as Open Source as "Eclipse TITAN" in 2015 ** Not just TTCN-3 compiler, but also extensive documentations and many protocol modules, test ports as well as Eclipse IDE, Log file viewer/visualizer, etc. * `eclipse-titan` part of standard Debian / Ubuntu archive, only one apt-get away Great, we can finally use TTCN-3 in FOSS! == Eclipse TITAN compiler workflow [graphviz] ---- digraph G { progra [label="Human Developer"]; ttcn3 [label="TTCN-3 source (ATS)"]; cpp [label="Generated C++ source"]; exec [label="Binary Executable (ETS)"]; tps [label="Other C++ sources, as needed"]; progra -> ttcn3 [label="writes code"]; ttcn3 -> cpp [label="ttcn3_compiler"]; cpp -> exec [label="GNU gcc / g++"]; tps -> cpp } ---- * TITAN actually _compiles_ into executable binaries, it is not using a VM or scripting ** ATS: Abstract Test Suite (source code) ** ETS: Executable Test Suite (executable code) == TTCN-3 Language Features (with TITAN) * comprehensive type system * parametric templates * variety of encoders/decoders * automatic / comprehensive logging framework * powerful program control statements * built-in notion of tests cases, test suites, verdicts, ... * runtime / executor for parallel test components + aggregating results == TTCN-3 Basic Types * Simple basic types such as `integer`, `float`, `boolen` * Basic string types such as `bitstring`, `octetstring`, `hexstring`, `charstring` (IA5) and `universal charstring` (UCS-4). * Structured Types `record`, `set`, `record of`, `set of` * Verdict type `verdicttype` ** can have either value `none`, `pass`, `inconc`, `fail`, or `error` ** verdict can only _deteriorate_ (`pass` -> `fail`) but never improve (`error` -> `pass`) ** every test case implicitly has a verdict, no need to explicitly declare a variable of `verdicttype` == TTCN-3 Structured Types A structured type is an abstract type comprised of other types, whcih can be nested. An example for a `record` type (similar to a C-language `struct`) is shown below -------- type record MyMessageType { integer field1 optional<1>, charstring field2, boolean field3 }; -------- <1> optional members may be present or not == TTCN-3 Union Type A union expresses a set of alternative types of which one alternative must be chosen. -------- type union MyMessageUnion { integer field1, charstring field2, }; -------- Difference to C-language union: `ischosen()` can be used to learn which of the union members is chosen/defined! == Not-used and omit * until a variable or field of structured type is assigned, it is _unbound_ * whenever a _value_ is expected, TTCN-3 runtime will create an error for _unbound_ * in case of absence of optional fields, explicit `omit` value must be assigned! == Sub-typing Sub-typing can be used to further constrain a given type. Typical examples include constrained number ranges, and string patterns -------- type integer MyIntRange (1..100); type integer MyIntRange8 (0..infinity); type charstring MyCharRange (”k”..”w"); type charstring SideType (”left”, ”right”); type integer MyIntListRange (1..5,7,9); type record length(0..10) of integer RecOfInt; type charstring CrLfTermStrin (pattern ”*\r\n”); -------- == Templates * Matching incoming messages against some kind of specification is one of the most common tasks in testing protocols ** some expected fields are static (message type) ** some expected fields are known (source address) ** some fields are chosen by sender (some identifier) ** some fields we don't care (optional headers that may or may not be present) * TTCN-3 Templates provide elegant solution for this, avoiding any explicit code to be written ** templates can even be parametric, i.e. they can be instantiated with "arguments" * templates can also be used for sending messages, if they are fully specified/qualified == Templates -------- // Value list template template charstring tr_SingleABorC := (”A”, ”B”, ”C”); -------- -------- // Value range template float tr_NearPi := (3.14 .. 3.15); template integer tr_FitsToOneByte := (0 .. 255); template integer tr_GreaterThanZero := (1 .. infinity); -------- -------- // Intermixed value list and range matching template integer tr_Intermixed := ((0..127), 200, 255); -------- == Matching inside values -------- // Using any element matching inside a bitstring value // Last 2 bits can be '0' or '1' template bitstring tr_AnyBSValue := ’101101??’B; -------- -------- // Matches charstrings with the first character "a" // and the last one "z" template charstring tr_0 := pattern "a*z"; -------- * more capabilities using `complement`, `ifpresent`, `subset`, `superset`, `permutation` constructs not covered here == Parametric Templates See below for an example of a parametric template: -------- type record MyMessageType { integer field1 optional, charstring field2, boolean field3 }; template MyMessageType trMyTemplte(boolean pl_param) := { field1 : = ?, // present, but any value field2 : = (”B”, ”O”, ”Q”) , field3 := pl_param }; -------- The built-in `match()` function can be used to check if a given value matches a given template. Some TTCN-3 statements such as `receive()` have built-in capabilities for template matching, avoiding even the explicit call of `match()` in many cases. == Template Hierarchy Using modified templates, one can build a hierarchy of templates: From the specific to the unspecific ---- template MyMsgType t_MyMsgAny := { msg_type := ?, foo := bar }; template MyMsgType t_MyMsg23 modifies t_MyMsgAny := { msg_type := 23, }; ---- where * _t_MyMsgAny_ matches a message with any message type and "foo=bar", while * _t_MMyMsg23_ matches only those that have "foo=bar" and "msg_type=23" == Encoders/Decoders * type system, templates, matching are all nice and great, but we need to get data from wire format into TTCN-3 abstract types * TTTCN-3 specifies importing of formal schema definitios, such as ASN.1, IDL, XSD (XML) and JSON * TITAN has additional codecs for those (many) protocols that lack formal syntax ** `raw` codec for binary protocols (e.g. GTP) ** `text` codec for text based protocols (e.g. HTTP, MGCP, IMAP, ...) * codecs allow you to _express/describe_ the format (declarative programming) rather than the usual imperative approach == TITAN raw codec: UDP Example How to express an UDP header using TITAN raw codec -------- type integer LIN2_BO_LAST (0..65535) with { variant ”FIELDLENGTH(16), COMP(nosign), BYTEORDER(last)” }; type record UDP_header { LIN2_BO_LAST srcport, LIN2_BO_LAST dstport, LIN2_BO_LAST len, LIN2_BO_LAST cksum } with { variant ”FIELDORDER(msb)” }; type record UDP packet { UDP_header header octetstring payload } with { variant (header) ”LENGTHTO(header, payload), LENGTHINDEX(len)” }; -------- == TITAN raw codec: GTP Example How to express an GTP header using TITAN raw codec -------- type record GRE_Header { BIT1 csum_present, BIT1 rt_present, BIT1 key_present, ... OCT2 protocol_type, OCT2 checksum optional, OCT2 offset optional, OCT4 key otional, ... } with { variant (checksum) "PRESENCE(csum_present='1', rt_present='1'B)" variant (offset) "PRESENCE(csum_present='1'B, rt_present='1'B)" variant (key) "PRESENCE(key_present='1'B)" } -------- == TITAN text codec: MGCP Example -------- type charstring MgcpVerb ("EPCF", "CRCX", "MDCX", "DLCX", "RQNT", "NTFY", "AUEP", "AUCX", "RSIP") with { variant "TEXT_CODING(,convert=upper_case,,case_insensitive)" }; type charstring MgcpTransId (pattern "\d#(1,9)"); type charstring MgcpEndpoint (pattern "*@*"); type charstring MgcpVersion (pattern "\d.\d") with { variant "BEGIN('MGCP ')" }; type record MgcpCommandLine { MgcpVerb verb, MgcpTransId trans_id, MgcpEndpoint ep, MgcpVersion ver } with { variant "SEPARATOR(' ', '[\t ]+')" variant "END('\r\n', '([\r\n])|(\r\n)')" }; -------- == Program Control Statements * `if` / `else` like in C * `select` statement similar to C `switch` * `for`, `while`, `do-while` loops like in C * `goto` and `label` * `break` and `continue` like in C == Abstract Communications Operations * TTCN-3 test suites communicate with _implementation under test_ through abstract TestPorts ** TestPorts can be implemented in TTCN-3 or C++ and linked in ** TestPorts must be _connected_ before using send/receive operaitons ** TITAN provides TestPorts for e.g. packet socket, IP/UDP/TCP/SCTP socket, ... * `.send()` performs non-blocking send ** Literal value, constant, variable, specific value template, ... * `.receive()` or `.receive` performs blocking receive ** literal value, constant, variable, template (with matching!), inline template '... but if receive blocks, how can we wait for any of N events? == Program Control and Behavior * program statements are executed in order * blocking statements block the execution of the component * occurrence of unexpected event may cause infinite blocking ---- // x must be the first on queue P, y the second P.receive(x); // Blocks until x appears on top of queue P P.receive(y); // Blocks until y appears on top of queue P // When y arrives first then P.receive(x) blocks -> error ---- This is what leads to the `alt` statement: `alt` declares a seto alternatives covering all events, which * can happen: expected messages, timeouts, ... * must not happen: unexpected faulty messages, no message received, ... * all alternatives inside `alt` are blocking operations == The `alt` statement ---- P.send(req) T.start; // ... alt { [] P.receive(resp) { /* actions to do and exit alt */ } [] any port.receive { /* handle unexpected event */ } [] T.timeout { /* handle timer expiry and exit */ } } ---- * [] is guard condition enables or disables the alternative ** usually empty `[]` equals `[true]` ** can contain a condition like `[x > 0]` ** very good for e.g. state machines to activate some alternatives only in certain states while others may occur in any state == The `alt` and `repeat` statements The `repeat` statement * takes a new snapshot and re-evaluates the alt statement * can appear as last statement in statement blocks of statements ---- P.send(req) T.start; alt { [] P.receive(resp) { /* actions to do and exit alt */ } [] P.receive(keep_alive) { /* handle keep alive message */ repeat } [] any port.receive { /* handle unexpected event */ } [] T.timeout { /* handle timer expiry and exit */ } } ---- == The `interleave` statement The `interleave` statement * enforces N matching events happen exactly once * permits any order! ---- interleave { [] P.receive(1) { /* actions, optional */ } [] Q.receive(4) { /* actions, optional */ } [] R.receive(6) { /* actions, optional */ } } ---- == The `altstep` construct * collection of set of common/shared `alt` ** such as responding to a keep-alive PING, depending on protocol * invoked in-line, inside `alt` statements or _activated as default_ Definition of an `altstep` for PING/PONG handling and guard timer: ---- altstep my_altstep() { [] P.receive(tr_PING) { P.send(ts_PONG); } [] T_guard.timeout { setverdict(fail, "Guard timer timed out"); } } ---- == The `altstep` construct Explicit Usage of the `my_altstep` defined on previous slide: ---- P.send(foo) alt { [] P.receive(bar) [] my_altstep() } ---- this dynamically adds the alternatives from `my_altstep` at the given location and priority of the above `alt`, ensuring that one doesn't have to repeat ping/pong handling as well as global guard timer timeout handling in every `alt` or every single test case all over again. == default `altstep` activation * in previous slide, `altstep` invocation was explicit * some behavior is so universal, that you want to activate it _always_ ---- altstep as_pingpong() { [] P.receive(tr_PING) { P.send(ts_PONG); } } ... var default d1 := activate(as_pingpong()); ... /* code executing with as_pingpong activated */ deactivate(d1); ---- * all alt-statements implicitly have as_pingpong active * but also all _stand-alone receive statements_ ! == TTCN-3 modules TTCN-3 code is written in _modules_ * a test suite consists of one or more modules * a module contains _module definitions_ and an optional _control part_ ** _parameters_ (automatically configurable via config file) ** definition of _data types_, _constants_, _templates_ ** definition of _communications ports_ ** definition of _test components_, _functions_ _altsteps_ and _test cases_ ** _control part_ determines default order/execution of test cases * modules can import from each other (think in python terms) == Examples Let's have a look at some real-world examples and do a bit of a walk-through before continuing with the slides... == Logging * TITAN runtime contains extensive logging framework * config file determines log level for various different subsystems ** e.g. any encode, decode, receive, transmit operations logged ** timer starts, expirations ** any changes to test case verdict * explicit logging from code by use of `log()` built-in function * `ttcn3_logformat` tool for pretty-printing log files * `ttcn3_logmerge` tool for merging/splicing multiple logs * log plugins e.g. for generating JUnit-XML available ** facilitates easy reporting / integration to Jenkins or other CI == Logging Log file format example: ---- // abstract data type before encode 13:30:41.243536 Sent on GTPC to system @GTP_CodecPort.Gtp1cUnitdata : { peer := { connId := 1, remName := "127.0.23.1", remPort := 2123 }, gtpc := { pn_bit := '0'B, s_bit := '1'B, e_bit := '0'B, spare := '0'B, pt := '1'B, version := '001'B, messageType := '01'O, lengthf := 0, teid := '00000000'O, opt_part := { sequenceNumber := '3AAC'O, npduNumber := '00'O, nextExtHeader := '00'O, gTPC_extensionHeader_List := omit }, gtpc_pdu := { echoRequest := { private_extension_gtpc := omit } } } } // 'msg' contains encoded binary data actually sent via socket 13:30:41.243799 Outgoing message was mapped to @IPL4asp_Types.ASP_SendTo : { connId := 1, remName := "127.0.23.1", remPort := 2123, proto := { udp := { } }, msg := '32010004000000003AAC0000'O } ---- == Logging The same log file lines if run through `ttcn3_logformat` ---- 13:30:41.243536 Sent on GTPC to system @GTP_CodecPort.Gtp1cUnitdata : { peer := { connId := 1, remName := "127.0.23.1", remPort := 2123 }, gtpc := { pn_bit := '0'B, s_bit := '1'B, e_bit := '0'B, spare := '0'B, pt := '1'B, version := '001'B, messageType := '01'O, lengthf := 0, teid := '00000000'O, opt_part := { sequenceNumber := '3AAC'O, npduNumber := '00'O, nextExtHeader := '00'O, gTPC_extensionHeader_List := omit }, gtpc_pdu := { echoRequest := { private_extension_gtpc := omit } } } } 13:30:41.243799 Outgoing message was mapped to @IPL4asp_Types.ASP_SendTo : { connId := 1, remName := "127.0.23.1", remPort := 2123, proto := { udp := { } }, msg := '32010004000000003AAC0000'O } ---- == Existing TITAN Source * Protocol encoding/decoding ** BSSAP+, BSSGP, BSSMAP, CoAP, DSS1, DUA, EAP, GRE, GTP, HTTP, ISUP, LLC, M2PA, M2UA, MQTT, MongoDB, NDP, NS, NTAF, ROSE, SCTP, SDP, SNDCP, STOMP, STUN, SUA, TLS, WTP, DNS, IP, SMPP, SNMP, IKEv2, DHCP, PPP, RTP, TCP, UDP, XMPP, DHCPv6, SMTP, ICMP, RTSP, ICMPv6, DIAMETER, FrameRelay, ProtoBuff, IUA, L2TP, M3UA, MIME, WebSocket, H.248, IMAP, IPsec, SRTP, MSRP, ICAP, RADIUS * Protocol Emulation ** M3UA, SCCP, SUA * Test Ports ** GPIO, MTP3, Serial, SocketCAN, SCTP, SIP, HTTP, Telnet, UDP, pcap file, pipe, SQL, TCP, SUNRPC, SSH, STDINOUT, sockets, LDAP == Further Reading * Ericsson TTCN-3 tutorial http://www.ttcn-3.org/files/TTCN3_P.pdf * An Introduction to TTCN-3, 2nd Edition * Modules https://github.com/eclipse * More Modules http://git.eclipse.org/ * Debian https://packages.debian.org/search?keywords=eclipse-titan * Ubuntu https://packages.ubuntu.com/search?keywords=eclipse-titan == EOF End of File