summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorHarald Welte <laforge@gnumonks.org>2017-12-05 19:39:14 +0100
committerHarald Welte <laforge@gnumonks.org>2017-12-05 19:39:14 +0100
commit8100f4975960f9dac3534cac97caca74b572bfc0 (patch)
tree9557c8f6118fb08cdd4cc88387d2146cc9f3cddb
parent3671fbfaf5575bc2c570f60b9800d62b3e4e4b4a (diff)
ttcn3 paper: final version
-rw-r--r--2017/ttcn-netdev2.2/paper/isea.tex57
1 files changed, 44 insertions, 13 deletions
diff --git a/2017/ttcn-netdev2.2/paper/isea.tex b/2017/ttcn-netdev2.2/paper/isea.tex
index d329f49..fc83e00 100644
--- a/2017/ttcn-netdev2.2/paper/isea.tex
+++ b/2017/ttcn-netdev2.2/paper/isea.tex
@@ -20,7 +20,7 @@
/Author (Harald Welte)}
% The file isea.sty is the style file for ISEA 2015 proceedings.
%
-\title{TTCN-3 and Eclipse TITAN for testing protocol stacks [draft]}
+\title{TTCN-3 and Eclipse TITAN for testing protocol stacks}
\author{Harald Welte\\
sysmocom - systems for mobile communications GmbH\\
Berlin, Germany\\
@@ -181,7 +181,7 @@ as {\tt universal charstring} (UCS-4).
Using the {\tt record}, {\tt set}, {\tt union}, {\tt record of} and {\tt set of} TTCN-3 structured types,
programmers can create abstract container types.
-FIXME: example
+% FIXME: example
TTCN-3 also knows a {\tt enumberated} type, like many other programming languages.
@@ -240,7 +240,7 @@ template MyMessageType tr_MyTemplate
Templates can be hierarchical, so from the most generic to the most specific case, you can specify ever more
concrete templates, depending on the need.
-FIXME: Example
+% FIXME: Example
Templates can further be specified as {\tt subset}, {\tt superset}, {\tt permutation} (any order of elements).
@@ -249,13 +249,13 @@ against one or multiple receive templates. Matching against a fixed template wo
over e.g. comparing with a 'const struct' in a C-language test case. However, TTCN-3 templates can have
wild-cards and pattern matching.
-FIXME: Example
+% FIXME: Example
The built-in {\tt match()} function can be used to determine if a received
message (or actually any value) matches the template. Even more so, the fundamental {\tt receive()} function,
which is used to receive any inbound message from a test port, has built-in matching capability, so explicit
-calls of {\tt match()} are rarely required. See FIXME later.
+calls of {\tt match()} are rarely required. See Section \ref{receive} below.
\subsection{TTCN-3 + TITAN encoders/decoders}
@@ -331,10 +331,11 @@ type record GRE_Header {
Much more complex constructs are possible in the TTIAN binary codec, e.g. the extension octet concept found in
many telecom protocols.
-FIXME: examples for a text based protocol
+% FIXME: examples for a text based protocol
\subsection{Abstract Communications Operations}
+\label{receive}
Abstract communications happens on the {\em test ports} which connect the test case of the abstract test suite
(ATS) with the implementation under test (IUT). TTCN-3 supports abstract communications operations for both
@@ -413,21 +414,51 @@ interleave {
}
\end{lstlisting}
+\subsection{Fuzzing extensions}
-FIXME: synchronous API calls example
+As the TTCN-3 type notation contains stringent information about what values are permitted, it is normally not
+possible to create illegal values. TITAN introduces the {\tt erroneous} extension which can be usd to
+generate invalid messages, e.g. those missing mandatory fields, with invalid values, etc.
-FIXME: diagram about netfilter conntrack test suite
+\section{Test execution}
-FIXME: test execution
+The test suite, comprised of any number of individual test cases is executed by the TITAN executor. The
+executor takes care of starting any of the parallel test components (on the local host or even on remote
+hosts), performing the test cases as indicated by either the command line, or by the configuration file, or in
+absence of that by the compile-time default list of testcases in the ({\tt control}) section of the TTCN-3
+code. It also manages opening all configured log files / plugins, as described below. The test executor
+is typically started using the {\tt ttcn3\_start} program.
-FIXME: configuration file / moduleparam
+\subsection{Configuration File}
-FIXME: logfile, including match error
+Every TITAN test suite has a configuration file, where many aspects of test execution can be specified. Among
+others, you can configure the log verbosity of the TITAN-internal logging. There are different log masks
+for the log file ({\tt FileMask}) and for the console ({\tt FileMask}). One extremely useful feature is
+the {\tt TTCN\_MATCHING} logging, which will provide detailed information about the exact (even nested) field
+of a value that did not match in any explicit or implicit matching against a template. Another useful example
+is {\tt TTCN\_ENCDEC}, which will automatically log the input and output of every encode/decode function,
+which is very useful during debugging of codec problems.
-FIXME: Junit-XML output plugin
+\subsection{Log Files}
+TITAN writes log files in a structured format, which makes them easy to parse by downstream log-processing
+tools. Every parallel test component will write its own log file. As logs are timestamped, the {\tt
+ttcn3\_logmerge} tool can be used to {\em splice} the individual log files into each other, providing one
+log file of all components, sorted by timestamps.
+
+The {\tt ttcn3\_logformat} tool can be used to convert the single-line nested value representation into a
+multi-line format with proper indenting which is more easy to understand as a human. Compare this to e.g.
+JSON or XML pretty-printing.
+
+\subsection{Junit-XML output plugin}
+
+The TITAN runtime understands the concept of log output plugins. Such plugins can be custom-developed to
+generate whatever output format from test case logs / results as needed. A number of standard plugins are
+included, among them the capability to write Junit-XML output, which is a standard format for reporting test
+results in a machine-readable format. This format is understood by the popular Jenkins continuous integration
+software, so that one can easily feed per-testcase results into the Jenkins test results analyzer as part of a
+CI test suite.
-FIXME: fuzzing extensions
\section{Acknowledgments}
The author acknowledges that like everyone in Free Software, he is merely standing on the shoulders of
personal git repositories of Harald Welte. Your mileage may vary