Eclipse Titan™

Eclipse Titan™ is a TTCN-3 compilation and execution environment with an Eclipse-based IDE.

TTCN-3 is a modular language specifically designed for testing (the acronym itself stands for Test and Test Conformance Notation), standardized by ETSI (see www.ttcn-3.org) and endorsed by ITU.

The user of the tool can develop test cases, test execution logic and build the executable test suite for several platforms.

Titan consists of a core part, executing in a Unix/Linux-like environment and a set of Eclipse plug-ins.

Titan

Titan compiles and executes test cases. It has four major roles:

• The TTCN-3 design environment provides an efficient way to create the TTCN-3 test suites (called the abstract test suite, ATS). The IDE is Eclipse‑based and is called Titan Designer.
  
  It has editors that provide the usual IDE features for TTCN-3, ASN.1 and the Titan runtime configuration file. Like creating and configuring Titan projects, syntax highlighting, name convention checking, jump to definition, on-the-fly syntax and semantic analysis, simple content assist, outline of the modules, mark occurrences etc. ASN.1 and XSD sources are typically not created by testers but are coming from specifications. Nevertheless, Titan contains a full-featured ASN.1 editor. XSD sources can be viewed, edited and validated by using any Eclipse XML editors capable of validating XSD.
  
  The Designer also allows building the project. It has a built-in Makefile generator for GNU and GCC makes and allows setting Titan compiler and GCC compiler and linker flags. The source code is first compiled by the Titan compiler to C++ code, then external C++ compiler and linker are called to build the executable file (which is called the host controller - HC). The whole building process is invoked and controlled by the Designer and is running in a command line shell in the background.
• The Titan compiler builds an executable test suite (ETS) from the ATS, the test port code (see below) and the Titan runtime library. The ETS is not always directly executable as the TTCN-3 language and Titan allows very flexible runtime parameterization of the test cases (e.g., IP addresses, port numbers etc. in the lab); the values of runtime parameters need not be defined in development time - though default values can be specified-, but they can be provided just before the test execution session. In this way flexible execution scenarios can be created without re-building the ETS.
• Titan runtime control has several tasks:
  • the Titan Main Controller (MC) read and distributes to all test components the runtime configuration parameters. Titan contains two MC implementations: a command line MC and an MC implemented in the Titan Executor Eclipse plugin.
  • it controls the execution of test cases in a distributed multiplatform environment.
  • it keeps up and running the test system: in case of a runtime error, Titan runtime control cleans up the test system, assigns an "error" verdict to the given test case and starts execution of the next test case).
  • it produces logs in different formats; Titan has a logging API and logging is done via plugins. Currently we have plugins for Titan’s own textual log format and for Jenkins (junit format). Thus, developing a new logging plugin, e.g., for LTTng doesn’t require much knowledge of Titan's code. Several logging mechanisms may be activated at the same time. Logging can be configured by verbosity and by event types. Logs can be written into files or be send to another process or via a network connection to a 3rd party tool.
• Command line log post-processing utilities and the Eclipse-based Titan LogViewer help analyzing the test results. These tools currently process Titan's own log format only.

In a model-based testing (MBT) scenario, the test cases generated from the model are necessarily abstract, as the model itself does not contain low level information. Therefore, the abstract test cases have to be completed by a test harness, to become executable test cases. TTCN-3 and Titan in several projects have proved to be an ideal platform for developing the test harness, for integrating the generated abstract test cases with the test harness and the test environment, and for test execution.

Though Titan is told to be "a TTCN-3 test tool", in fact it can use TTCN-3, ASN.1, XSD and IDL specifications describing the message and signal structures at the tested interfaces. ASN.1 is imported directly, while XSD and IDL are first converted to TTCN-3 and then the generated TTCN-3 modules are used in the projects. In case of XSD the TTCN-3 module is decorated with XML encoding instructions. Titan also supports codec control decorators in TTCN-3 files for binary and textual protocol encodings. Defining the test configurations and the dynamic behaviour of the tests are written in TTCN-3. Functions written in C/C++ can be also called in the TTCN-3 code.

Titan consists of the following components; these are all subject of this project:

• Titan Designer: Eclipse plugin, TTCN-3 & ASN.1 design (advanced editing, on-the-fly syntax & semantic checking) and build the executable.
• Titan Executor: Eclipse plugin, test execution and result reporting.
• Titan LogViewer: Eclipse plugin, offline log representation in tabular and graphical (UML SD-like) formats.
• Titanium: Eclipse plugin for code quality analysis (identifies code smells, draws module dependency graph).
• TTCN-3 and ASN.1 compiler: command line parser, semantic analyzer and C++ code generator. Input files can be TTCN-3 and ASN.1.
• xsd2ttcn: command line tool converting XSD documents to TTCN-3 modules, according to part-9 of the TTCN-3 standard.
• Runtime library: implementation of TTCN‑3 language elements (types, statements, operations, predefined functions etc.) and ETS side of runtime control.
• mctr_cli (command line main controller): runtime control of component distribution and central runtime control of test execution.
• makefilegen: command line Makefile generator.
• logmerge: command line utility to merge the log events events based on their timestamps from the set of textual logfiles produced by the different test components independently.
• logformat: command line utility to nice-format the textual log files.
• logfilter: command line utility to post filtering large log files based on the kind of logged events.
• repgen: command line utility to present not only the formatted log files but the description and TTCN–3 source code of test cases as well as the output of other network monitor programs (like tcpdump) in HTML format.
• tcov2lcov: Titan is able to instrument the generated C++ code and output code coverage data in xml during runtime. This command line utility collects and merges these output files into an LCOV input format.
• Documentation: installation, user, programmer reference guides and API specification.

Test Ports

The TTCN-3 code is generic: the interfaces between the tester and the tested entity (SUT, AUT etc.) are specified at the level of the exchanged abstract data messages and signals. Setting up and maintaining the transport connections, and sending/receving "real" messages and signals are the tasks of interface adaptors. Adaptors are called test ports (TPs) and are plugins written in C++. Titan has a C++ API for adaptors that complete the ATS with the connectivity layer(s) between the test system and the SUT. 

Titan test port (adaptors) included into this project proposal provide the following capabilities:

TCP

Provides communicatin over TCP-type connections in IP networks. It uses the Absrtact Socket library, therefore its supports the features of the Abstract Socket described below. Connections can be static for the time of the test case (opened/starts listening at the TTCN-3 map operation and closed at unmap) or can be opened/closed dynamically from the TTCN-3 code. Multiple TCP connections are supported by one port instance.

UDP

Provides UDP-type connections in IP networks. It maps between Titan's test port API and Linux kernel's UDP socket services; supports open socket, close socket, send and receive data. One test port can handle multiple UDP connections.

TELNET

Allows using remote telnet login from TTCN-3 via the TCP layer of the operating system. The test port supports client and server mode operation. In server mode operation the Telnet Test Port can handle one connection simultaneously. Supports the capabilities of Network Virtual Terminal. Telnet connection parameters (login creadentials, terminal type, prompt format with or without wildcards etc.) can be configured in Titan's runtime configuration file.

SQL

The SQL test port executes SQL statement against the SQL database. The SQL test port is able to handle different SQL engines and databases, and provides an unified interface towards them. Currently MySQL, SQLite C API-s are supported (and required from the database).

PIPE

The PIPE test port is developed to execute shell commands from the TTCN-3 test suite. It provides abstract service primitives to communicate with the user. The stdin, stdout and stderr of the process are returned to the user.

SCTP

Provides SCTP-type connections in IP networks. It maps between Titan's test port API and Linux kernel's SCTP socket services; supports open socket, close socket, send and receive data. One test port instance can handle multiple SCTP connections in either server or client mode.

HTTP

Allows sending and receiving HTTP messages between the test suite and SUT via a TCP/IP connection. It uses the Abstract Socket library. Both IP version 4 and 6 are supported and it can handle multiple connections.

PCAP 

The PCAP test port has basically two operating modes.

In reading mode, it is able to process recorded network traffic, saved in a file in libpcap format (used e.g. also by the open source Wireshark tool), and decode various application layer protocol messages. These messages are then delivered to TTCN-3.

In capturing mode the test port can be used to capture Ethernet packets to a file, controlled from the TTCN-3 environment.

Filtering of messages can be set both at capturing and reading modes.

LAN

L2 Allows communicating with the SUT at the low level Ethernet connectivity. The test port translates the LANL2 service primitives (ASPs) and messages (PDUs) sent from the TTCN-3 code to Ethernet II frames when sending and translates the received packets to LANL2 ASPs att receipt. The test port uses the Packet Socket on Linux and the DLPI interface on Solaris (using DLIOCRAW mode).

SIP

Handles SIP connections; The test port can handle SIP request and SIP response messages and it can use both UDP and TCP connection to send and receive messages. The built in encoder can encode SIP request and SIP response messages and it is possible to send raw and fragmented [28] messages trough the test port. The name of the header can be encoded in short or long format.

In basic mode the test port can handle only one TCP connection or one UDP socket. It is not possible to send and receive messages using both protocols at the same time, but the test port can switch between protocols and remote hosts.

In advanced mode the test port can handle several TCP connections and listen on both UDP and TCP ports at the same time. Each connection is distinguished by the protocol id, remote host name and the remote port number.

The test port used to be compatible with ETSI's SIP type definitions, therefore ETSI SIP library could be used with it. ETSI has slightly changed one of its type definition lately, therefore the test port needs to be updated to be compatible with ETSI's LibSip v2.0.

Abstract Socket

It is not a test port, but a library that can be used to build test ports that are using the TCP internet protocol connection; it uses the services provided by the UNIX socket  interface and implements basic sending, receiving and socket handling routines. It supports both IPv4 and IPv6, client and server modes and also supports SSLv2, SSLv3 and TLSv1secure connections (the used secure protocol is selected during the SSL handshake automatically).

Protocol Modules

Protocol modules implements static data structure and message encoding/decoding, defined in the specification of the given protocol. Protocols are most often specified by standard bodies for the interfaces, where equipment of different vendors may or need communicate with each other, but of course, interface specifications may also be vendor-specific. Protocol modules can support both cases and shall be in TTCN-3, ASN.1, XSD, or IDL (Corba). JSON support is being developed currently. The protocol modules made available in this project proposal are all defined by IETF and all these specifications are publicly available.

TTCN-3

TTCN-3 is a high-level, abstract language.  The code itself is platform independent (e.g. no integer value range or float precision requirements in the language, these are defined by tool implementations; memory allocation is completely solved by tools).

TTCN-3 is test environment independent. In TTCN-3 only the abstract messages/signals, exchanged between the test system and the tested entity (SUT, AUT, etc.) are defined, the transport layers and connections are provided and handled by the tools. Message/signal encoding (serialization) and decoding (deserialization) is completely done by the tool in the background. The user, if wants, can access the encoded data in TTCN-3, and the encoded data is also logged for debugging purposes. TTCN-3 also allows leaving data open in the source code and providing the actual values at execution time (i.e. typically IP- and other addresses, IDs and passwords, SUT/AUT parameters for automatic test case selection etc.). All this allows executing the test cases, unchanged, on different platforms and in different test environments.

Its main purpose is functional testing (conformance, function, integration verification, end-to-end and network integration testing) , and can also be used for performance testing. It supports testing of message-based (asynchronous), API-based (synchronous) and analog (continuous signals) interfaces and systems.

The TTCN-3 language has four major parts:

A rich data/type system

It allows describing the messages/signals of practically all possible protocols and APIs. 

Except defining data structures in TTCN-3, the language specify the ways of importing other data type/schema languages - as ASN.1, IDL, XSD and currently JSON mapping is being defined, i.e. using them in TTCN-3 test suites without the need of manual conversion. TTCN-3 can also be used as a schema language for XML and JSON (ongoing).

As an extension, Titan also allows adding encoding instructions to TTCN-3 types to automatically encode them in binary (bit-oriented) or (simple) textual forms, XML or JSON (for ASN.1 data BER/CED/DER encodings are supported).

Test configuration

TTCN-3 allows defining multi-process distributed test cases and test execution logic easily (processes are called “test components”). It is the tools' responsibility to deploy and control the test components on the available pool of machines, possibly running different OS-es. The user need not bother about the details (see also the example below). Types of the test components and their interfaces (ports) to other test components and the tested entity are defined in TTCN-3. The number of test component instances and their port connections are controlled from the test case code dynamically, using built-in language statements.

Test dynamic behavior

TTCN-3 is a procedural programming language specialized for testing. This means that is has the usual programming language features and in addition, constructs needed for testing are built-in the language. Sending/receiving messages, checking the content of the incoming messages, alternative test behaviors depending on the response  of the tested entity(including no answer), handling timers and timeouts, logging of events, verdict assignment and collection from the different components of the test system and alike. 

Test execution control

Test case execution logic and dynamic test selection can be controlled from the TTCN-3 code itself.

TTCN-3 is a dynamically developing language. In 2003, at publication of its first implemented version, it consisted of two documents (core language and operational semantics), 316 pages in total. In 2005, in TTCN-3 edition 3, tool implementation parts and interworking with ASN.1 has been added and the language specification was published in 7 documents, on 846 pages in total. The first version of TTCN-3 edition 4, in 2009, added interworking with IDL and XSD that increased the number of published pages to 948 in 8 documents (two earlier parts had been made historical in edition 4) . From 2010 language extensions for real time and performance testing, configuration and deployment, testing of interfaces with continuous signals and some advanced language features are published in separate language extension documents. Up to now 6 language extension packages have been published and today, in June 2014, the whole language specification consists of 1422 pages in 14 documents. ETSI is securing the maintenance and development of the language by establishing a project team from ETSI members. The project is financed from ETSI budget.

Example

The following TTCN-3 code shows a Hello World! example, but it doesn't simply prints out the string.

The control part of the module calls the parameterized test case TC with a string parameter. TC is executed as many times in a loop as many strings are defined in the variable vl_inputs. Each test case uses two threads (called test components): one is created implicitly and automatically, when the control part of the module is started and the  testcase TC is called on this test component (during test case execution this will be the main test component - MTC). The another one is created by TC explicitly, by executing the create statement. This test component is called a parallel test component - PTC. TC then connects the ports of the two test components, commands to start execution of the function f_PTC() on the PTC and sends the string, received as parameter, to it. The MTC then waits until the PTC finishes (the done statement) and returns to the control part.

The PTC first matches the received message against the pattern specified by the TTCN-3 template t_expected. "hello world!" in all small, all capital or small with capital first letters are allowed. Whitespaces are allowed (also at the beginning and end of the string), but there shall be at least one space between "hello and "world". Exclamation mark is optional. If this match successful, the pass verdict, if unsuccessful, the fail verdict is assigned to the test case and the reason is logged (expected/not the expected message has been received plus the message itself).

module hello_world {

//====================== Port Types ======================

type port MYPORT message {

    inout charstring

} with {extension "internal"}

//====================== Component Types ======================

type component MYCOMP {

    port MYPORT myport

}

//====================== Constants ======================

const charstring ws0 := "(\n| )#(0,)"

//====================== Templates ======================

// Strings containing the words "hello" and "world", in all small, or

// all capital letters, or small letters with first letter capital;

// exclamation mark is optional; whitespaces allowed

template charstring t_expected :=

    pattern "{ws0}(((h|H)ello {ws0}(w|W)orld)|HELLO {ws0}WORLD){ws0}!#(0,1){ws0}"

//====================== Functions ======================

function f_PTC() runs on MYCOMP {

    alt {

        [] myport.receive(t_expected) {

 setverdict(pass,"expected message received: ",t_expected) }

                [] myport.receive { setverdict(fail,"not the expected message received: ",t_expected) }

    }

}

//====================== Testcases ======================

testcase TC(charstring pl_toSend) runs on MYCOMP {

    var MYCOMP vl_PTC := MYCOMP.create;

    connect (self:myport, vl_PTC:myport); vl_PTC.start(f_PTC());

    myport.send(pl_toSend); vl_PTC.done

}

//=========================================================================

// Control Part

//=========================================================================

control {

    var charstring vl_inputs [6] :=

        {"HELLO WORLD!","hello world","Hello World!","hello WORD!","hELLO wORLD!","helloworld!"}

    var integer i, vl_noStrings := sizeof(vl_inputs);

    for (i:=0; i< vl_noStrings; i:=i+1) {

        execute (TC(vl_inputs[i]))

    }

} // end of control

} // end of module

 

Titan's screenshot, showing the results of the control part's execution is given below. The test case TC has been executed 6 times with different parameters. 3 times execution resulted pass 3 times fail verdict. On the left window at the middle row, the generated log file is shown. Test case results are extracted automatically from it and by clicking on a test case, its log is opened either in a graphical or in a textual/tabular form. In the middle window of the upper row the graphical view the fragment of the last execution of TC can be seen, when the string is received by the PTC and the fail verdict is assigned. By clicking on the "charstring", its content is opening in a value window that can be seen in the window below the graphical log window and at the same time the source code line producing the log event is highlighted in the TTCN-3 editor (rigth window of the middle row). The left window of the top row allows selecting the test cases/control part to be executed and shows the actual status of test components during test case runs.

Resources

Titan datasheet

Titan, the advent of an industrial-strength test system for multiple domains @EclipseCon Europe 2014 (presentation slides) (presentation video)

Eclipse Titan command line  installation

Eclipse Titan command line basic workflow

Eclipse Titan course material part1  

Eclipse Titan course material part 2  

TTCN-3 tutorials

TTCN-3 standards

TTCN-3 Quick Reference Guide

TTCN-3 Quick Reference Poster

TTCN-3 test suites and libraries:

       3GPP and ETSI test suites

       ETSI libraries

       ETSI examples

Please note that Titan documentation is available in the /doc directory of the downloadable packages.

Other useful resources

   Eclipse IoT-Testware project

   oneM2M Tester

   How To Test Your Projects With Titan link link (by Gustavo Gonnet)

Papers, video materials

   Advanced TTCN-3 Test Suite validation with Titan (scientific paper)

   TTCN-3 and Eclipse TITAN for testing protocol stacks (by Harald Welte)

   Installing TTCN3 on Windows Using Eclipse Titan Project (video)

   TTCN-3 and Eclipse TITAN for testing protocol stacks (presentation by Harald Welte at NETDEV 11/2017 and EasterHegg 2018)