Abstract Test Suites Development - A standard approach

INTRODUCTION

Over the years, the size and complexity of software in multi-process, distributed architecture systems running in different environment have grown dramatically. Verification & Validation has become an all-important activity in this context. The terms conformance and interoperability are both important characteristics for implementations. An implementation conforms if it meets all of the mandatory requirements of the written specification fully, and also meets those conditional requirements that it claims to meet. Conforming implementations have high probabilities of interoperating with other implementations conforming to the same specification.

Test suites formalize the means with which to establish conformance or interoperability. Test suites come in two forms: abstract and executable. A test suite comprises multiple test cases each designed to test a single requirement or option. An abstract test suite is like un-compiled source code. ISO (International Standards Organization) and ITU-T, while addressing this issue, suggest the usage of TTCN (Tree and tabular combined notation), which is test-equipment-independent. The TTCN language is part of the ISO/IEC 9646 (International Electro-technical Committee) conformance-testing framework and is specially designed for the specification tests of communication systems.

We will discuss standard approach for abstract test suite development. Definition of key concepts associated with the Abstract Test Suites. It also describes about form of abstract test suites and maintenance of ATS (Abstract Test Suite).

TTCN is the defacto standard test environment/Language for communication systems. This is used world wide to test telecommunication and data-communication equipment ranging from built in communication chips to huge switched and intelligent network services. TTCN is widely used for conformance testing. Conformance testing is the process of verifying that an implementation performs in accordance with a particular standard/specification. It is concerned with external behavior (black box) but not with performance/reliability/fault tolerance.

ISO/IEC 9646 (ITU X.290 series) is a seven part standard which defines a framework and methodology for conformance testing of implementations of OSI and ITU protocols. The TTCN is the third part of this standard, i.e. ISO/IEC 9646-3. The Versions 1 and 2 are developed by ISO as part of the widely used ISO/IEC-9646 conformance-testing standard. The ISO/IEC 9646-3 and ITU-T X.292 Updates/maintenance are done by ETSI. The Version 3 is developed by ETSI.

Conformance testing
Conformance testing is verification that an implementation meets the formal requirements of the referenced standards and, more precisely, that it meets the conformance clauses contained in the standards. During the test phase, the implementation is referred to as the Implementation Under Test (IUT). The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate. The testing of performance and robustness are not part of the conformance testing process. No amount of testing can give a full guarantee of successful inter-working. The exhaustive testing of every possible aspect of protocol behavior is unrealistic and impractical for technical and economical reasons.

Conformance testing can however give a reasonable degree of confidence that an implementation, which passes the tests, will comply with the requirements in its communication with other systems. As such, conformance testing can be regarded as a prerequisite for inter-working. Any test can be easily contentious. When comparing a product with its specification, using testing tools, we could ask, in case of discrepancies: Is the product wrong? Is the specification ambiguous? Is the test biased? Is the testing method suitable? Is the testing process agreed and understood?

One-way to solve some of these questions in advance is to standardize: Use of standard protocol specifications, use of standard tests, use of standard methods, use of standard testing process. This is what Conformance Testing is about, based upon a standard testing methodology, as defined by ISO and CCITT. The benefits of conformance testing can be increased further. The use of standard methods, based on approved test suites developed for each OSI standard protocol, and on testing procedures, lead to the comparability of results produced by different testers, and thereby to the mutual recognition of test reports. This will minimize the need for repeated conformance testing, and minimize the associated costs.