SIST-V ETSI/EG 202 237 V1.2.1:2016

Final draft ETSI EG 202 237 V1.2.1 (2010-06)
ETSI Guide


Methods for Testing and Specification (MTS);
Internet Protocol Testing (IPT);
Generic approach to interoperability testing

---------------------- Page: 1 ----------------------
2 Final draft ETSI EG 202 237 V1.2.1 (2010-06)



Reference
REG/MTS-00129-IOP-TstApproach
Keywords
interoperability, IP, telephony, testing
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: +33 4 92 94 42 00  Fax: +33 4 93 65 47 16

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2010.
All rights reserved.

TM TM TM TM
DECT , PLUGTESTS , UMTS , TIPHON , the TIPHON logo and the ETSI logo are Trade Marks of ETSI registered
for the benefit of its Members.
TM
3GPP is a Trade Mark of ETSI registered for the benefit of its Members and of the 3GPP Organizational Partners.
LTE™ is a Trade Mark of ETSI currently being registered
for the benefit of its Members and of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI

---------------------- Page: 2 ----------------------
3 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Contents
Intellectual Property Rights . 6
Foreword . 6
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions and abbreviations . 8
3.1 Definitions . 8
3.2 Abbreviations . 9
4 Types of testing . 9
4.1 Interoperability testing . 9
4.2 Conformance testing. 10
4.3 Combining interoperability testing and conformance testing . 11
5 Interoperability testing process overview . 11
6 Basic concepts . 12
6.1 Means of Testing . 12
6.2 Equipment Under Test (EUT) . 13
6.3 Qualified Equipment (QE) . 13
6.3.1 QEs and Devices . 13
6.3.2 Designating the first QE . 13
6.4 System Under Test (SUT) . 13
6.5 Test interface . 14
6.6 Test driver . 14
6.7 Test coordinator . 14
6.8 Interoperability test cases . 14
6.9 Means of Communication . 15
7 Generic interoperability test architectures . 15
7.1 Test architectures with a single QE . 15
7.2 Test architectures with multiple QEs. 16
7.2.1 An example using 3 QEs . 16
7.2.2 Testing IP hosts with multiple QEs . 17
8 Developing interoperability tests . 18
8.1 Overview . 18
8.2 Specify abstract architecture. 19
8.3 Prepare draft IFS Proforma . 19
8.4 Specify Test Suite Structure . 20
8.4.1 Identify test groups . 20
8.4.2 Define test coverage within each test group . 20
8.5 Write Test Purposes. 21
8.6 Write test cases . 21
8.6.1 Pre-test conditions . 21
8.6.2 Test steps and verdicts . 22
8.6.2.1 Test steps . 22
8.6.2.2 Verdicts . 22
8.6.2.3 Specification of test steps and verdicts. 22
8.6.3 Example . 23
8.6.4 Pre-amble and post-amble . 24
8.6.4.1 Alternative test case presentation forms . 25
8.7 Validate test cases . 26
8.8 Finalize IFS . 26
9 Interoperability testing process . 26
ETSI

---------------------- Page: 3 ----------------------
4 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
9.1 Overview . 26
9.2 Prepare for testing . 27
9.2.1 Test arrangement . 27
9.2.2 Test planning . 28
9.3 Testing . 29
9.3.1 Manual testing . 29
9.3.2 Automated testing . 29
9.4 Write test report . 29
Annex A: Example IFS (Internet Key Exchange protocol, IKEv2) . 30
A.1 Introduction . 30
A.2 Instructions for completing the IFS proforma . 30
A.2.1 General structure of the IFS proforma . 30
A.2.2 Additional information . 30
A.3 IFS proforma . 31
A.3.1 Implementation identification . 31
A.3.2 Protocol Summary, RFC 4306 . 31
A.4 IKEv2 entities . 31
A.4.1 Roles . 31
A.4.2 IKEv2 Initiator functions . 32
A.4.2.1 IKE exchange types . 32
A.4.2.1.1 IKE SA establishment functions . 32
A.4.2.1.2 Child SA establishment functions . 32
A.4.2.1.3 INFORMATIONAL exchange functions . 33
A.4.3 IKEv2 Responder functions . 33
A.4.3.1 IKE exchange types . 33
A.4.3.1.1 IKE SA establishment functions . 33
A.4.3.1.2 Child SA establishment functions . 34
A.4.3.1.3 INFORMATIONAL exchange functions . 34
Annex B: Example IFS (TIPHON Profile of SIP, Release 3) . 35
B.1 Introduction . 35
B.2 Instructions for completing the IFS proforma . 35
B.2.1 General structure of the IFS proforma . 35
B.2.2 Additional information . 35
B.3 IFS proforma . 36
B.3.1 Implementation identification . 36
B.3.2 Protocol Summary, EN 301 xxx . 36
B.4 SIP entities . 36
B.4.1 Roles . 36
B.4.2 User Agent capabilities . 37
B.4.2.1 Registration . 37
B.4.2.2 Basic call . 37
B.4.3 Registrar capabilities . 38
B.4.3.1 Registration . 38
B.4.4 Proxy capabilities . 38
B.4.4.1 Proxy in the serving and intermediate network. 38
B.4.4.1.1 Registration . 38
B.4.4.1.2 Basic call . 38
B.4.4.2 Proxy in the home network . 39
B.4.4.2.1 Registration . 39
B.4.4.2.2 Basic call . 39
B.4.5 Gateway capabilities. 39
B.4.5.1 Basic call . 39
ETSI

---------------------- Page: 4 ----------------------
5 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Annex C: Bibliography . 40
History . 41

ETSI

---------------------- Page: 5 ----------------------
6 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://webapp.etsi.org/IPR/home.asp).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Methods for Testing and Specification (MTS),
and is now submitted for the ETSI standards Membership Approval Procedure.
ETSI

---------------------- Page: 6 ----------------------
7 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
1 Scope
The present document, "A generic approach to interoperability testing", gives general guidance on the specification and
execution of interoperability tests for communication systems specifically in the context of product certification. It
provides a framework within which interoperability test specifications for a wide range of product types can be
developed. The guidelines are expressed as recommendations rather than strict rules and leave enough freedom to allow
test specifiers to adopt and adapt processes to suit each particular project while still ensuring that test specifications
accurately reflect the requirements of the base standards and can be executed consistently across a range of
configurations.
Interoperability testing is the structured and formal testing of functions supported remotely by two or more items of
equipment communicating by means of standardized protocols. It is not the detailed verification of protocol
requirements specified in a conformance test suite, neither is it the less formal development testing often associated
with "plug-fest" and "interop" events (frequently referred to as "bake-offs"). A methodology for the latter type of testing
is described in EG 202 810 [i.8].
Although some consideration is given within the methodology to the operating and reporting aspects of interoperability
testing, the primary focus of the present document is on the specification of interoperability testing architectures, test
plans and test suites.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI ES 201 873-1: "Methods for Testing and Specification (MTS); The Testing and Test Control
Notation version 3; Part 1: TTCN-3 Core Language".
[i.2] Void.
[i.3] ETSI TS 101 884: "Telecommunications and Internet Protocol Harmonization Over Networks
(TIPHON) Release 3; Technology Mapping; Implementation of TIPHON architecture using SIP".
[i.4] ETSI EG 202 107: "Methods for Testing and Specification (MTS); Planning for validation and
testing in the standards-making process".
[i.5] ISO/IEC 9646 (parts 1 to 7): "Information technology - Open Systems Interconnection -
Conformance testing methodology and framework".
[i.6] IETF RFC 3261: "SIP: Session Initiation Protocol".
ETSI

---------------------- Page: 7 ----------------------
8 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
[i.7] IETF RFC 4306: "Internet Key Exchange (IKEv2) Protocol".
[i.8] ETSI EG 202 810: "Methods for Testing and Specification (MTS); Automated Interoperability
Testing; Methodology and Framework".
[i.9] ETSI ES 202 553: "Methods for testing and Specification (MTS); TPLan: A notation for
expressing test Purposes".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
conformance: compliance with requirements specified in applicable standards ISO/IEC 9646 [i.5]
conformance testing: testing the extent to which an Implementation Under Test (IUT) satisfies both static and dynamic
conformance requirements ISO/IEC 9646 [i.5]
NOTE: The purpose of conformance testing is to determine to what extent a single implementation of a particular
standard conforms to the individual requirements of that standard.
device: item of software or hardware which either alone or in combination with other devices implements the
requirements of a standardized specification
Equipment Under Test (EUT): grouping of one or more devices which has not been previously shown to interoperate
with previously Qualified Equipment (QE)
Implementation Under Test (IUT): an implementation of one or more Open Systems Interconnection (OSI) protocols
in an adjacent user/provider relationship, being the part of a real open system which is to be studied by testing
(ISO/IEC 9646-1 [i.5])
interoperability: ability of two systems to interoperate using the same communication protocol
interoperability testing: activity of proving that end-to-end functionality between (at least) two communicating
systems is as required by the base standard(s) on which those systems are based
interoperability test suite: collection of test cases designed to prove the ability of two (or more) systems to
interoperate
InterWorking Function (IWF): translation of one protocol into another one so that two systems using two different
communication protocols are able to interoperate
Qualified Equipment (QE): grouping of one or more devices that has been shown and certified, by rigorous and
well-defined testing, to interoperate with other equipment
NOTE 1: Once an EUT has been successfully tested against a QE, it may be considered to be a QE, itself.
NOTE 2: Once a QE is modified, it loses its status as QE and becomes again an EUT.
System Under Test (SUT): one or more QEs and an EUT
test case: specification of the actions required to achieve a specific test purpose, starting in a stable testing state, ending
in a stable testing state and defined in either natural language for manual operation or in a machine-readable language
(such as TTCN-3) for automatic execution
test purpose: description of a well-defined objective of testing, focussing on a single interoperability requirement or a
set of related interoperability requirements
ETSI

---------------------- Page: 8 ----------------------
9 Final draft ETSI EG 202 237 V1.2.1 (2010-06)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
EP End Point
EUT Equipment Under Test
GFT Graphical presentation Format for TTCN-3
IFS Interoperable Features Statement
IUT Implementation Under Test
IWF InterWorking Function
MMI Man-Machine Interface
MoC Means of Communication
MoT Means of Testing
OSI Open Systems Interconnection
PICS Protocol Implementation Conformance Statement
QE Qualified Equipment
SIP Session Initiation Protocol
SUT System Under Test
TP Test Purpose
TSS Test Suite Structure
4 Types of testing
Equipment implementing standardized protocols and services can be formally tested in two related but different ways,
which each have benefits and limitations:
• conformance testing can show that a product correctly implements a particular standardized protocol:
- establishes whether or not the implementation in question meets the requirements specified for the
protocol itself. For example, it will test protocol message contents and format as well as the permitted
sequences of messages.
• interoperability testing can demonstrate that a product will work with other like products:
- assesses the ability of the implementation to support the required trans-network functionality between
itself and another, similar implementation to which it is connected.
Conformance testing in conjunction with interoperability testing provides both the proof of conformance and the
guarantee of interoperation.
4.1 Interoperability testing
The term "interoperability testing" is often used in relation to the semi-formal testing carried out at multi-vendor events
as part of the product development process. While such events, often referred to as "plug-fests", "interops" and
"bake-offs", are valuable sources of information on the ability of similar products to communicate, they generally do
not offer the structured and, therefore, repeatable, testing that is an essential part of a certification scheme. For a
certification (or branding or logo) scheme to be meaningful, it is necessary that interoperability testing is carried out in
accordance with a comprehensive and structured suite of tests. In the context of the present document, it is exactly this
type of testing which is referred to as "interoperability testing". For other types of schemes, such as those arranged
between manufacturers for marketing or other purposes this approach is still valid.
NOTE: It is possible that other organizations within the global standardization community will have
interpretations of this term which differ to a greater or lesser extent.
The purpose of interoperability testing is to prove that end-to-end functionality between (at least) two communicating
systems is as required by the standard(s) on which those systems are based.
E
...