L'architecture générale de test d'interopérabilité comprend un système sous test (SUT) composé de plusieurs implémentations sous test (IUT). En général, les méthodes de test d'interopérabilité peuvent être classées en deux grandes approches : les tests actifs et les tests passifs.
Background
Typical functional tests include conformance testing and interoperability testing, etc., of which interoperability testing is the subject of this thesis. For these reasons, we believe that passive testing is worth investigating, and we consider using it for interoperability testing in this thesis.
Scope, Motivations and Objectives
Compared to active testing, the main advantage of passive testing is its non-intrusive nature. Later on, passive testing is further applied to other fields such as network security [39], communication protocol testing [4, 5], etc.
Organization of the Thesis
- The Needs of Interoperability Testing
- Different Testing Techniques
- Testing Techniques According to Accessibility
- Testing Techniques According to Controllability . 17
- Definition of Interoperability
- Interoperability Testing Architectures
- Interoperability Criteria
- Interoperability Testing Process
- Preliminary: Compatibility of Specifications
- Interoperability Testing Activities
The general definition of interoperability is "the ability of two or more applications to communicate using a specific mechanism in a known environment in order to achieve user objectives". Specification of the System Under Test SUT, which may consist of one or more implementations.
State of the Art of Interoperability Testing
Active Interoperability Testing
- Active Interoperability Testing Overview
- Advantages and Drawbacks of Active Testing
Most of the recent research work in the field is related to the derivation of interoperability test suites. Test cases can be, for example, a specific error type or an important state of the specification.
Passive Interoperability Testing
- Passive Testing Techniques
- Advantages and Challenges of Passive Testing
- Conclusions
The trace matching approach is a passive testing technique that strictly compares each event in the trace produced by the SUT against that in its specification. The main difference between active testing and passive testing is that in active testing the test system is aware of the SUT states during the test.
Conclusions
In Chapter 2, we compared active testing and passive testing and drew a conclusion that passive testing is an appropriate technique to perform interoperability testing. Arguing for the passive testing technique, this chapter presents a specification-based methodology for passive interoperability testing.
Testing Architecture
The application of the proposed approach and the experimental results are shown in Section 3.5.
Formal Model
Similarly, In(q) is the set of possible inputs and Γ(q) is the set of all possible events in state q. By extension, Out(M, σ) is the set of executable events of system M following the trace σ. For example, the projection of M onto the set of feasible events onto its lower (or upper) interfaces ΣML (or ΣMU) is denoted by M/ΣML (or M/ΣMU).
Similarly, OutX(M,σ) corresponds to a projection of the series of outputs (M,σ) onto the series of events X.
Passive Interoperability Testing Method
Passive Interoperability Testing Method Overview
- Formalizing Interoperability Test Purpose
- Interoperability Test Case Generation
- Passive Interoperability Test Case Derivation
Finally, a trace analysis algorithm is proposed to evaluate the behavior of the IUTs relative to the ITP. The inputs to the algorithm are the specifications (S1 and S2) on which the IUTs are based, and an interoperability test target (ITP). Normally, the behavior of a SUT can be obtained by calculating the global behavior of the IUTs.
This result can be infinite if the size of the input FIFO queues is unbounded.
Trace Verification
Then, for each event a received in sequence by σ, we check whether a can be accepted by the states in the State_under_read array. Furthermore, we require that the initial state is always in the State_under_reading list. Inc_reached == True means that the trace includes a branch which is associated with the Inconclusive attribute in the PITC graph.
Same, the for loop will be executed N times where N is the number of states in States_under_reading.
Verdict Assignment
The transposition of this way of determining the decision in the passive test is trivial if the behavior of the IUTs is as expected. Therefore, the observed behavior of IUTs may be allowed by the specifications, but does not correspond to the test case. Recorded traces may be incomplete: If the traffic capture starts after the start point of the PITC, then only the end of the PITC may be visible in the trace.
Similarly, if the traffic capture is terminated before the end of the PITC, then only the start of the PITC can be observed.
Application on SIP Protocol
SIP Protocol Overview
In Figure 3.3-(c), the EchoRequest sent by the IU TB is fragmented, but only one fragment is received. If the user decides to hang up before the connection is established, the CMC cancels the invitation request (L1!Cancel). If the user decides to hang up after the connection is established (L1!Ack has been sent), the CMC must send an aBye request to the CMS (L1!Bye).
Figure 3.6 illustrates the ITC built with Algorithm 3.1: The interaction of S1 and S2 to reach the ITP is calculated.
Test Execution
Short trace length: due to the fact that the trace record was truncated before the completion of the PITC Inconclusive attributes. Inconclusive verdicts are reached: due to the fact that the observed behavior does not correspond to the passive test case, it is nevertheless allowed in the specifications. After Jtisi sends the Bye request, instead of an Ack_Bye, the Ekiga Siphone responds 481 Call/Leg Transition.
Conclusions
Therefore, in the following we will try to find another solution to perform passive interoperability testing. Due to the heterogeneous nature of distributed systems, the interoperability of these protocol applications becomes a crucial issue. This chapter presents a methodology for the interoperability testing of request-response protocols using the technique of passive testing.
Furthermore, in this chapter, we will present a test tool developed for automating the passive interaction testing tool.
Background and Motivation
To verify that the test goals are met, traces produced by protocol implementations are filtered and divided into a series of conversations related to the special request-response interaction model of request-response protocols. Promoted by the rapid development of computer technology, more and more protocols using the client-server request-response are on the rise. To ensure they work well together and therefore meet customer expectations, interoperability testing is an important step in validating request-response protocol implementations before they go to market.
In this context, in this chapter we will provide a passive interoperability testing methodology for request-response protocols.
Passive Interoperability Testing Method for Request-response Pro-
Testing Method Overview
Test objective is a commonly used method in the field of testing to focus on the most important properties of a protocol, since it is generally impossible to validate all possible behaviors described in specifications. Note that in this chapter we only focus on the desired property to be verified. On the contrary, in passive testing, ITCs are only used to analyze the observed trace produced by the IUTs.
In this chapter we choose to use the offline testing method, where the test cases are precomputed before they are executed on the trace.
Trace Verification
If so, the corresponding decision is issued in the corresponding test case. The global verdict for each test case is issued taking into account all its subdecisions registered judgment.T Ci. Additionally, the possibility that a test case may appear multiple times in the trace is also considered.
Therefore, the global verdict for a given test case is based on a set of sub-judgments, which increases the reliability of interoperability testing.
A Passive Interoperability Testing Tool
Motivation
- TTCN-3 Overview
- Main Issues
To automate testing, TTCN (Testing and Test Control Notation)2 is widely used in the testing community. The standardized specification of the language and its environment is included in the ETSI standard [57]. Within an alt block, each alternative is "tried" in the order of its syntactic appearance.
According to the pragmatics of active testing, within an alt block there is usually a list of branches in which the expected response is received and processed (ie, the receive operations for alternative SUT responses and the timeout operations indicating non-response).
Ttproto: A Testing Tool for Passive Interoperability Testing 71
- Description
In addition, TTCN-3 has other disadvantages, especially in terms of template propagation, which make it "inconvenient" in practical use: for example, it needs two different sets of templates to send and receive messages; parameterization and reuse are not easy, resulting in tedious code duplication; etc. These biases and patterns strongly influence the perception of testing needs, which can be challenged in the context of passive testing. In this way, ttproto is actually a device for empirically investigating the behavior of systems, which is not bound by the biases that are part of existing TTCN systems, and thus can be used for passive testing.
In ttproto, either a raw byte string or binary string representing the address can be used.
Application to the CoAP Protocol
CoAP Interoperability Testing Event
The next step is trace verification, which is performed by taking in two files as input: the set of test cases and the filtered trace. The trace is analyzed according to Algorithm 4.1, where the test cases are verified in the trace to check their occurrence and validity. Finally, unrelated test cases are filtered out, while other test cases are associated with a Pass, Fail, or Inconclusive verdict.
As one of the most important protocols for the future IoT, the number of smart objects using CoAP is expected to grow rapidly.
CoAP Protocol
- CoAP Protocol Overview
- Test Purposes Selection and Test Cases Generation 79
- CoAP Plugtest Overview and Testing Architec-
- Test Execution with Ttproto
- Results of the CoAP Plugtest
To ensure that ITPs are correct to specification, ITPs have been selected and cross-validated by experts from ETSI14, IRISA15 and BUPT16. Upon receiving the request, the server validates the message and downloads the content while echoing the message ID generated by the client. After receiving the request, the server divides the resource into 4 blocks and transfers them separately to the client.
In addition, the passive testing tool shows its ability to detect interoperability (cf. the second column of Table 4.1): among all tests, 5.9% of them show interoperability against basic RESTful methods; 7.8%.
Conclusions
Since IoT implies the provision of services in lossless networks, we also consider fundamental CoAP implementation interoperability tests in lossy context. Loss context is frequently encountered in operational networks, however neglected in most of the current works. In fact, due to the uncontrollable nature of passive testing, inconclusive judgments are often made, requiring expert repeat testing or post-analysis.
Future work will consider online trace verification to monitor the network for a long time and report abnormalities at any time.
Summary of Contributions
To solve these problems, in the rest of the thesis, we have proposed several contributions. Also in this chapter judgment transfer was discussed which must be carefully addressed in passive interoperability testing. In Chapter 4, another method of passive interoperability testing is proposed, especially for request-response protocols in connection with client-server communication.
To solve this problem, we consider the loss factor during the interoperability test event.
Future Work
Improve Trace Verification by Test Case Grouping
- Motivation
- State of the Art
- Grouping Passive Interoperability Test Cases
- Trace Verification
- Results
After all test cases are processed, we can obtain a global passive iop test case P IT CG. The constructed global passive iop test case P IT CG fully represents the transitions of each PITC in a structural hierarchy. Another goal of constructing the global passive iop test case P IT CG is to support parallel trace verification.
The global P IT CG aims to integrate all states and transitions into individual PITCs.
Perspectives
Nanxing Chen, César Viho: Passive Interoperability Testing for Request-Response Protocols: Method, Tool, and Application to the CoAP Protocol. Nanxing Chen, César Viho: A passive interoperability testing approach applied to the Constrained Application Protocol (CoAP). Aggarwal and K. Sabnani, editors, Proceedings of the Eighth International Conference on Protocol Specification, Testing and Protocol Verification, pages 63-74, North Holland, 1988.
From Conformance Testing to Interoperability Testing, in: Proceedings of the 3rd International Workshop on Protocol Test System, 1990.
