Dynamic Analysis of Safety Critical Systems

From the desk of
Sandeep Patalay

 Background: The dynamic analysis of the safety critical software is an important phase of the Independent Verification and Validation of the system. The EN 50128 has detailed the methods that shall be used for this phase of the verification life cycle. This phase is so critical to the project output that it demands meticulous planning and organization. Here we discuss the dynamic analysis methods suggested by the CENELEC standards for SIL 4 software.

 Boundary Value Analysis

The aim of this method is to remove software errors occurring at parameter limits or boundaries. The input domain of the program is divided into a number of input classes. The tests should cover the boundaries and extremes of the classes. The tests check that the boundaries in the input domain of the specification coincide with those in the program. The use of the value zero, in a direct as well as in an indirect translation, is often error-prone and demands special attention:

• Zero divisor;
• Blank ASCII characters;
• Empty stack or list element;
• Null matrix;
• Zero table entry.

Normally the boundaries for input have a direct correspondence to the boundaries for the output range. Test cases should be written to force the output to its limited values. Consider also, if it is possible to specify a test case which causes output to exceed the specification boundary values. If output is a sequence of data, for example a printed table, special attention should be paid to the first and the last elements and to lists containing none, 1 and 2 elements.

 Error Guessing

The aim of this method is to remove common programming errors. Testing experience and intuition combined with knowledge and curiosity about the system under test may add some uncategorised test cases to the designed test case set. Special values or combinations of values may be error-prone. Some interesting test cases may be derived from inspection checklists. It may also be considered whether the system is robust enough. Can the buttons be pushed on the front-panel too fast or too often? What happens if two buttons are pushed simultaneously?

Error Seeding

The aim of this method is to ascertain whether a set of test cases is adequate. Some known error types are inserted in the program, and the program is executed with the test cases under test conditions. If only some of the seeded errors are found, the test case set is not adequate. The ratio of found seeded errors to the total number of seeded errors is an estimate of the ratio of found real errors to total number errors. This gives a possibility of estimating the number of remaining errors and thereby the remaining test effort.. The detection of all the seeded errors may indicate either that the test case set is adequate, or that the seeded errors were too easy to find. The limitations of the method are that in order, to obtain any usable results, the error types as well as the seeding positions must reflect the statistical distribution of real errors.

Performance Modelling

The aim of the method is to ensure that the working capacity of the system is sufficient to meet the specified requirements. The requirements specification includes throughput and response requirements for specific functions, perhaps combined with constraints on the use of total system resources. The proposed system design is compared against the stated requirements by:

• Defining a model of the system processes, and their interactions,
• Identifying the use of resources by each process, for example, processor time, communications bandwidth, storage devices etc),
• Identifying the distribution of demands placed upon the system under average and worst-case conditions,
• Computing the mean and worst-case throughput and response times for the individual system functions.

For simple systems, an analytic solution may be possible whilst for more complex systems, some form of simulation is required to obtain accurate results. Before detailed modelling, a simpler ’resource budget’ check can be used which sums the resources

requirements of all the processes. If the requirements exceed designed system capacity, the design is infeasible. Even if the design passes this check, performance modelling may show that excessive delays and response times occur due to resource starvation. To avoid this situation engineers often design systems to use some fraction (e.g. 50 %) of the total resources so that the probability of resource starvation is reduced

Equivalence Classes and Input Partition Testing

The aim of this method is to test the software adequately using a minimum of test data. The test data is obtained by selecting the partitions of the input domain required to exercise the software. This testing strategy is based on the equivalence relation of the inputs, which determines a partition of the input domain.

Test cases are selected with the aim of covering all subsets of this partition. At least one test case is taken from each equivalence class. There are two basic possibilities for input partitioning which are:

• Equivalence classes may be defined on the specification. The interpretation of the specification may be either input oriented, for example the values selected are treated in the same way or output oriented, for example the set of values leading to the same functional result; and
• Equivalence classes may be defined on the internal structure of the program. In this case the equivalence class results are determined from static analysis of the program, for example the set of values leading to the same path being executed

Structure Based Testing

The aim of this method to apply tests which exercise certain subsets of the program structure. Based on an analysis of the program a set of input data is chosen such that a large fraction of selected program elements are exercised. The program elements exercised can vary depending upon the level of rigour required.

• Statements: This is the least rigorous test since it is possible to execute all code statements without exercising both branches of a conditional statement.
• Branches: Both sides of every branch should be checked. This may be impractical for some types of defensive code.
• Compound Conditions: Every condition in a compound conditional branch (i.e. linked by AND/OR is exercised).
• LCSAJ: A linear code sequence and jump is any linear sequence of code statements including conditional jumps terminated by a jump. Many potential sub-paths will be infeasible due to constraints on the input data imposed by the execution of earlier code.
• Data Flow: The execution paths are selected on the basis of data usage for example a path where the same variable is both written and wrote.
• Call Graph: A program is composed of subroutines which may be invoked from other subroutines. The call graph is the tree of subroutine invocations in the program. Tests are designed to cover all invocations in the tree.
• Entire Path: Execute all possible path through the code. Complete testing is normally infeasible due to the very large number of potential paths.