Challenges in Independent Verification and Validation of Safety Critical Railway Signalling Systems

From The Desk of

Sandeep Patalay

Challenges in Independent Verification and Validation of Safety Critical Railway Signalling Systems

Sandeep Patalay[1]

CMC Americas, Inc., Pittsburgh, PA, 15220

A railway signalling system is safety critical system that controls the traffic which includes train routes, shunting moves and the movements of all other railway vehicles in accordance with railway rules, regulations and technological processes required for the operation of the railway system. The overall signalling system consists of Microprocessor based Wayside controllers, On-board systems controlling the railway vehicle and supervision systems to monitor the vehicle movements from a centralized location. The complex nature of railway signalling rules and operational practices adopted by different railroads pose a difficult task for the software development of these systems. The complex nature of the software poses an even more challenging task during the Independent Verification and Validation of the system. The CENELEC set of standards is the widely accepted as the governing standard for design, development and Independent Verification and Validation (IV and V) of railway signalling systems. This paper describes the challenges faced during the different phases of IV and V of safety critical railway signalling software which is unique compared to other domains.

Nomenclature

IV and V               =    Independent Verification and Validation

ATP                  =    Automatic Train Protection

On-Board       =    Embedded systems used on the Train

CENELEC      =    European standards for Railway Signalling

   I.      Introduction

IV and V is the most important phase of any safety critical system life cycle. The result of this phase decides the final outcome of the project and decides whether the product is fit for use. The IV and V of safety critical software for railway signalling applications is faced with many challenges due to the complexity of the systems and the variations it has depending upon the geography and environment in which it needs to operate. This paper particularly focuses on the experiences and challenges during different phases of the IV and V in a railway signalling project. The following areas will be discussed:

1)          Systematic Problems

2)          Challenges during Software Analysis

3)          Challenges during System Integration and Field Validation Testing

4)          Challenges during Test Result Analysis

II.      Systematic Problems during IV and V of Railway Signalling Software

The following systematic problems are experienced during the IV and V of safety critical software developed for railway signalling applications:

5)          Lack of formal methods in developing the control algorithms results in poor understanding of the system by a test engineer.

6)          Lack of domain knowledge in railway signalling systems creates a technological gap between the software test engineers and the domain consultants. This leads to errors in software testing, which might lead to unsafe failures being un-detected.

7)          Since the software and hardware is so complex, complete test of the system is not possible and most of the faults are revealed at the field Installation stage or during normal working of the system in field.

8)          The software is often changed for every geographical location and results in specific code for each location. When the software structure is not in a generic form, it becomes difficult for the test engineer to develop test cases for every possible scenario.

9)          The lack of standardization in the railway working principles results in incomplete test cases as test engineers are not well versed with all types of railroads.

10)       Increase in the complexity of the software leads to difficulty in testing, since most of the railway systems are sequential machines they are error prone and are very difficult to test.

 III.      Challenges during Software Analysis

The following section describes the challenges faced during Static and Dynamic analysis of safety critical software developed for railway signalling applications:

1)          Static Analysis of software is the analysis of the software code without actually executing it. The railway signalling software particularly the vehicle braking algorithms are very complex and require the test engineer to be well versed with the dynamics of the vehicle as well as good mathematical knowledge. These algorithms require the test engineer to envisage all the possible states of the algorithms and then create a formal model of the system. In many cases the lack of Test Engineer’s knowledge about these algorithms results in insufficient test cases for the model and results in many of the errors revealing in the latter part of the dynamic analysis.

2)          Dynamic Analysis of software is the analysis of the software code by actually executing the software and observing the executions. The dynamic analysis of the safety critical software is an important phase of the Independent Verification and Validation of the system. The Test engineer should be well versed with the domain of inputs and outputs of the system. In many cases the test engineer chooses the boundary values based on the data range of the variable type. In the real world the boundary values depends on the actual working environment of the system, for example the GPS signal boundary values received by the vehicle to determine its position varies based on the geographical location of the railroad and is embedded in the vehicle database. When the test engineer creates test cases for this part of the software it should be changed based on the geographical location where the train is operating, instead the literal boundary value of the variable type would pass the dynamic analysis and this error would only be revealed during field validation tests.

3)          Inexperienced test engineers just follow the rule book and often result in insufficient test scenarios. Testing experience and intuition combined with knowledge and curiosity about the system under test may add some uncategorized test cases to the designed test case set. Special values or combinations of values may be error-prone. Some interesting test cases may be derived from inspection checklists.

4)          Test engineers generally are not well versed with the concept of error seeding and do not try to measure the effectiveness of their test cases. Some known error types should be inserted in the program, and the program should be executed with the test cases under test conditions. If only some of the seeded errors are found, the test case set is not adequate. The ratio of found seeded errors to the total number of seeded errors is an estimate of the ratio of found real errors to total number errors. This gives a possibility of estimating the number of remaining errors and thereby the remaining test effort.

5)          Performance testing of the system in the lab environment is often inadequate, since the simulators are not exactly replicating the field environment, this result in many errors being revealed during field validation phase.

6)          Test engineers often follow the concept of “Equivalence Classes and Input Partition Testing” to save time and testing effort. This principle is often flawed due to the inexperience of the test engineer or insufficient coverage of the data classes.

7)          Lack of formal methods in developing software prevents the test engineer to take full advantage of the “Structure Based Testing” concept where clearly defined states and modules are required for generating test cases for complete coverage of the system.

 IV.      Challenges during System Integration and Field Validation Testing

The following section describes the challenges faced during System Integration testing of safety critical railway signalling systems:

1)          The System Integration testing should be ideally started after the unit/Module tests are successful passed, but in reality due to the delayed nature of the railway signalling projects, the system integration tests are carried out in parallel with the unit/module tests, this causes many problems to be revealed only at the system level tests.

2)          Many unexpected behaviors of the system are revealed at this stage since the software has not completely gone through unit tests. This causes more delays in the system integration tests and changes in the requirements are required since all the scenarios at the unit level have not been accounted.

3)          The test engineers who have been devoted to find integration issues find themselves more involved in sorting out the problems that should have been caught during unit tests.

4)          The integration issues found during this phase often lead to design changes which are costly to fix and in-turn increase the complexity of the system.

5)          The Field Validation tests are executed in the actual field environment and many of these scenarios are not accounted in the lab, so the same software has different behavior in the lab and field, this leads to confusion and is hard for the developers to debug.

   V.      Challenges in Analysis of Test Results

        The Railways signalling systems generally have large test scenarios to be performed in the field and lot of the data collected during the tests require offline analysis, the below section describes the challenges and problems associated with this phase.

1)          In many railway projects, often the field engineers are recruited locally to ensure easy access to the test site, often these field engineers are new to railway signalling and have little or no training to execute the tests.

2)          The complex nature of the offline log analysis requires test analyst to be fully in synchronization with the field engineer who executed the test, in many cases the lack of communication between the field and offline test analysts result in falsely reporting the test as failed.

3)          In many cases, due to some inherent errors in the test procedure, the log file analyst reports the test as failed and goes through multiple cycles of test execution.

4)          In On-board ATP log analysis, often it is required to check the braking profiles of the systems and requires complete knowledge of the braking algorithms. In many cases the test engineer is not qualified to perform this analysis and results in poorly reported analysis.

5)          The lack of co-ordination between the test lead and the field technicians often result in incomplete tests and later results in incomplete log analysis.

 VI.      Conclusion

Railway signalling is very specialized and unique area where high level of planning is required for all the phases of the project lifecycle especially for the IV and V of safety critical software. Poor planning at the start of the project usually result in cost overruns and delays. In our experience with railway signalling projects, generally limited budgets and time is allocated to IV and V phase which in realty takes the majority of the project budget. If the IV and V phase is planned well in advance and sufficient managerial responsibility is assigned specifically for this task, the projects can be completed in time and with better results, which in turn makes the job of the safety assessor easy. We suggest the following mitigation measures to ensure a successful IV and V of railways signalling systems:

1)             Care should be taken to recruit test engineers who at least have basic knowledge of railway signalling and associated systems.

2)             In case the test engineers are new recruits, they should be put through rigorous training before being assigned critical tasks such as writing test procedures and analyzing the test data logs.

3)             Regular training sessions should be conducted for the test engineers in the project to impart in-depth knowledge of the system.

4)             Encourage test engineers to be innovative in their testing methods instead of just following the regular patterns, this way more errors in the system are revealed which often get undetected with traditional test methods.

5)             Create an environment where test engineers regularly interact with the design team to share each ones experiences and concerns

6)             Create a dedicated managerial team to monitor all the test activities occurring a different sites and co-ordinate them. Better co-ordination between the Lab and field test teams leads to better analysis of the system.

7)             Never follow the approach of parallel testing activities, for example, the system integration tests should never be planned in parallel with the unit tests.

Acknowledgments

The author would like to express his gratitude to Stephen A. Jacklin from the NASA Ames Research Center for his encouragement to take up this study and present my experiences with IV and V in railway signalling domain.

References

¹S.Vinogradov, V.Okulevich, M.Gitsels, “Approaches to meet Certification Requirements for Mission-Critical Domains”, Software Engineering Conference (Russia), 16th Nov. 2006

²Ulrich Haspel, Gunni S. Frederiksen., “The Automated Copenhagen Metro In The First Year Of Operation

- Experience And Outlook,” 9th International Conference On Automated People Movers 2-5 September 2003, Singapore

³K.K Bajpayee, “Emerging Trends in Signalling on Indian Railways” in IRSTE Conference, 2003

⁴Peter Wigger, “Experience with Safety Integrity Level (SIL) Allocation in Railway Applications”, WCRR 2001 – 25. – 29. November 2001, Köln

⁵Dr. Hendrik Schäbe, “The Safety Philosophy Behind the CENELEC Railway Standards”, ESREL 2002, Lyon, March 19-21, 2002

⁶G.Biswas, S.Kumar,T.K.Ghoshal,V.Chandra, “Independent Verification and validation of Software with reference to UFSBI”, presented at IRSTE Seminar, 1999.

⁷Chinnarao Mokkapati, Terry Tse, Alan Rao “A Practical Risk Assessment Methodology for Safety-Critical Train Control Systems”, Office of Research and Development Washington, D.C. 20590, DOT/FRA/ORD-09/15

⁸EN 50126: Railway Applications - The Specification and Demonstration of Dependability, Reliability, Availability, Maintainability and Safety (RAMS). Issue: March 2000.

⁹prEN 50129: Railway Applications- Communications, signalling and processing systems - Safety related electronic systems for signalling. Issue: May 2002

¹⁰prEN 50128: Railway Applications- Communications, signalling and processing systems - Software for railway control and protection systems. Issue: March 2001

---

[1] Senior Systems Engineer, Embedded Systems Group, CMC Americas, Inc., sandeep.patalay@cmc-americas.com.