From The Desk of
Sandeep Patalay
Challenges in Independent Verification and Validation
of Safety Critical Railway Signalling Systems
Sandeep Patalay[1]
CMC Americas, Inc., Pittsburgh, PA, 15220
A railway signalling
system is safety critical system that controls the traffic which includes train
routes, shunting moves and the movements of all other railway vehicles in
accordance with railway rules, regulations and technological processes required
for the operation of the railway system. The overall signalling system consists
of Microprocessor based Wayside controllers, On-board systems controlling the
railway vehicle and supervision systems to monitor the vehicle movements from a
centralized location. The complex nature of railway signalling rules and
operational practices adopted by different railroads pose a difficult task for
the software development of these systems. The complex nature of the software
poses an even more challenging task during the Independent Verification and
Validation of the system. The CENELEC set of standards is the widely accepted
as the governing standard for design, development and Independent Verification
and Validation (IV and V) of railway signalling systems. This paper describes
the challenges faced during the different phases of IV and V of safety critical
railway signalling software which is unique compared to other domains.
Nomenclature
IV and V
= Independent Verification and
Validation
ATP
= Automatic Train Protection
On-Board = Embedded systems used on the Train
CENELEC = European
standards for Railway Signalling
I. Introduction
IV and V
is the most important phase of any safety critical system life cycle. The
result of this phase decides the final outcome of the project and decides
whether the product is fit for use. The IV and V of safety critical software for
railway signalling applications is faced with many challenges due to the
complexity of the systems and the variations it has depending upon the
geography and environment in which it needs to operate. This paper particularly
focuses on the experiences and challenges during different phases of the
IV and V in a railway signalling project. The following areas will be
discussed:
|
1)
Systematic Problems
2)
Challenges during Software Analysis
3)
Challenges during System Integration and Field
Validation Testing
4)
Challenges during Test Result Analysis
II. Systematic Problems during IV and V of Railway Signalling Software
The following systematic
problems are experienced during the IV and V of safety critical software
developed for railway signalling applications:
5)
Lack of formal methods in developing the control
algorithms results in poor understanding of the system by a test engineer.
6)
Lack of domain knowledge in railway signalling systems creates
a technological gap between the software test engineers and the domain
consultants. This leads to errors in software testing, which might lead to
unsafe failures being un-detected.
7)
Since the software and hardware is so complex, complete
test of the system is not possible and most of the faults are revealed at the
field Installation stage or during normal working of the system in field.
8)
The software is often changed for every geographical
location and results in specific code for each location. When the software
structure is not in a generic form, it becomes difficult for the test engineer
to develop test cases for every possible scenario.
9)
The lack of standardization in the railway working
principles results in incomplete test cases as test engineers are not well
versed with all types of railroads.
10) Increase
in the complexity of the software leads to difficulty in testing, since most of
the railway systems are sequential machines they are error prone and are very
difficult to test.
III. Challenges during Software Analysis
The following section describes the challenges faced during
Static and Dynamic analysis of safety critical software developed for railway
signalling applications:
1)
Static Analysis of software is the analysis of the
software code without actually executing it. The railway signalling software
particularly the vehicle braking algorithms are very complex and require the
test engineer to be well versed with the dynamics of the vehicle as well as
good mathematical knowledge. These algorithms require the test engineer to
envisage all the possible states of the algorithms and then create a formal model
of the system. In many cases the lack of Test Engineer’s knowledge about these
algorithms results in insufficient test cases for the model and results in many
of the errors revealing in the latter part of the dynamic analysis.
2)
Dynamic Analysis of software is the analysis of the
software code by actually executing the software and observing the executions.
The dynamic analysis of the safety critical software is an important phase of
the Independent Verification and Validation of the system. The Test engineer
should be well versed with the domain of inputs and outputs of the system. In
many cases the test engineer chooses the boundary values based on the data
range of the variable type. In the real world the boundary values depends on
the actual working environment of the system, for example the GPS signal
boundary values received by the vehicle to determine its position varies based
on the geographical location of the railroad and is embedded in the vehicle
database. When the test engineer creates test cases for this part of the
software it should be changed based on the geographical location where the
train is operating, instead the literal boundary value of the variable type
would pass the dynamic analysis and this error would only be revealed during
field validation tests.
3)
Inexperienced test engineers just follow the rule book
and often result in insufficient test scenarios. Testing experience and
intuition combined with knowledge and curiosity about the system under test may
add some uncategorized test cases to the designed test case set. Special values
or combinations of values may be error-prone. Some interesting test cases may
be derived from inspection checklists.
4)
Test engineers generally are not well versed with the
concept of error seeding and do not try to measure the effectiveness of their
test cases. Some known error types should be inserted in the program, and the
program should be executed with the test cases under test conditions. If only
some of the seeded errors are found, the test case set is not adequate. The
ratio of found seeded errors to the total number of seeded errors is an
estimate of the ratio of found real errors to total number errors. This gives a
possibility of estimating the number of remaining errors and thereby the
remaining test effort.
5)
Performance testing of the system in the lab
environment is often inadequate, since the simulators are not exactly
replicating the field environment, this result in many errors being revealed
during field validation phase.
6)
Test engineers often follow the concept of “Equivalence
Classes and Input Partition Testing” to save time and testing effort. This
principle is often flawed due to the inexperience of the test engineer or
insufficient coverage of the data classes.
7)
Lack of formal methods in developing software prevents
the test engineer to take full advantage of the “Structure Based Testing”
concept where clearly defined states and modules are required for generating
test cases for complete coverage of the system.
IV. Challenges during System Integration and Field Validation Testing
The following section describes the challenges faced during
System Integration testing of safety critical railway signalling systems:
1)
The System Integration testing should be ideally
started after the unit/Module tests are successful passed, but in reality due
to the delayed nature of the railway signalling projects, the system
integration tests are carried out in parallel with the unit/module tests, this
causes many problems to be revealed only at the system level tests.
2)
Many unexpected behaviors of the system are revealed at
this stage since the software has not completely gone through unit tests. This
causes more delays in the system integration tests and changes in the
requirements are required since all the scenarios at the unit level have not
been accounted.
3)
The test engineers who have been devoted to find
integration issues find themselves more involved in sorting out the problems
that should have been caught during unit tests.
4)
The integration issues found during this phase often
lead to design changes which are costly to fix and in-turn increase the
complexity of the system.
5)
The Field Validation tests are executed in the actual
field environment and many of these scenarios are not accounted in the lab, so
the same software has different behavior in the lab and field, this leads to
confusion and is hard for the developers to debug.
V. Challenges in Analysis of Test Results
The Railways signalling systems generally have large test
scenarios to be performed in the field and lot of the data collected during the
tests require offline analysis, the below section describes the challenges and
problems associated with this phase.
1)
In many railway projects, often the field engineers are
recruited locally to ensure easy access to the test site, often these field
engineers are new to railway signalling and have little or no training to
execute the tests.
2)
The complex nature of the offline log analysis requires
test analyst to be fully in synchronization with the field engineer who
executed the test, in many cases the lack of communication between the field
and offline test analysts result in falsely reporting the test as failed.
3)
In many cases, due to some inherent errors in the test
procedure, the log file analyst reports the test as failed and goes through
multiple cycles of test execution.
4)
In On-board ATP log analysis, often it is required to
check the braking profiles of the systems and requires complete knowledge of
the braking algorithms. In many cases the test engineer is not qualified to
perform this analysis and results in poorly reported analysis.
5)
The lack of co-ordination between the test lead and the
field technicians often result in incomplete tests and later results in
incomplete log analysis.
VI. Conclusion
Railway signalling is very specialized and unique area where high
level of planning is required for all the phases of the project lifecycle
especially for the IV and V of safety critical software. Poor planning at the
start of the project usually result in cost overruns and delays. In our
experience with railway signalling projects, generally limited budgets and time
is allocated to IV and V phase which in realty takes the majority of the
project budget. If the IV and V phase is planned well in advance and sufficient
managerial responsibility is assigned specifically for this task, the projects
can be completed in time and with better results, which in turn makes the job
of the safety assessor easy. We suggest the following mitigation measures to
ensure a successful IV and V of railways signalling systems:
1)
Care should be taken to recruit test engineers who at
least have basic knowledge of railway signalling and associated systems.
2)
In case the test engineers are new recruits, they
should be put through rigorous training before being assigned critical tasks
such as writing test procedures and analyzing the test data logs.
3)
Regular training sessions should be conducted for the
test engineers in the project to impart in-depth knowledge of the system.
4)
Encourage test engineers to be innovative in their
testing methods instead of just following the regular patterns, this way more
errors in the system are revealed which often get undetected with traditional
test methods.
5)
Create an environment where test engineers regularly
interact with the design team to share each ones experiences and concerns
6)
Create a dedicated managerial team to monitor all the
test activities occurring a different sites and co-ordinate them. Better
co-ordination between the Lab and field test teams leads to better analysis of
the system.
7)
Never follow the approach of parallel testing
activities, for example, the system integration tests should never be planned
in parallel with the unit tests.
Acknowledgments
The author would like to express his gratitude to Stephen A.
Jacklin from the NASA Ames Research Center for his encouragement to take up
this study and present my experiences with IV and V in railway signalling
domain.
References
1S.Vinogradov, V.Okulevich, M.Gitsels, “Approaches to meet Certification
Requirements for Mission-Critical Domains”, Software Engineering Conference
(Russia), 16th Nov. 2006
2Ulrich Haspel, Gunni S. Frederiksen., “The Automated Copenhagen Metro In The First
Year Of Operation
- Experience And Outlook,” 9th International Conference On Automated People Movers
2-5 September 2003, Singapore
3K.K Bajpayee, “Emerging
Trends in Signalling on Indian Railways” in IRSTE Conference, 2003
4Peter Wigger, “Experience
with Safety Integrity Level (SIL) Allocation in Railway Applications”, WCRR
2001 – 25. – 29. November 2001, Köln
5Dr.
Hendrik Schäbe, “The Safety Philosophy
Behind the CENELEC Railway Standards”, ESREL 2002, Lyon, March 19-21, 2002
6G.Biswas, S.Kumar,T.K.Ghoshal,V.Chandra, “Independent Verification and validation of
Software with reference to UFSBI”, presented at IRSTE Seminar, 1999.
7Chinnarao Mokkapati, Terry Tse, Alan Rao “A Practical Risk Assessment Methodology for
Safety-Critical Train Control Systems”, Office of Research and Development
Washington, D.C. 20590, DOT/FRA/ORD-09/15
8EN 50126: Railway Applications -
The Specification and Demonstration of Dependability, Reliability,
Availability, Maintainability and Safety (RAMS). Issue: March 2000.
9prEN 50129: Railway Applications-
Communications, signalling and processing systems - Safety related electronic systems
for signalling. Issue: May 2002
10prEN 50128: Railway Applications-
Communications, signalling and processing systems - Software for railway
control and protection systems. Issue: March 2001
[1] Senior
Systems Engineer, Embedded Systems Group, CMC Americas, Inc., sandeep.patalay@cmc-americas.com.
