Software Testing

We use a real time testing tool to assist in our automation of testing. This provides us with the optimum testing strategy providing sufficient samples through a system's input space and the range of metrics identified below. The tool doesn't just manage the results of testing: it also helps by generating the code required to undertake tests on components of software as they are developed.

Software is tested during its development cycle to provide assurance about its quality. Test Cases are based upon scenarios and sample data, which are defined alongside the product's requirements. When the test cases are run, the system is monitored in a variety of ways:

• The "answers" yielded by the test cases are checked to ensure that the expected values are obtained. These answers should be retained so that they may be compared with the values obtained from later test runs to ensure that changes made to software have not affected the results it produces.
• The source code should be monitored during the running of test cases to ensure that all detectably distinct pathways through the code have been covered. This analysis provides a minimum degree of assurance that the tests have covered the whole of the code.
• Memory tests should be performed to ensure that wherever the program under test has acquired memory it has also released it. If it has not, it will be leaking memory, which is a common sign of an underlying design error.
• The code's performance whilst running test cases should be monitored to ensure that the program's resource demands are in line with expectations.
• For a real time system, the tests must be analysed to verify that all time critical responses are made within the required timing window.

It should be clear what testing achieves: it is able to identify the presence of faults in a system. It does not prove that a system is free of faults. The major problem with software testing is that the complexity of software is due to the potential for very large numbers of separate paths through the software. These are the result of not just branching, due to loops or conditional statements, but also to the ranges of values that may be encountered. In most systems complete verification through exhaustive testing is simply impossible.