3 The ASPICE-Processes (VDA Scope)
3.10 SYS.5 – System Qualification Test
What is the purpose?
System qualification tests ensure that the integrated system meets the system requirements and can be delivered to the customer.
What is the benefit?
These tests are carried out after the system has been integrated from hardware, software, mechanics, and possibly other components. These tests ensure that the complete system works as expected and meets the system requirements.
What is the content?
- The system test strategy for the tests of the overall system is defined (BP1). This supports all parties involved (testers, test managers, requirements engineers, project management,…) to develop a common understanding of how to accomplish the work (BP1).
- It must be prevented that changes to the system have negative side effects on already successfully tested requirements. For this purpose a regression strategy needs to be provided (BP1).
- The tests that demonstrate the implementation of the system requirements are to be specified in detail (BP2).
- Test cases are selected for execution in line with the test strategy, the regression strategy, and the release plan (BP3). Tests are performed and results are recorded (BP4).
- There needs to be bidirectional traceability between system requirements, test cases, and test results (BP5). The system requirements and the test cases need to be consistent (BP6). Consistency means that the traceability links are correct and complete and that the test cases are correct with respect to the requirements. This is typically checked through reviews.
- A test summary report is produced and disseminated to the relevant people (like project management and customer) (BP7).
Experiences, problems and hints
- The system qualification test process (SYS.5) basically expects the same activities as the software qualification test process (SWE.6). The difference is just the object that is tested. In practice, both test processes are often combined on the target hardware. This is acceptable as long as one can demonstrate that the system requirements have been tested completely.
- The system qualification test is closely connected to the system requirements activities (SYS.2). The requirements engineers develop the verification criteria which contain their expertise and hints on what should be observed when developing the tests.
- Traceability between requirements and tests is essential to be able to demonstrate coverage of requirements by tests and to monitor the test progress. This requires automated traceability using a highly integrated requirements tool and test tool.
- Consistency checking is often underestimated and poorly implemented. This requires reviews which check whether the tests are correct and the requirements are covered completely.
- If the project uses a platform system the tests against platform requirements and those against the project-specific requirements should be seamlessly adjusted.
Automotive SPICE text of System Qualification Test (SYS.5)
The purpose of the System Qualification Test Process is to ensure that the integrated system is tested to provide evidence for compliance with the system requirements and that the system is ready for delivery.
BP1: Develop system qualification test strategy including regression test strategy. Develop a strategy for system qualification test consistent with the project plan and the release plan. This includes a regression test strategy for re-testing the integrated system if a system item is changed.
BP2: Develop specification for system qualification test. Develop the specification for system qualification test including test cases based on the verification criteria according to the system qualification test strategy. The test specification shall be suitable to provide evidence for compliance of the integrated system with the system requirements.
BP3: Select test cases. Select test cases from the system qualification test specification. The selection of test cases shall have sufficient coverage according to the system qualification test strategy and the release plan.
BP4: Test integrated system. Test the integrated system using the selected test cases. Record the system qualification test results and logs.
NOTE 1: The See SUP.9 for handling of non-conformances.
BP5: Establish bidirectional traceability. Establish bidirectional traceability between system requirements and test cases included in the system qualification test specification. Establish bidirectional traceability between test cases included in the system qualification test specification and system qualification test results.
NOTE 2: Bidirectional traceability supports coverage, consistency and impact analysis.
BP6: Ensure consistency. Ensure consistency between system requirements and test cases included in the system qualification test specification.
NOTE 3: Consistency is supported by bidirectional traceability and can be demonstrated by review records.
BP7: Summarize and communicate results. Summarize the system qualification test results and communicate them to all affected parties.
NOTE 4: Providing all necessary information from the test case execution in a summary enables other parties to judge the consequences.
Output Work Products
Test plan, Test specification, Test result, Traceability record, Communication record, Review record.
If you wish to continue to the next chapter click here: