5.1 How do assessments work?
Why are assessments carried out?
This can be derived from the acronym for SPICE: Software Process Improvement and Capability DEtermination. Assessments are thus carried out either as a status determination for internal process improvement (Process Improvement) or to determine the process quality of a supplier (Capability Determination) and thus to assess the risk regarding the supplier selection.
In the meantime SPICE’s “S” is also used for “system”, but the main focus is still on the software. This also corresponds to the original intention of the OEMs, who were able to trace a large part of the problems of electronic systems to software errors.
Which types of assessments are performed?
Regarding the rigor applied in the evaluation methodology we can distinguish between process improvement assessments and process-related product risk assessments.
Regarding the organizational scope of the assessment we can distinguish:
- Project Assessments,
- Organizational maturity assessments (an entire organization unit is examined regarding its process maturity – details in chapter 5.5).
Project assessments currently account for more than 99 % of the assessments,but there is growing interest in organizational maturity. The following statements are mainly related to project assessments, unless explicitly described otherwise.
What is being assessed in assessments and how are they performed?
A team of assessors interviews people who use the processes in the assessed projects or in the organization. They check documents, tools, databases, and, where necessary, orkplaces. For assessments including capability level 3, standard processes of the organization are also examined in interviews with process experts.
Based on interviews and checks of artifacts the assessment team tries to find weaknesses or risks related to the Automotive SPICE model. Most assessments are carried out against a standardized process scope (VDA Scope).
Important principle: Only processes are assessed, not people!
How are ratings performed?
The Process Attributes are evaluated using the NPLF rating scale (see chapter 2.5). The evaluation is typically derived from the NPLF assessment of base practices and generic practices (see chapter 2.5 and chapter 2.6).
The Capability Levels are calculated using a simple algorithm: To achieve a level, its process Attributes need to be “Largely” or “Fully achieved” and all Process Attributes of the lower Capability Levels need to be “Fully achieved” (see chapter 2.5).
How long does an assessment take?
The interview phase of assessments on the VDA-Scope up to Level 2 usually lasts five days or more. This number is related to assessments before the VDA Automotive SPICE guidelines became mandatory. In 2018 there were no reliable data on the additional time needed for the Guidelines.
If a project is assessed up to Capability Level 3, an extra day should be added for checking the standard processes. If the processes of the extended VDA scope are assessed additionally (cf. chapter 1.1) another day is required.
Other factors that affect the length of an assessment include:
- Number of instances (Instances need to be distinguished if a process is performed in different ways. The number of instances increases the evaluation effort a lot ).
- Complexity of the project (Some projects are not only divided into several sub-projects, but have several other layers of project management (“sub-sub-projects”). In those cases more time should be calculated for the interviews in project management and meaningful samples of the sub-projects for the engineering and support processes has to be checked.),
- Distributed projects (Multisite projects can cause additional effort when the assessment team must travel between sites. Alternatively, one can let the different teams travel to one site for the assessment, provided that they have access to their respective work products, tools and data bases).
- Customer and Platform project (Often a customer project is based on an internal platform development. If possible, the assessment should be performed for both development teams, the customer project and the platform project).
To read about roles involved in assessment please continue here: