4. The Automotive SPICE Guidelines
These guidelines have been developed by the VDA working group 13 to improve the quality and reproducibility of assessment results. They give guidance on how to interpret the model. They also contain rules and recommendations which need to be observed when performing ratings.
A quantitative comparison of the guidelines and the model:
- Automotive SPICE PAM
- 132 pages
- 133 Base Practices and 23 Generic Practices
- Automotive SPICE Guidelines
- 271 pages
- 484 Rules and Recommendations
How relevant are these guidelines?
These guidelines are mandatory for assessments performed with Automotive SPICE version 3.0 onwards. They require a more detailed planning of assessments. The rules provided need to be observed by the Lead Assessor and a written justification needs to be given if rules were not followed. In contrast to this the recommendations are just what the name suggests and there are no justifications required if not followed.
Types of assessments according to the guidelines
The guidelines distinguish two types of assessments. Which type was chosen shall be explicitly specified in the assessment plan.
- Process improvement assessments
These don’t need to be representative in terms of coverage of the whole product and the whole project team. This means that the product scope might be reduced to just a few “golden samples” and the project scope might be limited to just one of many development teams within the project. Also negative impacts from one process to another can be ignored (e.g., the effect of poor requirements on their test can be ignored). The purpose of these assessments is a proof of concept of the processes.
- Process-related product risk assessment.
These assessments are the exact opposite. They need to have a representative scope and all impacts between processes are relevant for the rating. The purpose of these assessments is to evaluate the risks of the product when delivered to the OEM.
Requirements on instances in an assessment
In assessments it is very common that processes are performed in different ways. Examples are:
- development approaches may vary (e.g., manual coding vs. Code generation)
- different cases may require different procedures (e.g., managing customer-reported problems vs. managing problems reported internally)
- a process may be handled differently by different teams (e.g., managing software requirements vs. managing hardware requirements)
- a process may vary because of different tools being used (e.g. test tools for JAVA vs. test tools for C)
- a process may vary because of different locations and cultural backgrounds (e.g. testing done by the Detroit team vs. testing done by the Mumbai team)
- processes were changed at some point in time (which means the process before the milestone vs. the process after the milestone)
The guidelines refer to these different ways as “instances” and require the following practices:
- Instances are listed and the ones in scope are identified.
- Process Attributes for all instances in scope are rated.
- Ratings are aggregated across instances.
Aggregation of ratings if there is more than one instance
The guidelines define the following procedure for aggregation of ratings:
- Rate every Process Attribute for every instance
- Determine the Capability Level for the instances (optional)
- Aggregate the PA ratings of the instances
- Determine the overall Capability Level based on the aggregated PA ratings
Subject matters on which guidance is provided
Most of the material deals with these topics:
- assessment planning and ratings (see above)
- the processes of the VDA scope
- Capability Level 2
- Capability Level 3
However, the guidelines give also guidance on:
- verification criteria
- model-based development
- agile development
- distributed development
- usage of third party software (including free and open source software, licenses)
- platform and legacy software
- application parameters
Rules and recommendations
The guidelines use the concept of rules and recommendations as follows:
- Rules are mandatory. Lead Assessors need to observe the rules and give written justifications if rules were not followed. (No documentation is required if rules have been followed.) There are 287 rules.
- Recommendations are just suggestions. They don’t need to be followed and there is no documentation required. There are 197 recommendations.
Typology of rules and recommendations
- Some rules and recommendations give a short guidance (which is contained in the rule itself).
- Example: “If the system requirements and their interdependencies are not evaluated in terms of correctness, technical feasibility and verifiability the indicator BP3 must not be rated F.”
- Some rules and recommendations refer to a list of properties of a work product (sometimes almost a full page of text).
- Example: “If one of the aspects a), b) or f) is missing in the verification criteria, the indicator SYS.2.BP5/SWE.1.BP5 must not be rated higher than P.”
- Some rules and recommendations refer to a rating dependency between practices. Example: “If the BP for communication of work is downrated or the BP for summarizing and communicating test results is downrated, this should be in line with the rating of GP 2.1.7.”
The challenges of the guidelines
The challenge when applying the guidelines in an assessment is the additional workload on the lead assessor. Although some of the rules go without saying others are not so obvious and require checking guideline contents during the interview. This requires advanced assessment tools to
- display the required information for every practice assessed
- automatically check rating dependencies between practices and show the conflicts
Reading additional material during interviews or checking rating dependencies requires time. The same is true for writing justifications if rules were not followed. Although the time required for checking and documenting may be small the number of times this happens may be high. A statistical analysis we did provided the following figures (for an assessment comprising all 16 VDA standard processes and to Capability Level 3):
- A traceability matrix covering the Automotive SPICE model elements in columns and the Automotive SPICE Guidelines elements (rules only) in rows will be composed like this:
- 581 columns consisting of
- 133 columns for the Base Practices of the 16 processes
- 448 columns for the 23 Generic Practices + 5 Process Attributes for each of the 16 processes
- 287 rows for the rules
- This results in 166,747 cells.
- 581 columns consisting of
- 1,774 cells of the matrix will be checkmarked (every time a rule applies to a Practice or Process Attribute)
- The practice which has been hit by the most rules received 13 hits. This means 13 rules need to be taken into account when assessing this practice.
- In average a practice received 3.1 hits.
One can assume that a reasonable number of the 1,774 cells of the matrix will require attention and effort. This number multiplied with the average number of minutes required for checking and documenting will be the likely additional effort for the assessment.
If you wish to continue to the next chapter click here: