Bug Classification
This guide describes how bugs – when they are created in Azure DevOps (ADO) – also are classified to ensure that they are handled with regards to their severity and potential impact.
Together with other bug-related guides, it provides information to help ensure correct handling of all types of bugs. It relates to PCP R&D’s overall bug management process, described in How-to Manage Bugs, as visualized below.
Severity evaluation
When testing software or hardware, managing bugs can be a daunting task. With limited resources, time pressure, or upcoming deadlines - the teams quickly need to analyze and decide the order the defects should be corrected, starting with the most important ones first.
The submitter (typically the test engineer or the L4 engineer) sets the severity of the bug based on the estimated customer impact. Discuss any changes to the severity with the submitter, and document the motivation for the change in the bug work item.
If it's a security bug, the submitter also sets the security effect. If the security effect is set to "To be determined", it will later be evaluated by the change control board (CCB). The cyber security engineer calculates the CVSS score as part of the severity evaluation, and the severity may have to be revised based on the score. If it isn't a security bug, make sure the security effect is set to "No security bug".
The product manager or product owner (e.g., in a CCB meeting) typically sets priorities based on the order of importance or urgency.
Severity definition
Severity describes the failure or vulnerability impact that a bug could cause (from the perspective of) a customer, determined by the organization responsible for the software or hardware.
Evaluate security bugs with the common vulnerability scoring system (CVSS), and use FIRST's calculator to calculate the score. Document the CVSS score, including the complete vector, the severity value, and the severity rating (low, medium, high, or critical) in the bug. For a security bug not yet released, calculate the CVSS base score. For a security bug in a released product, also calculate the temporal CVSS score.
The CCB and the cyber security engineer decide if the severity given by the CVSS score should be used directly as the severity for the bug or if the severity should be different from the calculated CVSS score. E.g., if the criticality of a component is low, the severity may be set lower than the CVSS score indicates.
1 - Critical
The bug causes impairment of critical system functions or unrecoverable data loss, and no workaround solution exists. Typically, these are bugs that stop a single function or executable, and normal operation of the function is not possible.
For the customer, the bug could cause loss of life, loss of production, or serious security/safety violations.
A critical-security bug has a CVSS score between 9.0-10.0.
2 - High
The bug causes impairment of a system function, but a workaround exists. Typically, these bugs affect functionality, produce erroneous or unexpected results or prevent the function from completing.
For the customer, the bug could cause significant instability for a function, extra effort for the workaround, or result in a customer requirement not being met.
A high-security bug has a CVSS score between 7.0-8.9.
3 - Medium
The bug causes a minor loss in a system function. Typically, these bugs produce erroneous or unexpected results. An easy workaround allows the function to continue and does not prevent further progress in other areas.
For the customer, the bug means the product requirement is met but causes customer irritation could result in a loss of sales.
A medium-security bug has a CVSS score between 4.0-6.9.
4 - Low
The bug causes inconvenience or annoyance. Typically, these are more of a cosmetic nature, such as documentation issues, spelling errors, and changes to labels or colors.
For the customer, the bug doesn't adversely affect the function or usability.
A low-security bug has a CVSS score between 0.1-3.9.
Security effect definition
"Security effect" in the bug tracking system is based on the Microsoft STRIDE threat model.
This model looks at threats from an attacker’s perspective and defines six threat categories. If there is more than one applicable threat category, select the most relevant one.
| Security effect / Threat category | Description |
|---|---|
| Spoofing | A bug that may allow an attacker to pose as something or somebody else. |
| Tampering | A bug that may allow modification of data or code (at rest or in transfer) |
| Repudiation | A bug that may allow a user to deny having performed an action. For example, if a user performs an illegal operation that the system can’t trace. |
| Information Disclosure | A bug that may allow exposure of information to users who are not supposed to have access to it. |
| Denial of Service | A bug that may deny or degrade service to valid users. These are typically bugs that affect the availability and reliability of the system. |
| Elevation of Privilege | A bug that may allow a user to gain increased capability, typically an anonymous or standard user that may gain root or admin capabilities. In an ICS this could also be an operator account that suddenly gains the capability of a safety user or engineer |
| Attack Surface Reduction | This Security Effect is not a STRIDE threat but is used to indicate an action needed to reduce the attack surface of the system. Could be to add a closure of a TCP port not needed, or by default disabling a service only needed in special use cases. |
| To be Determined | The CCB evaluates the security bug later |
| Not a Security Bug | Indicates that the bug has no known security effect. |
More information about security threats and software vulnerabilities is available in the Software Vulnerabilities and Threats guide.
Priority definition
The priority indicates the order bugs should be corrected and set by the CCB. The priority is defined relative to other reported bugs. Engineers use this field to prioritize their work.
CCB is allowed to reprioritize the bugs if it helps the teams to understand what to correct first.
1 - Immediate
Correct the bug immediately - it's blocking the development and must be fixed.
2 - High
Fix the priority 2 (high) bugs before the priority 3 (medium) bugs.
3 - Medium
Fix the priority 3 (medium) bugs before the priority 4 (low) bugs.
4 - Low
Fix the priority 4 (low) bugs if time allows.