How-to Manage Bugs
A bug is an unexpected problem in the software or hardware which can be reported for any issue in a product by e.g. product managers, product owners, test engineers, or customers (via L3 or L4 Support).
Managing bugs is a complex task, offering many options depending on the nature of the bugs. This guide describes the overall bug management process, and links to other bug-related guides as visualized below.
This means that – when creating a bug – it’s important to:
- Classify the bug.
- Know if it is a “usual” bug, an enhancement bug, a regression bug, or a bug existing in multiple releases.
Intended for
Release owners, product owners, scrum masters, software engineers, hardware engineers, test engineers, and L4 engineers.
Activities
Create bug
A bug work item in Azure DevOps (ADO) captures the problem in a way that can be communicated and corrected. The goal of creating a bug is to accurately report a problem in a way that allows the reader to understand its full impact and background. For how to set the severity, security effect, and priority of a bug, see the Bug Classification guide.
If the problem is already reported by another bug, the duplicate one must be closed and linked to the already existing one.
Bugs are created for defects found in completed work items. Under some circumstances it is possible to handle enhancements as bugs, see How-to Handle Enhancements.
Regression bugs should be marked and handled according to Regression Bugs.
Each bug must describe one specific problem:
- If more than one problem is found, more than one bug must be entered in ADO.
- If the bug impacts several function areas or product lines, then clone the bug to each impacted area.
- If the bug affects more than one release, see How-to Handle Bugs in Multiple Releases.
When creating a bug, refer to Standard Bug Template and Area and Iteration Paths for mandatory fields to fill in.
Note: The assigned area path, which identifies the product or version where the issue was found, should never be changed. If it's necessary to assign the bug to a different area path, please follow the process for deferred bugs, see How-to Handle Deferred Bugs.
Check DoR
All available information must be clearly described in the bug work item’s mandatory data.
Bugs related to specific work items (e.g., features, user stories, etc.) must be linked with the work item itself.
The new bug is assigned to the product owner or the scrum master.
The product owner or the scrum master verifies that the definition of ready (DoR) is met.
CCB decision
The change control board (CCB) decides if the bug shall be fixed or not, and whether it should be accepted for the current release or assigned to a future release. If a bug is not properly assigned, the CCB is responsible for making sure that the bug is correctly addressed. See State and transitions for further details.
CCB involvement is strictly required for post-release bugs including L4 bugs and their definition of done (DoD), see DoD for L4 bugs. CCB decisions must be recorded appropriately.
Note: Product owners, release owners, and scrum masters - if they are also part of the CCB - can close anything within the scope of the CCB group, including bugs, using the recommended best practices and the “4 eyes rule”. They can also decide the priority of the introduced bugs, and are expected to take decisions within two weeks and to finalize (close) the bugs in two sprints.
The goal is to have virtual zero test (introduced) bugs by the end of the sprint. If any bugs slip to the next sprint, they must be prioritized by the development team and fixed. All possible reasons for closure are accepted as long as they are documented.
Changes in the project’s scope and decisions on what bugs shall be included in a specific release can only be done by the CCB.
Only product owners or the CCB can change the severity. The reason must be documented, and the bug author must be notified.
Bugs that impact safety, security, or multiple teams must be escalated to the CCB. CCB approval is also required to defer introduced bugs to future releases, see How-to Handle Deferred Bugs.
Investigate
Impact analysis must be done for every new bug: which requirements and/or functions need to be re-tested, which documents need to be updated, and the impacts on other parts of the system must be evaluated and described; impact analysis needs to be tailored to the stream needs.
The standard impact analysis template for bug consists of 5 questions:
| No. | Question | Explanation |
|---|---|---|
| 1 | What is affected? | Please identify the specific product name, version, and components that are impacted by the change or issue. |
| 2 | How is it affected? | Analyze the nature and extent of the impact on the identified areas. Consider how general system functions, performance, properties, and libraries are affected by the change. |
| 3 | What is the solution and how long will it take? | Propose viable solutions to fix the issue. This may involve technical fixes, process adjustments, or resource allocation. Estimate the time required to implement the proposed solutions, considering the complexity and resources needed. |
| 4 | What needs to be changed? | Clearly define the changes required to address the impact. This includes modifications to design and documents including user documentation and security documentation. For document please list the document ID, title and version. For code please list the code file name in repository. |
| 5 | What tests need to be executed? | Please list here the required test cases to be executed, including test cases to be rerun and/or new test cases to be developed, to ensure the bug is fixed without introducing new issues, e.g. regression testing related to security for changes to security critical components. |
For bugs in Functional Safety related releases (interference free included), please use IA Questions for Error Corrections: 3BSE042623 and IA Guidance Safety - 3BSE058012.
Security analysis must be completed for every new bug, and critical issues are discussed with the cyber security engineer.
Investigation information, such as impact analysis, CCB decisions, implementation proposals, and other critical details, must not be shared in the “Discussion” field since it will be lost if a bug is cloned. Instead such information can be stored in other fields such as "System Info", "Description", "CCB Discussion", "Impact Analysis", or "Workarounds".
Plan
For each bug, once the bug is “Active” or “Approved”, it is good practice for team members to create tasks in ADO to best handle all the required actions. The tasks must be linked to the bug filling the “Estimate” field according to the information provided in the estimate comment, e.g.:
- Development
- Test
- Documentation writing
- Others
If the bug was identified without a test case, a new test case at the appropriate level (unit test, component test, or product test) should be created. If the test case already exists but some test steps are missing, the test case should be updated.
A test case affected by a bug must be set to “Failed”, regardless of the bug severity. If required by the project, notes can be added to the test report, and/or the test case can be split into smaller test cases.
Fix
For ”High” and ”Critical” bugs, corrective action should be identified to avoid future regression: improvements in development, improvements in technical documentation, and new tests (automatic functional tests, unit tests, and/or test cases to be added to the test plan).
A good practice is to frequently add comments to the bug with the work-in-progress details. Once a bug is fixed and integrated into the new build, set the bug state to “Resolved”, add a comment in the discussion, and assign a bug to the tester for verification.
Code review is mandatory for new code and code changes (pull request). The bug work item must be linked to the code changes.
Verify
Once fixed, the bug is verified by the test engineer and then assigned back to the scrum master for the last check or reactivated if the issue is not resolved. It is the product owner's responsibility to identify the most appropriate test environment.
Check DoD
Before closure, the scrum master or the product owner verifies that DoD is met.
Close bug
The product owner is accountable for the bug’s closure and can delegate to someone else.
Details
State and transitions
| State | Reason | Explanation |
|---|---|---|
| New (Assigned to: product owner or scrum master) | New | The bug is created, but no decision has been taken if it shall be implemented or not. It may be incomplete in this state. |
| Investigation complete | Investigation is done, awaiting CCB decision for next step. | |
| Not fixed | The defect was not fixed when set to "Resolved" or "Closed", and it needs to be re-evaluated. | |
| Test failed | Test failed after the defect was set to "Resolved" or "Closed", and it needs to be re-evaluated. | |
| Active (Assigned to: HW/SW engineer or technical writer) | Investigate | The bug is sent for investigation. |
| Approved | The bug is approved by the CCB to be corrected. | |
| Not fixed | The defect was not fixed when set to "Resolved", and it needs to be corrected. | |
| Test failed | Test failed after the defect was set to "Resolved", and it needs to be corrected. | |
| Regression | A closed bug that was once fixed and verified has re-appeared. It is preferred to create a new bug when this occurs. | |
| Reactivated | A deferred bug is reactivated (within the same release). | |
| Resolved (Assigned to: scrum master, test engineer, or product owner) | Fixed | The bug has been resolved and awaiting formal closing. |
| As designed | The bug is suggested to be rejected. The product works as designed. | |
| Cannot reproduce | The bug cannot be reproduced. | |
| Obsolete | The bug is no longer valid. | |
| Resolved in error | The bug was closed without being resolved. | |
| Will not fix | The bug is valid. CCB decided to accept the defect for the product as reported in the bug and not to invest resources to fix it. | |
| Closed (Accountable: product owner) | Verified | The bug is resolved and verified, and all associated work has been completed. |
| Fixed and verified | The bug is fixed and verified, and all associated work has been completed. | |
| As designed | This is not a bug. The product works as designed. | |
| Cannot reproduce | The bug cannot be reproduced. | |
| Copied to Backlog | A post-release bug affecting an already delivered release. | |
| Deferred | The bug is deferred to another product release. A new bug is created in the deferred context. | |
| Duplicate | The bug is a duplicate of another bug. | |
| Obsolete | The bug is no longer valid. |
Note: Streams can use different ADO templates using other states and transitions.