Skip to main content

Configuration Audits

This conceptual guide explains the purpose of configuration audits and presents the main parts of a configuration audit and the high-level approach adopted in PCP.

A detailed guide How-to Perform Configuration Audits is also available to explain how to perform a configuration audit.

Intended for​

  • Configuration managers who implement and execute configuration audits.

  • Release owners, scrum masters, product owners, and quality control managers who need to visualize configuration audits.

Configuration audits​

According to the IEEE Std 828-2012, IEEE Standard for Configuration Management in Systems and Software Engineering,

"The purpose of configuration auditing is to objectively assess the integrity of the product both from a functional perspective (how the different technical processes of product development—from requirements to testing—were performed) and from a physical perspective (how the as-is product was built and changes were applied)."

Any configuration management discrepancies and deficiencies discovered during audits shall be recorded and reported to project and organizational management. Actions taken to correct these and the results of those actions shall also be recorded and reported.

Configuration audits are typically classified as functional configuration audits (FCA), physical configuration audits (PCA), and baseline configuration audits.

Functional configuration audit​

An FCA is an audit conducted to verify that the development of a configuration item has been completed satisfactorily, that the item has achieved the performance and functional characteristics specified in the functional or allocated configuration identification, and that it is operational and support documents are complete and satisfactory. (ISO/IEC 24765:2009)

FCA-1

Does it wash clothes?

Physical configuration audit​

A PCA is an audit conducted to verify that a configuration item, as built, conforms to the technical documentation that defines it (ISO/IEC 24765:2009).

For software, the purpose of the software PCA is to ensure that the design and reference documentation is consistent with the as-built software product.

PCA-2

Are all the pieces there and in the right place?

Baseline configuration audit​

Baseline configuration audits shall be performed to assess the proper creation of baselines.

The baseline configuration audit shall inspect the following:

  • Baseline configuration item readiness.
  • Baseline identification.
  • Baseline approval.

PCA

Which build shall we test?

DevOps configuration audit approach​

The IEEE DevOps Standard 2675-2021 suggests this approach:

  1. Conduct functional configuration audits and communicate results.
  2. Audit the system configuration periodically to avoid configuration drift.
  3. Apply automated procedures to verify the integrity of baselines (e.g., physical configuration audit).

We adopt this approach in the context of the PCP R&D organization and its processes. With Azure DevOps (ADO), a large part of a configuration audit can be performed automatically and integrated into the software development lifecycle.

The FCA largely overlaps with the release criteria, and most questions can be answered automatically using the available data in ADO. Release pipelines can verify quality gates automatically with work item queries and track the outcome of the audits as soon as possible.

Most of the PCAs and baseline configuration audits are not necessary if a solid automatic build and release process is in place. This process should be checked early as part of the definition of done (DoD) of CM activities, not when G4 is approaching.

The remaining part of the configuration audit that cannot be automated can be performed with a manual checklist during a planned periodic configuration audit session. Its execution, findings, and corrective actions are tracked in ADO.

Plan configuration audits​

Plan configuration audits during the product development lifecycle.

  • According to the deliverable and milestone tracker, it is mandatory to have performed the configuration audit at M4. The gate assessor can use the material at M4 to assess all aspects of the project and present it at the G4 gate meeting.

It is recommended that a configuration audit session be executed at least once at each program increment to promote good habits and fix issues as soon as possible before they become risks or bigger problems.

Configuration audits shall be automated as much as possible and integrated into the tools used by teams.

Both the automation of configuration audits and configuration audit sessions shall be planned.

Collaboration among configuration managers across different teams is encouraged. It is a good practice to involve a configuration manager from another team or project in performing a configuration audit.

Automate configuration audits​

Configuration audits shall be automated as much as possible and integrated into the tools used by teams, such as:

  • Work items: Planned audit activities, audit findings, corrective actions.
  • Queries: Find data for configuration audits.
  • Dashboards: Release criteria, traceability issues, data quality.
  • Wiki: Summary of audits and related actions.
  • Build and release pipelines: Quality gates, approvals, tools/scripts.

Configuration audit sessions​

Configuration audit sessions shall be planned as work items (typically feature or user story) and follow a checklist directly in the work items. Any comments shall be recorded in the audit work item, and actions shall be tracked as children to track them to closure.

The configuration manager plans and leads configuration audit sessions, involving other roles and experts as needed and monitoring the state of the actions.

Actions are assigned to owners according to the appropriate roles, and action owners are responsible for planning and closing them.

If required for a project, data and results from configuration audit sessions can be exported and stored in a designated location, such as a document management system.

Visualize configuration audits​

Information about configuration audits can be visualized:

  • Work items: Planned audit activities, audit findings, corrective actions.
  • Dashboards: Release criteria status, traceability issues.
  • Wiki: Summary of audits and related actions.
  • Build and release pipelines: Quality gates status, approvals, and reports from tools/scripts.
Owner: Configuration Management Team