Skip to main content

How-to conduct internal FSM Audits

This guideline aims to describe how to conduct internal Functional Safety Management (FSM) audits in ABB PCP R&D, as well as how to monitor and handle external FSM audits conducted by the Independent Safety Assessor (ISA) for ABB PCP R&D.

General​

Please find the generic description of the FSM audit here.

Intended for​

Internal FSM auditors and coordinators.

Purpose​

The holder of the FSM certificate Q4B 029049 0007 is ABB PCP R&D. The certificate covers the PCP R&D sites in Västerås, Malmö, Minden, and Hangzhou. Other parts of PCP, after agreement with ISA, can develop interference-free products or functions, perform different phases of development or maintenance, or serve as a sub-supplier that produces safety-classified hardware for ABB’s SIL3-certified products, such as the AC 800M HI controller. To achieve this, there is a requirement from ISA to perform internal audits of these sites to verify adherence to the defined development process and to assess the suitability of these sites for interference-free development.

In addition to the internal audits conducted for sites outside the FSM certificate, internal audits will be conducted for sites situated within the FSM certificate (Västerås, Malmö, Minden, and Hangzhou) as a means to perform sanity checks on the SIL development sites.

For the external audits conducted by the ISA, the resulting findings shall be handled appropriately and monitored within ABB PCP R&D. This is also in the scope of the FSM for ABB PCP R&D.

ISA conducts internal functional safety audits to ensure compliance with functional safety processes (QMS) as per IEC 61508 by the development organization.

Scope​

The audit shall focus on safety-related releases (SIL and interference-free) and general knowledge of ABB PCP R&D processes.

The following topics are suggested to be addressed in the agenda:

Planning​

Requirement​

According to IEC 61508-1-6.2.7: “Requirements for periodic functional safety audits shall be specified”. The following applies to ABB PCP R&D regarding the frequency of functional safety audits:

This has been interpreted into requirements regarding:

  1. The frequency of the functional safety audits. The following applies:

    • Internal Audits (performed by the ABB Safety Team): Conducted annually per site.

      Note that any deviation requires official approval from ISA.

    • External Audits (by ISA): Once a year. The current audit sites consist of Bangalore, Minden, Malmö, Västerås, and Hangzhou. The ISA can pick any of the sites for an FSM audit. In principle, the external FSM audit could be combined with the internal FSM audit in the same year.

  2. The level of independence of those carrying out the audits, where the ABB approach is covered by Independence

  3. The necessary documentation and follow-up activities, where the ABB approach is covered by the Documentation.

Preparation for planning​

It is required to have an annual audit plan for all sites ready by the end of Q1, including the appointed auditor and co-auditor for each site, as well as the quarter or SPI increment in which the audit will take place. A detailed audit schedule shall be planned in collaboration with the audit site and local coordinator at least eight weeks before the audit.

note

The audit plan should be approved by the FSM Manager and shared with the ISA. Any deviation shall be approved by the FSM Manager and communicated to the ISA.

Ideally, both the auditor and the co-auditor shall be present at the audit site during the FSM audit. If this is not possible due to travel restrictions or other logistical issues, then virtual participation shall be arranged instead.

When planning for the audit, there are some key points to take care of:

  • A local coordinator shall be appointed, and he/she shall be responsible for ensuring the availability of relevant personnel.
  • Avoid having more than two audits in the same increment.
  • The auditors shall develop the audit plan in collaboration with the local coordinator.
  • The auditors, together with the local coordinator, shall review the status of any relevant global and all local findings.
  • The releases to be audited shall be discussed and agreed upon before the audit. It shall be documented in the plan.
  • The plan, along with an agenda, shall be sent to the local organization to enable their preparation.
  • It is the responsibility of the co-auditor or local coordinator to document the evidence and related documents (including document ID) during the audit. The FSM Audit Session Record Template is recommended for use during the audit sessions.

Examples of plans can be found in the audit folders in the FSM audits.

Independence​

Individuals conducting a functional safety audit must adhere to a minimum level of independence as specified in IEC 61508-1. The minimum level of independence shall be assessed according to the following:

Table 5 from IEC 61508-1
Table 5 from IEC 61508-1

Note ABB PCP R&D orients itself for internal audits regarding the rules for assessments as defined by Table 5 in IEC 61508-1. Any deviations from this orientation are documented with a corresponding rationale.

The minimum level of independence shall be based on the highest applicable safety integrity level developed at the site.

When performing internal audits for the sites within the FSM certificate (Västerås, Malmö, Minden, and Hangzhou), the level of independency according to the standard IEC 61508 may be challenging to fulfill, since the organization is developing up to SIL3. When performing internal FSM audits at those sites, the highest level of independency possible within the sites shall be strived for.

Auditing​

General information regarding the organization/release management​

This topic aims to provide general information about the organization and knowledge of release management about the QMS process.

The topic can include auditing of:

  • Follow up on releases, such as KPIs and Dashboards, in Azure DevOps, etc.
  • Management of information/escalation/findings from the releases within the organization
  • Management and distribution of information/escalation/findings from outside the organization to the organization
  • Communication (release-wise) with other stakeholders (both within other ABB organizations and with external organizations, for example, sub-suppliers)

The session covering this topic can be conducted as an interview of and/or presentation by the Release Owner and/or Cluster Lead.

The audit team will determine the selection of specific questions for this audit session and may vary depending on the focus of a particular audit. Here follows some examples of what could be answered:

  • What does the organization look like? How many employees work there? How are you divided internally? How many employees are involved in the safety-related releases? What do you work with?
  • How do you follow up your releases? KPIs, Dashboards in Azure DevOps, etc.
  • How do you manage information/escalation/findings from your releases within your organization?
  • How do you manage and distribute information/escalation/findings from outside your organization to your organization?
  • How do you communicate (release-wise) with other stakeholders (both within other ABB organizations and with, for example, sub-suppliers)?
  • Does the organization hold any other relevant certifications, such as those for Quality Management?

This shall be answered by providing some examples/evidence.

Follow-up of findings from the last audit​

This topic aims to evaluate the organization’s ability to implement (mandatory) improvements into the organization.

The topic can include auditing of:

  • Completion and implementation of findings from the last audit:
    • Is everything handled and motivated? If not, why not?
    • Are the motivations for closing a finding clear enough?
    • Are the findings implemented in the organization?
  • Implementation of findings from relevant global findings from other sites.

The session covering this topic can be conducted as an interview of and/or presentation by the Local Coordinator, Audit Finding Responsible, and/or Cluster Lead.

This will be addressed through a walkthrough of the findings list. Priority shall be given to items that have not been handled. The audit team can request evidence for some handled findings if needed.

QMS overview​

This topic aims to evaluate the organization’s ability to follow procedures and processes.

The topic can include auditing of:

  • Usage and updates of QMS
  • Training log
  • Introduction of new co-workers
  • Interview with some co-workers

Usage and updates of QMS

The session covering this topic can be conducted as an interview of and/or presentation by R&D Management, Release Owner, Quality Control Manager (QCM), and/or Quality Responsible.

The following shall be answered:

  • How is the knowledge of QMS implemented?
  • How is the knowledge of the updates of QMS implemented?
  • How is the knowledge of the updates of QMS evaluated regarding understanding them?

This will be addressed through a walkthrough of the updates from the last audit, where R&D Management and QCM can present answers to the questions, both in general and by providing specific examples.

The QMS baselines are stored in PCP R&D Processes and can be found on the front page by hovering over the PDF button.

Training log

The session covering this topic can be conducted as an interview and/or a presentation by the Training Log Responsible.

The following shall be answered:

  • Is the training log available and updated regularly?
  • How was the training organized and conducted since the last audit?
  • If gaps are identified:
    • What are the reasons for the gaps?
    • What are the consequences of the gaps?
  • Training coverage at the site regarding Safety and QMS for employees involved in safety and/or interference-free development
  • Are there any training sessions missing that the organization needs?
  • Are there any local trainings that could be beneficial for other organizations?
  • How do you introduce new co-workers?
  • If used, mentoring as a training format should also be logged

This will be addressed through a walkthrough of the training log and interview, where the responsible party will provide answers to the questions.

Interview with some coworkers

The session covering this topic can be conducted as an interview of R&D Team Members participating in safety development and/or interference-free development.

For instance, one junior developer, one manager, etc., can be selected to answer the following during a short interview regarding their roles:

  • What do you do? What experience do you have? Etc.
  • Focus on the question “What do you do if a problem arises during your workday/in a release?”

This session then aims to target the implementation and understanding of the QMS processes, organization, and receipt of information on QMS updates and changes.

The interviews will be conducted and recorded anonymously to prevent targeting individuals. The focus shall be on the possible improvement of the organization and the existing QMS.

Release detailed development within the organization and detailed release work​

This topic aims to evaluate the release development within the organization to verify adherence to the defined development processes and suitability for performing safety-related development.

This topic can include auditing of currently running releases at the site:

  • Adherence to the (IEC 61508) Safety Lifecycle and Safety Handbook
    • Modification/change management, including reviewing impact analyzes (IAs), handling of Azure DevOps work items such as document updates (DUs) and bugs, as well as CM handling such as version management
    • Tailoring of a release
    • Traceability
    • Formalities (for instance, correct templates & correct test equipment)
    • Documentation
    • Follow-up of actions and remarks
    • Tool selection
    • Discrepancies about QMS
  • Release management
    • STECO & CCB meetings and reporting, requirement change, action list, etc.
  • Quality management
    • Quality activities and responsibility, Quality Plan and assessment, etc.

The sessions covering this topic can be conducted as interviews with and presentations by the different release owners and members.

These general questions shall be answered:

  • How is/was a/the actual release initiated at the site?
  • How do you tailor a release?
  • Why are some activities/documents omitted? What is the motivation?

New development release - full safety life cycle

A development release will go through the phases of the Safety Lifecycle, as described in the Safety Handbook.

The audited releases can be reviewed following the development life cycle from the entry point to the exit point. This can be achieved through both general presentations provided by the release owners and members, as well as by sampling key documents, ADO work items, and other relevant materials. The release owners and members shall be prepared to demonstrate all aspects of the release.

Rollup release - tailored safety life cycle

The checklist for rollup releases needs to be followed. The template can be found in Templafy by searching for 3BSE040876.

In a rollup release, there are fewer phases than in a full release:

  • Initiating = G0 → PHASE 1 – Requirements Definition
  • Startup = G1-G3 → PHASE 1 – Requirements Definition, PHASE 2 – Analysis and Functional Design, PHASE 3 – Detailed Design
  • Release = G3-G5 → PHASE 3 – Detailed Design, PHASE 4 – Implementation/Manufacturing, PHASE 5 – Module/Design and low-level integration Test, PHASE 6 – Function/Component Type Test, PHASE 7 – Validation PTT/SVT
  • Release Closure = G6 (focuses on following up open actions from G5)

For a safety release (including interference-free), the following documents must be maintained according to the correct templates and review process: Quality Plan, Names on RACI, CM Plan, Detailed Test Plan, and Impact Analysis Report.

Modification/Configuration Change Control in TFS/ADO

Which items that shall be under formal configuration control shall be documented in a Product CM Plan (preferably) or the CM Plan for each release. The following items are subject to formal configuration change control:

  • Requirements
  • Technical Product Documents (e.g., Descriptions of Functions, Design Descriptions, Test Descriptions, etc.)
  • Code modules and other sources for generating the product
  • Test code for testing the product
  • Code to build the product
  • User documentation
  • Manufacturing documents
  • Version Documents
  • Tool Management

For all baselined work products, Formal Change Control shall be applied to preserve the determined quality and contents. For code, this means Formal Change Control on all code modules that can affect the baselined implementation.

Documentation​

Audit report​

The audit shall be documented in a report, where the relevant information from all audit days is summarized and the findings are initially documented. This report shall then be distributed to the local organization as well as the ISA.

Examples of audit reports can be found in the audit folders in the FSM audits.

It is highly recommended that the approved audit report be submitted to the ISA within 3 weeks after the audit is conducted.

Findings & follow-up​

All findings should be classified as local or global. The definitions are as follows:

  • Local finding: Found only in the local site, no global process issue
  • Global finding: Cannot be solved locally and should be solved on a global level

The findings should also be assessed as minor or major. The motivation of the classification/criticality shall be documented and stated during the audit summary.

All findings are currently documented and tracked in Azure DevOps, in addition to being initially noted in the audit report. The work item type to be used for a finding is the Feature, and there should be a link to a parent Epic for the audit itself. The feature shall be tagged with "FSM Audit Finding" to facilitate the establishment of a query and further analysis of the total findings.

For global findings, a tag ‘global’ should be created, which can be followed up further in a global query for all sites.

The respective finding shall be assigned a target date for closure, depending on urgency and complexity. For major findings, a due date of 4 months after the audit should be set. For minor findings, a due date of 10 months after the audit should be established.

The local coordinator from the audited ABB PCP R&D site is responsible for planning and implementing corrective actions to address any findings identified in the audit. All reporting is to be done in Azure DevOps. The finding shall contain:

  • The rectification of the finding
  • An analysis of the reason(s)/root cause(s) of the finding
  • Proposal(s) to the organization for additional improvements to ensure that the finding will not recur
  • For global findings, the local coordinator shall consider the globally implemented corrective actions at the local site.

The listed bullets shall be included as Acceptance Criteria for the feature in Azure DevOps and shall be completed before closing the finding.

If the finding can be avoided in the future by a process change, then create a PCR and link it to the finding.

In the audit report, the latest date by which all corrective actions shall be resolved will be specified.

The audit team shall conduct a follow-up on the findings. This shall be done both continuously in Azure DevOps and by booking meetings with the local coordinator throughout the year. It is the audit team that appoints a Safety Team member to follow up on the actions and update the Safety Team and ISA.

When any finding is found to be open until the next audit, the following actions will be triggered:

  1. Escalation to the direct manager of the person responsible for the finding will be triggered. The manager needs to conduct an assessment, make a decision to close the finding or defer it, and provide justification for why the finding cannot be completed before the next audit.
  2. The safety team will submit this finding with higher priority during the next SPI planning meeting to highlight the importance of resolving the issue.

In addition to the findings, some positive observations can also be made. These are findings that should give the local organization credit and be shared with other sites as examples of best practice.

Improvements​

Identified improvements will not be actively followed up on. The reason for proposing an improvement instead of reporting a finding may be that the observed issue has not yet become a problem, and the severity of the consequence for this issue is low. These proposed improvements also serve as an input to future audits.

External FSM audit

For findings made by the ISA, the coordinator from ABB PCP R&D is responsible for resolving them. All reporting must be done in Azure DevOps. The finding shall contain:

  • the rectification of the finding
  • an analysis of the reason(s)/root cause(s) of the finding
  • proposal(s) to the organization for additional improvements to ensure that the finding will not recur

The listed bullets shall be included as Acceptance Criteria for the feature in Azure DevOps and shall be completed before resolving the finding.

Evidence and motivation for closing shall be presented to the ISA. The ISA will then decide whether the finding can be closed. If agreement is reached, the coordinator from ABB PCP R&D can mark the finding as closed.

Annex​

FSM Audit Session Record Template​

Session:
Add the title of the session from the audit agenda here.		Date / Time:
Note the date and time of the session, as they may differ from the audit plan.
Participants:
Add names of participants and add notes such as "part-time", "unavailable" etc. if needed.		References:
Add titles and identification of any reference material.
Notes:
Use this section for notes and follow-up actions.		Findings:
Use this section for findings (global/local, major/minor) and observations.
Owner: Functional Safety Team