AI Governance in Safety-Critical Industries: What It Means and Who Is Accountable

The question I get asked in every enterprise procurement review now is not what the AI can do. It is who signs for it. My team built the AI inside ComplyFlow and I have led the product since 2021, so the question lands on me, and the honest answer is that it has to be a person, named.

That is a governance question, and it has a real answer in Australia today: the six practices the government publishes, the duty that never moves, and the record an auditor can follow. If you want the capability question instead, what AI can and cannot be trusted with in safety work answers that one, and the five AI skills a WHS manager should be building answers the personal version.

What AI Governance Means When the Output Is a Safety Decision.

The phrase arrived from finance. A United Kingdom skills report covering 122 job postings on four job boards in the third quarter of 2025 found AI governance, risk, compliance and quality was the second largest category of advertised AI roles, behind machine learning engineering, and the fastest growing of them.¹ The driver it names is the European Union’s AI Act. Australia has no equivalent law, so take the observation and leave the driver behind; what the AI labour market research means for a compliance team works through how much of that evidence actually travels here.

What travels is the shape of the work. Governance is not a team, a title, or a policy document. It is four things held in place: somebody named as accountable for each AI system, a risk assessment of what that system could do wrong, a check a person performs before an output changes anything, and a record that survives being asked about two years later. A safety manager builds that structure around every other hazard already.

Australia chose deliberately not to build a separate legal regime for this. The National AI Plan of 2 December 2025 said the government’s regulatory approach “will continue to build on Australia’s robust existing legal and regulatory frameworks”, with agencies and regulators keeping responsibility for AI harms inside their own policy domains.² For a safety-critical business, the binding obligations are the WHS ones you already hold.

Safe Work Australia made that explicit in July 2026. If you are a person conducting a business or undertaking, you must manage the health and safety risks from AI and digital technologies as much as you reasonably can, just as you would with any other hazard.³ The duty that bites earliest is consultation: it applies before you introduce a technology that may affect health and safety, not after the rollout email.³

One jurisdiction has gone further. New South Wales inserted a new paragraph into the primary duty of care requiring that workers’ health and safety is not put at risk from the use of digital work systems, and a new section 21A covering how such a system allocates work, naming excessive workloads, unreasonable performance metrics, excessive monitoring, and unlawful discrimination as the risks to consider. A digital work system is defined as an algorithm, artificial intelligence, automation or online platform. The amending Act was assented to on 18 February 2026 and those duties have not commenced; they start on proclamation, no earlier than one month after the regulator publishes its first guidelines.⁴ Passed and in force are two different facts.

The Six Practices, and What Each One Asks of You.

The live Australian framework is the National AI Centre’s Guidance for AI adoption, whose implementation guidance was published on 5 May 2026. It sets six essential practices, and they are worth reading beside a safety management system rather than beside a technology strategy.⁵

1. Decide who is accountable. Assign, document and communicate who is accountable for each AI system, including where contractors and third-party providers are involved, and define the competencies and authority that person needs.⁵ This is the hazard owner column in your risk register, applied to a system.
2. Understand impacts and plan accordingly. Document the scope of each system, its intended use, its foreseeable misuse, its limitations, and who could be harmed, then give those people a way to challenge an outcome.⁵
3. Measure and manage risks. Run a risk assessment for each system against your own stated risk tolerance, including risks that arrive from other parties in the supply chain, and write a treatment plan.⁵
4. Share essential information. Keep an organisation-wide register of every AI model and system in use, and tell the people affected when they are dealing with AI.⁵
5. Test and monitor. Set acceptance criteria before deployment, test against them, record the result, and get a documented authorisation from the accountable person based on those results.⁵
6. Maintain human control. Keep the ability to intervene, and train whoever oversees the system in its capabilities, its limitations, and its failure modes.⁵

Now read Safe Work Australia’s own process next to that. Risks from AI must be managed using the same process as any other hazard: identify the hazards, assess the risks, control them, and monitor and review the controls, in consultation with workers.⁶ The controls it suggests include human oversight and audits to make sure the technology is working as planned, and written policies and procedures supported by training and supervision.⁶ That is practices five and six in a regulator’s vocabulary. If you run a safety management system, you have done this before with a different noun in the middle.

Who Is Accountable When an AI Output Is Wrong.

Three answers, in the order they are usually got wrong.

Not the vendor. A WHS duty cannot be contracted out. If our document review misreads a certificate and a worker starts without a current licence, the duty holder is still the business that let them start. What the supplier owes you is the information to govern it, which is why the guidance asks developers to be transparent with the organisations downstream of them about model risks, expected behaviours, and changes.⁵ Ask for that in the tender, not after the incident.

Not the worker who was handed the output. This is the failure mode I would watch hardest for, and Safe Work Australia names it directly: workers may be held responsible for responses from AI or digital technologies which they cannot fully control, and may have limited authority to override an output or be unclear on how to do so.⁷ A reviewer who cannot say no is not a control. They are a signature. If the override path is not written down and the person is not authorised to use it, your human check does not exist.

The named person who acted, inside a business that holds the duty. That is the answer, and it only works if the name is decided in advance. The distinction worth holding is between a system that observes and a system that concludes. An agent reporting that a criterion is not identified is handing you an observation. An agent returning a verdict on whether a contractor meets the law is making a determination it is not qualified to make, and an inspector will ask a person to justify it. In ComplyFlow, agents grade documents against criteria you define and return feedback; a person acts on it.⁸ The decision about whether somebody may be on site stays with whoever holds the safety role.

What a Record an Auditor Can Follow Looks Like.

The test is simple, and it is not about the model. Can you show, two years later, what the tool was asked, what it answered, why, who checked it, and what they decided.

Six things make that record. The criteria the output was assessed against, written before it ran. The version of the system and the date it last changed. The output itself, with the reasons attached rather than a score on its own. The name of the person who reviewed it. What they verified, in one line. And the date.

Criteria are the one worth designing for, because criteria written afterwards are a rationalisation. We put that in the product on purpose in the AI Tools release of 29 June 2026: you define the review criteria in a wizard, you can run an agent against a real sample document and read the actual result before you save it, a contractor can see the criteria their answer will be assessed against before the review runs, and every review is recorded in the AI Log.⁸ Pre-deployment testing, transparency, and a record, which is practices four and five arriving as buttons.

The second is access. If an assistant can read your compliance data, an auditor will ask what it could reach. ComplyFlow’s MCP server is read-only, so a connected assistant can look up and summarise but cannot create, change or delete anything, and it runs on a scoped token that expires and can never see more than the person who created it.⁹ That is a governance answer rather than a feature, and your compliance and legal team should ask every vendor for the equivalent. The setup is documented in the ComplyFlow Help Centre.

The Formal Option, and When It Earns Its Place.

If somebody wants a certificate rather than a practice, the standard exists. AS ISO/IEC 42001:2023 identically adopts ISO/IEC 42001, and sets requirements for establishing, implementing, maintaining and continually improving an AI management system. Standards Australia published the Australian edition on 16 February 2024, adopting the international standard of 18 December 2023.¹⁰

It is built the same way as the management system standards your procurement team already reads, so it slots into an existing audit cycle rather than starting one. My honest view from the buying side: certify when a customer or an insurer asks for it in writing. Until then the six practices give you the same structure, and the gap between an organisation that has done them and one that has not is visible without an auditor.

Where to Start, and What It Costs You.

Take one page to your next WHS committee. List every AI tool that touches a safety or compliance decision, including the ones nobody approved. Against each, write the accountable person, the named human check and where it is recorded, and whether the output could change a decision about a person’s right to be at work. That last column decides how much governance the tool needs.

Then add AI to the risk register as a hazard with controls, and get the consultation minuted before the next tool arrives. That is a morning’s work, and it is the whole of practice one.

The reason to do it this quarter is not that a law is coming, because in Australia it mostly is not. It is that the first time somebody asks you to explain an automated decision, it will be in an incident investigation, and the record either exists by then or it does not.