From automating the drafting of marketing copy to helping developers write code, enterprises are embracing AI at an increasing rate. According to recent research, 78% of organisations now use some form of AI, and for every dollar invested companies report a $3.70 return.
However, alongside the enthusiasm sits a growing unease. Nearly seven in ten enterprises cite AI-powered data leaks as their top security concern, while almost half operate without any AI-specific security controls in place.
This dual reality, with unprecedented potential on one hand and mounting risks on the other, defines the GenAI debate. The challenge for boards and executives is to capture the benefits while managing the dangers, and this is a balancing act that will shape enterprise strategy during the coming decade.
When AI innovation goes wrong
Recent incidents highlight the perils of moving too fast without guardrails. In May this year, researchers uncovered exposed data belonging to 571 customers of data integration specialist JedAI. The breach was traced back to an unsecured public-facing database, revealing both the dangers of shadow AI and poor governance.
Another cautionary tale could be read about technology giant Samsung. In April 2023, the company suffered three separate leaks in a single month, including source code and chip optimisation data.
Some of that material is now suspected to have been ingested into large language models, raising concerns about “introspection attacks” where sensitive training data can be extracted.
Meanwhile Microsoft’s Copilot suite has faced a series of vulnerabilities, including so-called zero-click exploits that allowed attackers to access internal systems via email or collaboration tools. In some cases, flaws in the Copilot Studio platform enabled attackers to exfiltrate data or launch further intrusions.
Collectively, these incidents underline a new risk category for the enterprise: data exposure through training, prompt injection attacks, and unmonitored shadow AI use.
It is a known fact that GenAI adoption is accelerating across all markets, with organisations rapidly integrating AI into operations without regional differences in uptake. However, awareness of the risks and challenges associated with this adoption remains relatively immature, as many view AI purely as a business benefit without recognising its impact on security and data integrity.
This widespread reliance on AI increases the attack surface and, without strong data quality controls, can lead to flawed analyses and poor decision-making.
Building security and governance foundations
Experts argue the solution is not to slow AI adoption but to embed risk management from the outset. Having clear guardrails allows companies to innovate quickly while reducing the chance of damaging missteps.
Frameworks are also emerging to provide structure. The US National Institute of Standards and Technology (NIST) released its AI Risk Management Framework 1.0, which sets out four steps: govern, map, measure, and manage.
This approach includes establishing accountability structures, identifying AI-specific risks, deploying continuous monitoring, and implementing systematic responses.
Meanwhile, the MITRE ATLAS framework maps the adversarial threat landscape, helping organisations to threat-model AI systems, conduct red-team exercises, and build effective detection rules.
Both approaches are designed to integrate with traditional cyber security controls, giving enterprises a starting point for developing AI-specific defences.
Governance in practice
The most effective organisations are moving beyond just having policies to building cross-functional governance structures. Four pillars are emerging as critical: data strategy and leadership, architecture and integration, governance and quality, and culture and literacy.
Enterprises must ensure real-time data integration, deploy bias detection tools, track data lineage, and apply privacy-preserving techniques.
Just as importantly, staff need clarity on what is and isn’t allowed. Surveys suggest that employees often turn to consumer AI tools simply because workplace policies are vague. By providing clear frameworks, companies can accelerate adoption while staying within safe boundaries.
The regulatory maze
If technology leaders are grappling with governance, regulators are equally busy. The European Union’s AI Act, which came into force in February this year, has already become the de facto global standard.
The Act bans certain high-risk uses outright and imposes strict conformity assessments on others. Since August of this year, all high-risk AI systems have had to undergo detailed risk assessments and mitigation planning.
The United States, by contrast, has taken a fragmented path. After revoking its federal AI Safety Framework, regulation has devolved to the states. By the end of 2024, 45 states had passed laws, with another 250 pieces of legislation expected this year. The result is a complex tangle of obligations for businesses operating nationally.
Australia sits somewhere in between. Canberra’s current policy framework requires agencies to appoint accountable AI officials and publish transparency statements, while a voluntary AI Safety Standard sets out ten guardrails.
Organisations should begin preparing now for the likely implementation of the EU AI Act or similar regulations in Australia. Many are rushing headlong into AI adoption without governance frameworks, documentation, or clarity on data lineage and access. If organisations do not start documenting and building governance structures today—covering issues such as anonymisation, data inputs, and regulatory disclosure requirements—they risk having to unpick and redo their AI initiatives within six to 12 months when regulation arrives, potentially from bodies like APRA. Proactive preparation will not only ease compliance but also mitigate the growing number of breaches and consumer-law issues already emerging from poorly governed AI systems.
Proposed legislation would make these requirements mandatory. The government is aiming for a framework that evolves alongside technology, thus balancing accountability with innovation.
The road ahead
Australian enterprises find themselves at a crossroads. On one side lies the promise of faster innovation, improved productivity and enhanced competitiveness. On the other lies the risk of data leaks, reputational damage and regulatory penalties.
It’s clear that the GenAI revolution is no longer theoretical. It is here, it’s being adopted at scale, and it’s reshaping how enterprises operate. For Australian business leaders, the question is not whether to use generative AI, but how to harness its benefits responsibly.
http://itwire.com/guest-articles/guest-opinion/balancing-innovation-and-risk-with-genai.html