AI and the Australian Privacy Act: Obligations When Using AI Systems
Most Australian organisations that use AI systems think about privacy only when processing personal data for their primary purpose—a bank training a credit model on customer records, for example. But the Privacy Act 2024 reaches far beyond primary purpose. It covers how you source training data, what you do with it, who else can access it, how long you keep it, and how you handle automated decisions about individuals. Miss these obligations and you’re facing OAIC enforcement, penalties up to AUD 50 million or 10% of turnover, and reputational damage.
Overview: Privacy Act 2024 and AI Processing
The Privacy Act doesn’t have an “AI exception.” When you use AI to process personal information, you’re subject to all 13 Australian Privacy Principles, regardless of whether the processing is novel or traditional. The Act focuses on how you *handle* information, not the technology you use to handle it.
However, AI creates unique Privacy Act risks. Machine learning models often require large volumes of training data and can process personal information in ways even their builders don’t fully understand. Models trained on one dataset may be repurposed for uses far beyond the original collection. Automated decision-making using AI can affect individuals unfairly if the underlying data or model is biased. December 2026 brings new transparency obligations specifically for automated decisions.
The OAIC’s 2024 AI and Privacy guidance makes clear: organisations can’t treat AI as exempt from privacy obligations. If anything, AI increases your obligations because it amplifies risks.
Privacy Principles and AI: The Critical Pathways
APP 1 (Open and transparent management). You must have a clearly expressed and up-to-date privacy policy that explains how you handle personal information. If you use AI, your policy must explain: what AI systems process personal data, what purposes they serve, what personal information they use, who they’re shared with, and how individuals can access or correct information used by AI. Many organisations’ privacy policies predate AI and don’t mention it. That’s not compliant.
APP 3 (Collection of solicited personal information). You must collect personal information only when reasonably necessary for your functions. If you’re training an AI model, collect only the data the model actually needs. If your hiring AI needs job history and skills but not ethnic background, don’t collect ethnic background “just in case.” This is data minimisation, and it’s a Privacy Act obligation, not optional good practice.
APP 5 (Notification). When you collect personal information, you must notify the individual about the collection and use, including if AI will process it. If you’re training a model on employee data for performance prediction, employees must be told: that their data will be used for this purpose, that AI will process it, and how long you’ll keep it. Collecting data “for HR purposes” then using it for an undisclosed AI model is a breach.
APP 6 (Use and disclosure). You must use and disclose personal information only for purposes the individual was told about, or related purposes. If you collected data for hiring decisions and then repurpose the model to assess contractor potential, that’s likely an APP 6 breach unless individuals were told about both uses upfront. The OAIC has found that organisations often violate APP 6 by using training data for purposes beyond what individuals consented to.
APP 11 (Security). You must take reasonable steps to protect personal information. If your AI training data includes sensitive personal information and it’s stored insecurely or accessible to unauthorised staff, you’ve breached APP 11. If you’re outsourcing AI training to a cloud provider, ensure your contract specifies security standards and audit rights.
APP 12 (Access and correction). Individuals have the right to access personal information held about them and correct it if it’s inaccurate. If AI training data includes personal information and an individual asks to see or correct it, you must comply. This is often overlooked: an individual who knows their data is in your training set and requests access should get it (with appropriate redactions for others’ data). Correction gets trickier: if the individual’s data is embedded in a trained model, correction may require retraining. Document this complexity.
APP 13 (Correction, access and complaints). Individuals have the right to complain about privacy breaches. If they do, you must respond within 30 days. If your AI system makes an unfair decision about someone and they complain, you must investigate, even if the AI’s decision was “accurate” by its own logic.
Automated Decision-Making: The December 2026 Transparency Obligations
From December 3, 2026, a new obligation arrives: if an AI system makes a decision that has the potential to significantly affect an individual, you must take steps to ensure the individual can ask for an explanation and has a mechanism to seek review. This applies to decisions about credit, employment, education, housing, and other significant life domains.
What counts as “significantly affecting” someone? The OAIC guidance suggests: decisions that deny, restrict, or disadvantage the individual; decisions that profile or assess the individual for risk or other attributes; decisions that treat the individual differently from others. In practice, this covers most consequential AI decisions.
Your obligation: Provide the individual with information explaining the decision, including the factors the AI considered, any personal information it used, and the basis for the decision. You must also provide contact information so the individual can request review or express concerns. If the decision was based solely on automated processing and it has a legal or similarly significant effect, you must offer human review unless there’s a contractual relationship already specifying this (like a customer’s account terms).
This is moving Australia closer to the EU’s GDPR automated decision-making regime. The December 2026 date is less than nine months away. Organisations using AI for hiring, credit, insurance, or welfare assessment decisions must audit their systems now to ensure they can provide meaningful explanations.
Data Minimisation in AI Training and Prompts
Data minimisation isn’t just good hygiene; it’s a Privacy Act obligation. The principle: use only the personal information you need for the stated purpose. In AI training, this often fails.
Consider a retail bank training a model to predict customer churn. It collects purchase history, account balances, transaction frequency, customer service interactions, and behavioural indicators. It then trains the model on five years of customer data. But does it actually need five years? Or all those interaction logs? Probably not. Data minimisation means: train on only what improves model performance, and only for as long as necessary. If you can achieve the same accuracy with one year of data, use one year.
Similarly, in LLM prompts, don’t include personal information you don’t need. If you’re using an AI to draft a customer service response, don’t include the customer’s full interaction history if only recent messages matter. That reduces both privacy risk and cost. And if you’re using a cloud-based LLM (ChatGPT, Claude, etc.), be aware that your prompts may be used to train the provider’s future models unless you’ve signed an agreement restricting this.
Retention also matters. How long do you keep training data? If your model is trained once and then used in production, you may not need to retain the training data indefinitely. The OAIC guidance suggests: delete training data once the model is trained and validated, unless you have a separate, documented reason to keep it (e.g., for retraining schedules).
Cross-Border Data Transfers and Cloud AI
Many organisations use cloud-based AI services: OpenAI’s API, Google Cloud AI, AWS AI, or Australian alternatives. If you’re sending personal information to these services, you’re making a cross-border data transfer, and APP 1 and APP 1A (outbound transfers) apply.
APP 1A requires that before you transfer personal information overseas, you must take reasonable steps to ensure the recipient doesn’t breach the APPs in substantially similar ways to the Australian Privacy Act. This is a high bar. If you send customer data to a cloud AI provider in the US, you’re responsible for their privacy practices. If they process it insecurely, breach access controls, or use it for undisclosed purposes, *you’re* the one liable under the Privacy Act.
Practically: ensure your AI service provider has a privacy commitment matching Australian standards, has encryption and access controls, limits data retention, and commits not to use your data for AI model training unless you’ve explicitly agreed. Get a written agreement (DPA or similar) specifying these terms. If the provider won’t commit to these standards, consider an Australian alternative or an on-premises solution.
FAQ
If I’m using an off-the-shelf AI tool to draft emails, do I need to worry about the Privacy Act? Only if you’re sending personal information in the prompts. If you’re drafting a generic response template, no. If you’re asking the AI to summarise a customer’s full interaction history, yes—you’re transferring personal information overseas, and APP 1A applies. Use only the information you need.
Can I train an AI model on anonymised data without Privacy Act obligations? If the data is truly anonymised (permanently irreversibly de-identified), the Privacy Act doesn’t apply to it. But “anonymised” is a high bar. Hashing names or masking obvious identifiers isn’t anonymisation if someone could re-identify individuals by combining the data with other sources. Get a privacy expert to assess if your data is truly anonymised before assuming it’s out of scope.
What happens if an individual asks me for all their personal information used in an AI training dataset? Under APP 12, if you hold personal information about them, they have a right to access it. If they’re asking for data used in a training dataset, you must provide it (redacting others’ data as necessary). Practically, large datasets can make this complicated—work with a privacy lawyer on how to respond proportionately without refusing the request altogether.
Do I need consent to use AI for hiring decisions? Not explicit consent, but you must notify candidates (APP 5) that AI will be used to assess their application, explain how it works, and ensure the decision-making is fair under discrimination law. You must also offer human review if the candidate asks. The Privacy Act and discrimination law work together here.
Conclusion
The Privacy Act 2024 treats AI as just another information processing technology, but one with multiplied risks. Australian organisations deploying AI systems must audit their data handling against each APP, design data minimisation into training workflows, manage cross-border transfers carefully, and prepare for the December 2026 automated decision-making transparency obligations. Privacy isn’t blocking AI; it’s building AI responsibly.