Your Face, Their Data: Is New Zealand's Privacy Law Keeping Up with AI?
by Matt Childs
I Introduction
Think about the last time you applied for a job, asked your favourite generative AI tool a question, or walked into your local supermarket. Chances are, somewhere in that process, an algorithm collected data about you without you ever knowing. Artificial intelligence is making decisions about people's lives at an unprecedented scale and speed. New Zealand's main framework for protecting personal data, the Privacy Act 2020, was enacted before much of this became routine.
Part II of this article outlines New Zealand's existing privacy framework. Part III explains how AI has transformed personal data collection, including the specific challenges posed by large language models. Part IV examines real-world privacy concerns arising from AI in New Zealand. Part V evaluates the adequacy of the current legal framework, and Part VI considers the enforcement and access to justice implications of relying on it.
II New Zealand's Privacy Laws
The Privacy Act 2020 (the Act) is New Zealand's main tool for protecting personal data. It sets out thirteen Information Privacy Principles (IPPs), covering how organisations collect, store, use, and share personal information. In practical terms, these rules govern everything from why an organisation can collect your data in the first place to who it can share it with. The Act also introduced mandatory breach reporting and gave the Privacy Commissioner powers to investigate and require remedial action.
The framework has continued to develop. In 2025, the Privacy Amendment Act added IPP 3A, requiring organisations to notify you when your personal information is collected indirectly, through data brokers, third parties, or scraped from public sources. This places new obligations on organisations using AI to gather data about individuals. The key limitation is that this requires notification, not consent. You will be told your data is being collected, but you will not have a say in whether it should be.
Also in 2025, the Privacy Commissioner enacted the Biometric Processing Privacy Code 2025. This is New Zealand's first specific set of rules for governing technologies like facial recognition. Organisations now have an obligation to demonstrate that any biometric data collection is necessary, effective and proportionate before deploying it. Non-compliance can result in a compliance notice from the Privacy Commissioner, which is typically published publicly; and can lead to referral to the Human Rights Review Tribunal, which has powers to make orders and award damages
III AI and the New Age of Data
With that framework in mind, it is worth understanding exactly what AI does with personal data and why existing law struggles to keep up.
Data collection is not new. Businesses have tracked purchasing habits, browsing behaviour, and customer preferences for decades. What AI has changed is the scale, speed, and invisibility of that process. AI systems can now aggregate and combine data from multiple sources, such as your location, browsing history, and purchasing patterns. None of this requires your direct participation.
What makes this particularly significant is the shift from data you choose to share to data that gets inferred. AI systems do not just record what you tell them. They draw conclusions about who you are from information you may not even know they have access to. Amazon's AI recruitment tool, scrapped in 2018 after four years of use, illustrates this well. The tool inferred applicants' genders from indirect signals on their CVs, such as membership of women's organisations, and used those inferences to screen them out without their knowledge. The data people shared for one purpose was used to draw conclusions they never agreed to and had no way to challenge.
IV The LLM Problem
Large language models, or LLMs, are the technology behind tools like ChatGPT, and they raise a distinct set of privacy concerns. These models are built on vast datasets scraped from the internet which can contain personal information: names, email addresses, social media posts, and other identifying details that people posted publicly, but never intended to contribute to a commercial AI system.
The deeper problem is what happens once that data is inside a model. LLMs can memorise segments of their training data and reproduce them in responses, meaning a model might output someone's personal information when prompted in the right way, without that person ever knowing their data was included in the first place. Unlike a database, you cannot easily query where information came from or remove it once it is embedded in a model's parameters. In the European Union, individuals have a legal right to erasure, which allows them to request the deletion of their personal data. New Zealand has no equivalent, and removing information from a trained LLM may require retraining the model entirely, something no company will do for an individual request. Once your data is in, it is likely there permanently.
The same problem extends to everyday interactions with these tools. When using most chatbots, your conversations, by default, are stored and used to further train the model, unless you specifically opt out. Many people feed sensitive personal information into these tools without any awareness that they may be doing so permanently.
V AI-Related Privacy Concerns in New Zealand
These are not abstract concerns. AI-driven data collection is becoming a feature of everyday life in New Zealand, and its privacy implications are already being tested.
Facial recognition technology (FRT) in retail is the most visible example. In 2024, Foodstuffs trialled the technology across 25 supermarkets, scanning over 225 million faces and matching them against a database of known offenders. There were 117 potential misidentifications, nine of which resulted in people being wrongly confronted. The most publicised of these incidents was the misidentification of a Māori mother at a Rotorua New World. While shopping at her local supermarket, she was approached by two staff and told to leave. The FRT alert that led to her being confronted was a low quality image that produced only a 90.54% match. Consumer New Zealand’s Chief Executive, Jon Duffy, described the technology as "highly invasive”, and “like using a sledge hammer to crack a nut”.
The Privacy Commissioner reviewed the trial and found it compliant with existing law. The Privacy Act's broad principles were satisfied, despite the glaring accuracy and proportionality concerns. The Commissioner acknowledged he could not be "completely confident" the technology had addressed its bias issues, including potential impacts on Māori and Pasifika people. Because our privacy law contains no specific requirements around algorithmic accuracy or bias testing, there was no legal standard the trial could be found to have breached.
Facial recognition in retail has not slowed down since the Foodstuffs trial. Bunnings has since rolled out the technology across all 34 North Island stores, with South Island rollout planned. Bunnings states it completed a Privacy Impact Assessment, engaged a Māori digital sovereignty expert, and designed its approach to comply with both the Privacy Act 2020 and the Biometric Processing Privacy Code 2025. This constitutes a more directed effort at protecting the privacy of its customers, and thus far there has been no reports of misidentification.
VI Evaluating the Framework
The New Zealand government has been explicit that it prefers a "light-touch, proportionate and risk-based" approach to AI regulation, relying on existing frameworks rather than passing AI-specific legislation.
The advantage of this approach is flexibility. The IPPs are broad enough to apply to privacy risks that were not envisioned when they were written, and the Biometric Processing Privacy Code shows this working in practice: a specific, enforceable response to a technology that general privacy law could not adequately address on its own.
The difficulty is that a reactive model means the law tends to arrive after the harm has occurred. The Biometric Code came after facial recognition trials had already run. IPP 3A addresses indirect data collection that had been happening without any notification obligation for years. For LLMs, no equivalent targeted response exists yet, meaning the specific privacy risks they create, including training data scraping and the absence of any right to erasure, remain largely outside the reach of existing law.
The contrast with other jurisdictions is worth noting. The European Union's AI Act proactively bans certain high-risk applications, requires pre-deployment testing, and specifically prohibits compiling facial recognition databases by scraping images from the internet or CCTV footage. Australia is also moving forward, with new amendments to its privacy law requiring organisations to disclose the use of automated decision-making where it could significantly affect an individual's rights or interests. New Zealand has no equivalent obligation.
The scale of the problem is reflected in the numbers. Privacy complaints rose 21% between 2024 and 2025, reaching 1,598 cases, many involving algorithmic bias and opaque data practices. The Privacy Commissioner himself has called for financial penalties for serious breaches, a right to erasure, and mandatory transparency around automated data processing.
VII Enforcement and Access to Justice
Even where the law provides rights, enforcement depends on people knowing those rights exist and being able to act on them. The primary avenue for redress under the current system is lodging a formal complaint with the Privacy Commissioner. This assumes you know a breach has occurred, which with AI systems is rarely guaranteed. A person whose face was incorrectly matched to an offender database, or whose personal data was scraped to train a commercial AI model, may never find out it happened. The Biometric Processing Privacy Code does require organisations to have complaint-handling procedures in place, but a procedure is only useful to someone who knows how to invoke it.
This is not a neutral problem. Amazon's recruitment tool did not target women explicitly. It targeted proxies for being a woman, and the women screened out had no way of knowing it happened. The New Zealand Law Society has warned that "black box" algorithms, where the system's internal reasoning is hidden even from its operators, make it harder for individuals to understand whether decisions affecting them are fair and lawful, let alone challenge them.
The bias embedded in AI datasets compounds this further. Māori AI and data ethicist Karaitiana Taiuru has highlighted that facial recognition systems trained primarily on European datasets are less accurate when applied to Māori and Pasifika faces. A 2025 Privacy Commissioner survey found that 49% of Māori respondents were concerned about facial recognition technology in retail stores, compared to 41% overall. For these communities, the failures of the law are not theoretical, they are playing out in real life as seen by the misidentification of the Māori mother in her local supermarket.
VIII Conclusion
New Zealand's privacy framework is moving, but the question is whether it is moving fast enough. AI systems are inferring private characteristics from data people never knowingly shared. LLMs are retaining personal information in ways the law has no clear mechanism to address. Facial recognition runs across retail stores, with marginalised communities often the most affected. A framework that is flexible and improving is not the same as one that is keeping pace. Whether New Zealand's light-touch approach can close that gap before the harm becomes entrenched is the question the next round of reform will need to answer.
==
The views expressed in the posts and comments of this blog do not necessarily reflect those of the Equal Justice Project. They should be understood as the personal opinions of the author(s). No information on this blog will be understood as official. The Equal Justice Project makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The Equal Justice Project will not be liable for any errors or omissions in this information nor for the availability of this information.