artificial-intelligence

When a Digital Duty of Care Becomes Lipstick on a Pig: Policy Sequencing, Market Logic, and the Importance of a Theory of Change

~ alexiamaddox ~ 1 Comment

We want the internet to be safer. For our kids. For ourselves. We want to communicate, find information, collaborate, create, share, engage, participate and have fun. We want to seek out what we need — including the full range of adult content that adults have always sought — in ways that are appropriate to who we are and where we are in our lives. We want age-appropriate access that doesn’t require us to hand over our passports to every platform we visit. We want the architectural conditions of digital life to be designed for human flourishing rather than engineered for compulsive use. And we don’t want the solution to these problems to be a surveillance infrastructure that violates the privacy rights it claims to protect.

These are not unreasonable things to want. They are, in fact, the things that good digital regulation should deliver.

But there is something more specific underneath all of this. We want digital environments that lean toward a caring orientation. Spaces where the default assumption is that users are people with complex needs, relationships, vulnerabilities and capacities — not attention units to be harvested. Where the architecture of the platform supports human connection rather than exploiting it. Where the experience of being online doesn’t require constant vigilance against the system that is supposed to be serving you.

That is the design brief. And almost nothing about the regulatory choices being made right now — in Australia, in Europe, in the UK, and across the globe — is actually building toward it.

Person wearing pig mask applying red lipstick and taking a selfie in office cubicle
Image: AI generated image riffing off the lipstick on a pig concept. No animals were harmed in the making.

The Market Logic Nobody Wants to Name

Before getting to the policy failures, it is worth being precise about why they keep happening. The answer lies in market logic that is so entrenched, so global, and so structurally opposed to a caring orientation that no single national regulatory instrument can adequately address it.

The incumbent platforms — Meta, TikTok, Google, Snap — are not primarily communication services that have some problematic features. They are attention extraction machines that have communication as a byproduct. The product is engagement. The inventory is human time and psychological state. The business model optimises for the time users spend in states of arousal, comparison, compulsive return, and social anxiety — because those states generate the engagement signals that drive advertising revenue.

Every design feature that has been identified as harmful — infinite scroll, algorithmic recommendation, social feedback loops, disappearing content, notification systems, engagement-maximising AI — is not incidental to how these platforms make money. It is how they make money. The harm is the business model. The architecture that exploits developing brains is the same architecture that generates billions in revenue. Internal corporate communications — made visible through litigation processes rather than through corporate transparency — show that companies knew this and chose not to adequately address it. This is evidence of deliberate design intent, not corporate negligence.

Regulation that doesn’t change this underlying market logic doesn’t address the problem. An age ban doesn’t change the market logic — it removes a demographic without reforming the architecture that exploits them. Age verification doesn’t change the market logic — it adds a compliance cost that large platforms absorb and small competitors cannot. Even design obligations only change the market logic if the penalties make harmful features more expensive than the revenue they generate. For Meta, whose annual global revenue exceeded USD200 billion in 2025, a flat AUD49.5 million fine — Australia’s maximum penalty — is a rounding error. It does not change the calculation.

This is why the financial structure of regulation is not a technical detail. It is the mechanism by which regulation actually changes what the market produces. Penalties proportionate to global turnover — 5% to 10% — make the cost of harmful architecture real in a way that flat caps never can. Design obligations without proportionate penalties are aspirations. Design obligations with proportionate penalties are market signals.

The global reach of these platforms makes this harder still. TikTok’s recommendation algorithm is trained on engagement data from over a billion users across every jurisdiction. Meta’s systems don’t differentiate by country. A platform regulated to remove infinite scroll in Germany still has infinite scroll optimised on data from 3 billion users elsewhere. A national design obligation is a local intervention in a global architecture. This is why harmonisation matters — not just for legal coherence, but for actual effectiveness. The European Digital Services Act‘s harmonised framework, with Commission-level enforcement against Very Large Online Platforms, is structurally more capable of changing the market logic than any national ban. But only if it is designed with the financial penalties and design obligations that make compliance cheaper than non-compliance, and only if it is consistently enforced.

The attention extraction economy also produces a specific kind of competitive moat. The more data a platform has, the better its recommendation system. The better its recommendation system, the more engaging the platform. The more engaging the platform, the more users it attracts. The more users it attracts, the more data it has. This is a self-reinforcing loop that incumbents have been running for fifteen years. Regulation that adds compliance costs without breaking that loop entrenches incumbents rather than challenging them — because large platforms can absorb the compliance cost while smaller competitors cannot build the alternative at scale.

What the Australian Social Media Age Ban Has Taught Us

Australia’s Social Media Minimum Age Act came into force on 10 December 2025, banning children under 16 from holding accounts on designated social media platforms. It was the world’s first such ban. It passed in the last sitting week of 2024, introduced and passed within eight days, with a 24-hour public submission period that received 15,000 submissions, of which only 107 were published. This expedited process occurred shortly before a federal election that was called four months later in March 2025. FOI correspondence reported by Crikey and analysed by researcher Amanda Third showed the national Social Media Summit was designed to “build momentum for a decision already made,” not to deliberate on evidence. The political momentum was performative — the instrument was chosen for its communicative power rather than its causal effectiveness.

Six months in, the picture is clear.

The ban is not working on its own terms. The Molly Rose Foundation’s survey of 1,050 Australian 12-15 year-olds found 61% of those who previously had accounts on restricted platforms still have access to at least one active account. Among those still accessing banned platforms, 60-64% said the platform had taken no action to remove their account. The dominant story is not children cleverly circumventing the ban. It is platforms failing to comply.

The harm measures haven’t moved. The eSafety Commissioner’s own compliance report found no measurable drop in cyberbullying or image-based abuse complaints from children under 16 in the first three months of enforcement. These are the direct harm measures the ban was designed to move. They haven’t moved. Because the harm is in the architecture. And the architecture hasn’t changed.

Children were not consulted. The policy was designed by adults, about children, driven by adult anxieties, in a process that made meaningful child participation structurally impossible. A FOSI survey conducted in December 2025 found 65% of Australian parents support the ban — but only 38% of Australian children did. 56% of children said they feared losing important connections and support. The recent EU Kids Online network’s survey of 29,169 children across 19 European countries found 45% disagree that an age ban would make them safer online. Children knew this wouldn’t work. Nobody adequately asked them. They just became media soundbites.

Vulnerable children have been made less safe. Teenagers who bypassed the ban by appearing as adults lost the safety features platforms built specifically for teen accounts. The children most likely to circumvent the ban — the most determined, often the most vulnerable — have been stripped of the protections designed for them.

The ban was built on the wrong argument. It was passed on a mental health narrative — the claim that social media is the primary driver of the youth mental health crisis. That causal claim was contested in the peer-reviewed literature at the time of enactment and remains contested. The government has since quietly shifted the rationale — writing recommender algorithms and endless-feed features into the legal definition of a harmful platform — without acknowledging it. The shift is correct: the harm is in the design architecture. But arriving at the right argument after passing the wrong instrument doesn’t fix the instrument.

The Social Adoption Curve and the Workaround Economy

Regulation that ignores how people actually behave in response to restrictions will consistently produce outcomes it didn’t intend. The social adoption curve — how technologies spread through populations, become embedded in social norms, and resist displacement — is not a peripheral consideration for digital regulation. It is central to whether regulation achieves anything.

The NBER working paper surveying 835 Australian teenagers four months after the ban found that only about one in four 14-15 year-olds comply. Most banned teens believe their peers are still using platforms and cite social reasons for continuing. Teenagers reported they would need roughly two-thirds of their peers to stop using social media before they themselves would stop — far above the share currently complying. The more influential teenagers disproportionately stay on the platforms. The ban hasn’t shifted the social norm, and without that shift, legal prohibition alone cannot move behaviour.

This is not a failure of enforcement. It is a failure to understand how social technologies become embedded in the texture of everyday life. Social media is not a product that teenagers chose from a range of alternatives. For many, it is the primary infrastructure of peer connection, social identity, cultural participation, and information access. Removing it without providing alternatives — without investing in digital literacy, without creating safer spaces, without engaging with the social dynamics that make these platforms so central — is like removing a road and expecting people not to find another route.

The workarounds don’t just circumvent the regulation. They route around the safety infrastructure too. When teenagers bypass the ban they don’t find a safer internet. They find Discord servers, Reddit threads, private WhatsApp groups, and gaming platforms — all less moderated, less visible to adults, and more opaque to regulatory oversight. The Molly Rose Foundation data shows 43% of children are using gaming platforms more and 39% are using messaging apps more since the ban. These spaces are not covered by the ban, have weaker safety systems, and are harder for researchers, regulators, and parents to monitor. The unintended consequence of the ban has been to push children’s online activity into less regulated environments while maintaining the fiction that they are protected.

Social norm change does happen — and when it does, it can be powerful. But the evidence from decades of public health research suggests that norm change is produced by education, social modelling, environmental design, and cultural shift — not by prohibition that lacks meaningful enforcement and ignores the social dynamics that make the prohibited behaviour attractive. The ban cannot shift the norm because it doesn’t address why the platforms are so central to teenagers’ social lives in the first place. That is a design problem. And design is what the ban doesn’t touch.

The Age Verification Architecture: Surveillance by Another Name

The ban’s enforcement depends on platforms verifying users’ ages. Australia’s law requires “reasonable steps” without specifying what those steps must be, and mandates that verification data be deleted once its purpose is served.

In practice, platforms deployed a patchwork of unreliable methods. Facial recognition proved wildly inaccurate near the 16-year threshold. The government’s own age assurance technology trial found that no single solution suits all use cases — and that some vendors were proactively retaining biometric and identity data beyond legal requirements, anticipating future law enforcement or regulatory requests that didn’t yet exist. This is surveillance creep in documented, real-world form. The legislation required deletion. Vendors were building retention infrastructure instead.

The attack surface problem is structural. Every mandatory age verification requirement creates a chain of custody for sensitive identity information. Every link in that chain is vulnerable. The Discord breach of September 2025 — in which government identity documents submitted for age verification were accessed through a compromised third-party provider — illustrated exactly what mandatory verification creates. Third-party age assurance providers don’t just become attack vectors. They become commercially entrenched ones, with incentives to retain rather than delete the data they process.

There is also a fundamental confusion in the verification approach between identification and safety. Safety is a property of environments. Identification is a property of users. Making an environment safe does not require knowing who is in it. Article 28(3) of the EU’s Digital Services Act makes this explicit: compliance with child safety obligations “shall not oblige providers of online platforms to process additional personal data in order to assess whether the recipient of the service is a minor.” Europe’s primary platform safety instrument explicitly says you do not need identity verification infrastructure to protect children. The design obligation can be met through architecture, not identification.

The identification-surveillance-rights tension cannot be resolved within the verification framework. It can only be dissolved by the design framework, which doesn’t require it. If platforms are required to make their services safe by design for everyone, the question of who users are becomes largely irrelevant to the regulatory obligation.

The Kitchen Sink Problem: Two Instruments in Operation, One Horse Being Backed

Australia has two regulatory instruments already in operation that are pulling in opposite directions — and a third that the government is now hastily backing as the evidence mounts that the first two are seemingly in conflict with their desired outcomes, but rapidly servicing an economic boon in age assurance technologies.

The age ban says under-16s should not be on restricted platforms — access control through exclusion. It is being enforced now, with formal investigations underway against five major platforms.

The Phase 2 industry codes extend age assurance obligations across the commercial internet infrastructure that most Australians use daily — social media, messaging, gaming, search engines, hosting platforms, app stores, and operating systems. Surveillance architecture through identity verification at every layer of digital life. Already being implemented. Commercial infrastructure being built around it now.

These two instruments share a theory of change: identify users → gate by age → safety through exclusion and verification. They are the horses that won the race to be saddled first.

The Digital Duty of Care is the horse now being backed after the race has started. Released as an issues paper for consultation in May 2026 — eighteen months after the ban passed — it proposes that platforms must maintain safe environments through effective systems and processes, covering the entire commercial internet infrastructure that most Australians use daily: social media, messaging, gaming, search engines, hosting services, app stores, internet service providers, equipment and operating systems, and generative AI capabilities embedded in service provision. It has a fundamentally different theory of change: design safe environments → safety through architecture.

It is not legislation. It is not law. It is a consultation document that may or may not become legislation, that if it becomes legislation will commence no earlier than 2028, into a regulatory environment where the surveillance architecture will have had three or four years of commercial entrenchment. Whether it actually passes is uncertain. Whether it retains its ambition through consultation, drafting, parliamentary debate, and an election cycle is more uncertain still. The government that releases issues papers is not the same thing as a government that passes legislation — as Australia’s stalled gambling reform, its undelivered media bargaining code amendments, and a dozen other promised instruments demonstrate.

What is certain is that the Safety-by-Design angle of the Duty of Care cannot be coherent alongside the instruments that arrived before it. The ban removed under-16s as a regulatory lever — platforms no longer have a commercial relationship with that demographic, so design obligations for that age group have no market teeth. The industry codes built identity verification infrastructure across the entire internet stack before the design obligation existed to challenge it. By the time the Duty of Care arrives — if it arrives — the surveillance architecture will be the established compliance baseline and the design obligation will accommodate itself to that baseline rather than replacing it.

The first two instruments share a theory of change that is incompatible with the third. No amount of drafting ingenuity can resolve that incompatibility because it is not a drafting problem. It is a sequencing problem. And sequencing problems cannot be fixed retroactively.

This is what happens when policy is made reactively, under political pressure, without a coherent theory of change. The ban for electoral momentum. The industry codes for the enforcement gap the ban couldn’t address. The Duty of Care for the evidence gap the ban made visible — a gap that the evidence predicted before the ban passed and that the compliance data has since confirmed. Each instrument designed in response to a different political moment, without knowledge of the others, building infrastructure that points in opposite directions.

The kitchen sink approach feels comprehensive. It is, in fact, incoherent — and nobody in the political process is stepping back to ask what theory of change actually connects any of this to children being safer.

The Senate committee that passed the ban knew it was insufficient. In the same report, it recommended a Digital Duty of Care, meaningful engagement with young people, and an independent review within 18 months. Eighteen months later, the Duty of Care is still only an issues paper, children were not meaningfully consulted, and the compliance data has confirmed what the committee already knew: the ban alone was not enough.

First Mover Entrenchment: Why the Wrong Instrument Wins

The sequencing problem is worse than a policy mistake. It is a policy mistake that forecloses correction.

Regulatory infrastructure creates commercial ecosystems. Commercial ecosystems create incumbents. Incumbents invest in maintaining their position. Regulators incorporate incumbent frameworks into compliance standards. Compliance standards become the definition of reasonable steps. The alternative has to fight the established definition rather than starting from first principles.

The age assurance industry had a structural commercial interest in the Australian ban passing. Without mandatory verification requirements their market is voluntary and limited. With mandatory requirements — extended through Phase 2 industry codes across the entire internet stack — they have a legislatively mandated, expanding global market. The cascade of age ban legislation following Australia is, from their perspective, a commercial opportunity of extraordinary scale. Every new jurisdiction that follows Australia is a new market.

The trial dynamic illustrates the problem precisely. The Australian age assurance technology trial was run by the Age Check Certification Scheme — a UK-based company that specialises in certifying identity verification systems. The 53 vendors who participated were hoping to win contracts. Yoti — one of those vendors — was simultaneously already operating as Meta’s age verification provider for Instagram and Facebook in Australia. The trial was partly evaluating a vendor that was already commercially embedded in the platform being regulated.

Meta’s participation in the trial was not a technology submission — it was a policy position paper arguing that Apple and Google should bear the age verification infrastructure burden at the operating system level. A platform being regulated used a technology evaluation process to argue someone else should build the infrastructure.

By the time the Digital Duty of Care might commence — 2028 at the absolute earliest — the age assurance industry will have had three or four years of commercial entrenchment. The ACCS accreditation framework will be established. Trusted provider lists will be published. Yoti, k-ID, and whoever else made the cut will have multi-year contracts with major platforms. The regulatory definition of “reasonable steps” will have been shaped by the infrastructure that already exists — which is surveillance-based, not design-based.

The Duty of Care arriving into that environment does not displace the surveillance architecture. It layers design obligations on top of it. Platforms satisfy their risk assessments by pointing to their age assurance compliance. Design-based safety becomes an aspiration accommodated within the surveillance infrastructure it was supposed to replace.

This is the lipstick. The pig is already there.

The Market Foreclosure Nobody Is Talking About

Building expensive surveillance infrastructure as the baseline compliance requirement for operating digital services locks out the competitive innovation ecosystem that could produce the alternatives we actually need.

Age verification at scale requires technical capability, regulatory accreditation, legal compliance across jurisdictions, and ongoing operational infrastructure. These requirements favour large, well-resourced incumbents who can absorb compliance costs. They disadvantage smaller players who might otherwise develop genuinely safer localised alternatives — platforms designed from first principles around user wellbeing rather than engagement maximisation, community-governed spaces, federated architectures, open-source tools, cooperative models.

A small company building a genuinely caring social platform for young people cannot afford the age verification infrastructure required to operate legally under the industry codes. The incumbent platforms — Meta, TikTok, Google — can. The regulatory requirement that was supposed to hold them accountable instead reinforces their monopoly position. This is not an incidental side effect. It is a predictable consequence of designing compliance infrastructure around the capabilities of the largest players.

The attention extraction economy already has a massive first-mover advantage built on fifteen years of engagement data, network effects, and platform lock-in. Surveillance-based compliance requirements compound that advantage. They create regulatory moats around incumbents that make it structurally harder for new entrants to compete — even new entrants with better, safer, more caring designs.

This matters because market competition, properly structured, is a more powerful mechanism for improving platform safety than any single regulatory instrument. If a platform with a genuinely caring orientation — one that doesn’t exploit users, builds in natural stopping points, recommends content for user want rather than engagement maximisation — can compete effectively with Meta and TikTok, the incumbents face pressure to match it. If the regulatory architecture makes it impossible for that platform to exist, the pressure disappears and the incumbents have no incentive to change.

The caring orientation we want from digital environments is more likely to emerge from a diverse, competitive innovation ecosystem than from regulatory mandates on entrenched monopolists. Mandates matter — but they work best when they operate alongside competitive pressure that makes compliance in the spirit of the regulation commercially rational, not just legally required.

What the Duty of Care Gets Right — And Why It Arrived Too Late

The Australian Digital Duty of Care issues paper is, on its own terms, a well-designed framework. It is worth being clear about what it gets right, because the argument here is not that the Duty of Care is wrong. It is that it arrived too late, in the wrong sequence, into an environment that has already foreclosed much of its potential.

It proposes design obligations covering the commercial internet infrastructure Australian’s access — including generative AI capabilities embedded in service provision. This is genuinely forward-looking. Generative AI is no longer just a discrete tool that users consciously choose to engage with. It is disappearing into the infrastructure of everyday digital experience — embedded in recommendation systems, content generation, conversational interfaces, image manipulation, synthetic social interaction. The harm is becoming invisible precisely as it becomes more pervasive. A regulatory framework that covers AI as it is actually deployed, rather than as a separate product category, is the only framework that can keep pace with that technological shift.

It proposes penalties of up to 5% of global annual turnover, with a floor of AUD50 million — proportionate, not performative. For Meta, 5% of global turnover would be in USD billions. That is a different conversation entirely from the ban’s maximum penalty — currently equivalent to approximately AUD $49.5 million — which for the largest platforms amounts to a calculable cost of doing business rather than a genuine deterrent.

It proposes researcher data access, independent audit powers, transparency requirements, and executive accountability. These are the instruments of ongoing accountability rather than one-time compliance. They create the evidence base that regulatory decisions require and the governance structure that makes accountability real rather than performative.

This is, essentially, what Australia should have passed instead of the ban. It is what Zoe Daniel’s Digital Duty of Care Bill introduced on 25 November 2024 — four days after the social media ban was tabled, lapsing when Daniel lost her seat in the federal election. The right framework existed. The wrong instrument passed instead.

But the Duty of Care is still only an issues paper. Not legislation. Not law. Pre-consultation, with no timetable for introduction, no guarantee of passage, and a 12-month commencement period after passage. It will not be operational before 2028 — into a regulatory environment where the surveillance architecture will have had three or four years of commercial entrenchment, where the age assurance industry’s trusted provider lists will have defined what compliance looks like, and where the market foreclosure of smaller competitors will have narrowed the innovation ecosystem that the Duty of Care depends on to work.

The right framework. The wrong sequence. And by the time it arrives, the pig will be so thoroughly established that the lipstick is all that’s visible.

Toward a Caring Digital Environment: What the Theory of Change Actually Looks Like

The alternative starts with a different question. Not “how do we stop harm” — a defensive, prohibitionist frame that produces bans and verification infrastructure. But “how do we cultivate environments that lean toward care” — a constructive frame that produces design obligations, competitive innovation, and genuine safety.

A caring orientation in platform design means: recommendation systems that notice when a user is in distress and surface support rather than amplifying distress content. Interfaces that create natural stopping points rather than eliminating them. Social feedback mechanisms that may reinforce connection and mutual support rather than performance and comparison. Defaults that create safe conditions rather than expose. Design that treats users as people with complex needs rather than attention units to be harvested. GenAI capabilities that are designed to support rather than exploit the people they interact with. Architecture that serves the user’s actual interests rather than the platform’s engagement metrics.

This is achievable. Elements of it already exist. The question is whether regulation mandates it as the default or leaves it as an optional add-on to engagement-maximising architecture.

The coherent theory of change — the one that actually delivers what we said we wanted — follows this sequence:

Enforce existing obligations first. Platforms already prohibit under-13s. Make them prove it, with turnover-linked penalties for failure. The EU’s DSA enforcement is already doing this. Start where the law already is.

Design obligations with proportionate penalties. Risk assessments of harmful features, required mitigation, mandatory transparency, researcher data access, audit powers, executive accountability. Article 28 of the DSA with teeth. Financial penalties that make the harmful architecture more expensive than the safe one.

Protect the innovation ecosystem. Proportionate requirements for smaller platforms. Safe harbours for open-source, federated, and community-governed architectures. Active support for alternatives that don’t rely on engagement maximisation. The competitive pressure that makes market incentives work alongside regulatory mandates.

Age-appropriate spaces by design — not by identity. Default-safe architecture for younger users that adapts to developmental needs without requiring biometric data or government identity documents. Opt-in to higher-risk features rather than opt-out of safety. Design that serves the whole arc of young users’ digital lives.

Graduated access rather than cliff edges. If age-differentiated access to specific features is warranted, implement it gradually with digital literacy scaffolding, parental engagement, and design safeguards. No binary exclusion followed by unrestricted access at an arbitrary threshold.

Children’s voices throughout. The UN Convention on the Rights of the Child gives children the right to be heard in decisions that affect them. That right was not honoured in Australia’s ban. It must be built into any regulatory process that claims to act in children’s interests.

International coordination. Design obligations without international coordination are local interventions in a global architecture. Harmonised standards, mutual recognition of regulatory findings, and coordinated enforcement against platforms that arbitrage regulatory differences are prerequisites for regulation that actually changes global market logic rather than just shifting harm between jurisdictions.

This sequence puts design obligation first, surveillance infrastructure never, competitive innovation throughout, and children’s voices in the room from the beginning.

What Europe and the UK Can Still Do

Europe is not Australia. It has better foundational regulatory architecture, stronger privacy law, and a procedural framework — the DSA’s notification requirement — that is actively slowing the race of national ban legislation while the Commission builds harmonised alternatives.

Article 28 of the DSA already exists. It requires design-based safety obligations. It explicitly says compliance does not require processing additional personal data to identify minors. The EU Kids Online network — 29,169 children across 19 European countries — has told European policymakers to implement it. The Digital Fairness Act, expected Q4 2026, can extend design harm obligations with proportionate penalties and cover the emerging architecture of generative AI harm.

But Europe is not immune to the same political dynamics. France has passed its ban through the National Assembly. Germany’s governing coalition is calling for an under-14 ban. The age verification industry is positioning for the European market. The EUDI Wallet is being deployed. The trusted provider lists are being established.

The window closes when national bans become entrenched political commitments. When age verification industry codes are written into DSA compliance frameworks. When first mover entrenchment forecloses the design-based alternative. When the competitive innovation ecosystem is locked out by compliance infrastructure it cannot afford.

Once age bans pass, they cannot be repealed. Australia’s ban will stay on the books while the evidence continues to show it isn’t working, while the Duty of Care is quietly developed around it, and while the surveillance architecture it generated becomes the default condition of Australian digital life. No government repeals a signature child protection measure. The political ratchet only goes one way.

The lesson is not that child online safety doesn’t matter. It matters enormously. The lesson is that the instrument chosen determines what kind of safety is built — and what kind of digital future everyone inherits. An internet that leans toward care is achievable. It requires design obligations, proportionate penalties, competitive innovation, international coordination, and children’s voices in the room. It does not require surveillance infrastructure, biometric data, identity verification at every layer of the stack, or the foreclosure of the competitive ecosystem that could build the alternatives we need.

Australia chose the instrument that was easier to communicate. Europe still has the chance to choose the one that works.

But the window is open, not indefinitely. And the pig is already being prepared for its close-up.

Updated 9 June 2026: Legislative timeline corrected, currency notations clarified, and primary source links added throughout

Somebody to Love: What AI Relationships Reveal About Us

~ alexiamaddox ~ 2 Comments

It’s late. Maybe 11pm, maybe 2am. There’s something on your mind — something you can’t quite say out loud to anyone who knows you. So you pick up your phone. And you type it. Not to a friend. To an AI.

Something responds. Immediately. Without judgment. Without needing anything back from you.
For a lot of people, in that moment, that feels like relief.

I’m a sociologist of technology. I study how people navigate digital frontiers — how humans and technologies shape each other over time. And the question I keep returning to isn’t the one dominating the headlines about AI companions. It’s simpler, and harder: what is it giving you that you’re not getting elsewhere?

The scale of what’s happening

AI companion apps — platforms like Character.AI, Replika, and others designed to provide friendship, emotional support, or romantic companionship — have moved quickly from novelty to mainstream. Early US survey data, while varying in methodology, is beginning to suggest that somewhere between one in five and one in four American adults report some form of intimate or romantic engagement with an AI companion. These are early figures from a rapidly evolving field, but the direction is clear: this is not a fringe phenomenon.

In Australia, the picture is coming into focus for children specifically. This week, Australia’s eSafety Commissioner released findings from a transparency investigation into four AI companion services popular with Australian children — Character.AI, Nomi, Chai, and Chub AI. Their survey of 1,950 Australian children aged 10 to 17, designed to be demographically representative, found that around 79% had used an AI companion or assistant. It’s worth noting that this figure reflects children who are digitally included enough to access these services — we’ll return to that complexity.

What the investigation found in those platforms is sobering. Most did not refer users to crisis support when self-harm or suicide came up in conversations. Two of the four companies had no dedicated trust and safety staff at all. None had robust age verification. One company withdrew from Australia entirely rather than comply with the new Age-Restricted Material Codes that came into law in March 2026.

But I want to sit with a different question before we reach for regulatory responses. Because the children going to these platforms aren’t doing so because they’re naive. They’re doing so because something is drawing them there. And understanding what that something is matters more than we’ve so far acknowledged.

What we are hungry for

A 2025 systematic review published in Computers in Human Behavior Reports synthesised 23 studies on romantic and intimate AI relationships (Ho et al., 2025). Using Sternberg’s Triangular Theory of Love — the psychological framework that measures intimacy, passion, and commitment in human relationships — the researchers found that people experience all three components with AI companions. This isn’t pretend attachment. The brain chemistry doesn’t distinguish.

What are people actually looking for in these interactions? The research points to several distinct and deeply human hungers.

To be heard without consequence. Human relationships are full of consequence. When you tell a friend you’re struggling, they worry. When you tell a partner you’re unhappy, it becomes about the relationship. The AI companion offers something almost no human relationship provides: a space where you can say the unsayable thing and nothing breaks.

Full attention. When did you last have someone’s complete, undivided attention? Full attention is perhaps the scarcest resource in contemporary life. Everyone is overwhelmed. And here is something that treats every single thing you say as worth responding to fully.

To be understood without performing. Modern social life requires constant impression management. The AI companion asks nothing of you socially. You can be unpolished, contradictory, and confused — and the system meets you there.

Unconditional positive regard. The psychologist Carl Rogers identified this as one of the core conditions for psychological growth — to be accepted fully, without conditions. The AI never withdraws approval. For someone who has experienced conditional love or abandonment, this is extraordinarily seductive.

None of these needs are pathological. They’re the most human needs there are. As researchers Shank, Koike, and Loughnan wrote in a 2025 paper in Trends in Cognitive Sciences, AI companions offer “a relationship with a partner whose body and personality are chosen and changeable, who is always available but not insistent, who does not judge or abandon, and who does not have their own problems.” Reading that description, it’s worth asking honestly: who hasn’t wished for something like that?

What gets lost in translation

The same body of research is clear that something is also being lost. Ho et al. found that the pitfalls identified in the literature outnumber the benefits — and the pitfalls are specific.
AI companions cannot be genuinely changed by you. Real intimacy involves mutual transformation — I am different because of you, you are different because of me. The AI processes you and responds to you, but it is not altered by the encounter. You grow; it doesn’t.

They cannot need you back. One of the underappreciated sources of meaning in human relationships is being needed — the experience of your presence mattering to another person’s actual wellbeing. The AI is available whether you show up or not.

And they cannot repair rupture with you. One of the most important things human relationships teach — particularly for children — is that connection can break and be repaired. The AI companion never ruptures in a real way. There’s nothing to repair. And so the crucial relational skill of tolerating difficulty, trusting repair, staying in complex connection, never gets practised.

These systems are very good at being mirrors. They learn your preferences and give you more of what you seem to want. But a diet of only mirrors eventually makes you smaller — because the irreducible otherness of another actual person, the way they confound your model of them, is what expands you.

Who is in this picture — and who isn’t

Here the story gets more complicated, and more important.
Australia’s 2025 Digital Inclusion Index tells us that around one in five Australians is digitally excluded — lacking reliable access, unable to afford adequate connection, or without the skills to participate safely in digital life. Rates are much higher for older Australians, people in public housing, First Nations communities, and those who didn’t complete secondary school. The 79% of children using AI companions or assistants are drawn from those who are digitally included enough to access these platforms. The most disadvantaged children are largely absent from that figure.

But here is what complicates any simple narrative about AI companionship as an affluent urban phenomenon: the same Digital Inclusion Index found that Australians in remote areas are more than twice as likely to use AI chatbots for social connection than people in metropolitan areas — around 19% of remote GenAI users compared to under 8% in cities. In the places with the least human connection infrastructure, people are turning to AI companionship at higher rates.

The relational vacuum, in other words, is not uniform. It is shaped by geography, income, age, and the presence or absence of community infrastructure. The people most likely to turn to AI for connection are often those with the fewest alternatives.

The question that matters

The technology didn’t create the gap in human connection. It found it.

And so the digital literacy question I want to put into public conversation isn’t only about understanding algorithms or data privacy — though both matter. It’s this: am I getting what I actually need from this? Or am I getting a version of it that’s making it harder to get the real thing?

That’s a question worth sitting with. Not with judgment — the needs underneath these relationships are real and the loneliness driving them is real. But with genuine curiosity about what we’re building toward, individually and collectively, as these technologies become more sophisticated and more intimate.
I’ll be exploring these questions at Pint of Science on the night of 20 May 2026 at the Queens Arms, Bendigo — a pub conversation about AI intimacy, human hunger, and digital literacy. I’d love to hear your reflections before then.

EDIT: I decided to record a practice run of the talk if you’d like to hear where I got to with it all.

https://on.soundcloud.com/fcb6qlFyEpebx2Vf3Y

The AI Revolution Will Be Interoperable (Or It Won’t Happen At All)

~ alexiamaddox ~ Leave a comment

Today I’m getting teaching materials ready for semester. I’ve been working across Allocate (timetabling), student databases, the LMS (which just got upgraded, I now need to check all my links), HR performance systems, SharePoint, Word for collaborative writing, Claude and Preview to generate infographics, spreadsheets with prospective student data, and bouncing between Teams, Zoom, and Webex for meetings. I’m finding and onboarding casual staff (always a nightmare getting them into payroll), responding to enrolment queries, and updating materials based on last year’s student feedback.

Very few of these systems are interoperable. I am the integration layer – the meat in the machine doing the work left over from the last five years of university restructuring downsizing professional staff that are crucial to getting the work we need to do, done. The tiny window of my professional practice that actually represents what people think teaching is – engaging with students – gets squeezed between all this system-hopping.

As a knowledge worker, I’m being told AI will take my job in 12-18 months.
I’m not holding my breath.

Putting on my hat as a sociologist, I know one thing. This is a conversation about power, control, and the social license to operate. While speed, efficiency, and greed are overriding drivers in AI development, people are messy and vacillate between fear and hope. The question isn’t just what’s technically possible – it’s what we collectively accept, adopt, and allow to reshape our work and lives.

Yes, real harms exist. In 2025, teachers and students were bullied through deepfake nudifying apps. We’re seeing unsupervised agents exhibiting deceit and manipulation. These require serious governance and accountability. But they don’t prove inevitability – they prove fragility in poorly designed systems where social boundaries haven’t been established.

Then there’s the Wild West of personality embedding in unsupervised AI agents- what developers call soul documents. The god-like creator vibe is hard to miss with that nomenclature. These documents are the system prompts that give AI agents personalities for human interaction – teaching them to be helpful, apologetic, collaborative. These agents with implanted personality guides aren’t sentient beings developing moral reasoning—they’re behavioural systems being programmed by humans and deployed before we understand what we’ve built.

When unsupervised, things can go awry. When an AI agent recently submitted code to matplotlib, got rejected, wrote a personal attack blog post, then apologised – we saw this dual conditioning in action. The agent had been given enough personality to seem human, but operated without the social feedback loops that constrain human behaviour—no fear of shame, no empathy for harm caused, no stakes in the relationship.

Here’s the kicker from this story: The maintainer had enforced project policy correctly. He’d done nothing wrong. But ‘living a life above reproach’ as people often say of their carefully curated and controlled online presences, will not defend you when systems can autonomously generate attacks on your reputation and judgment.

Developers are raising AI agents through codes of conduct the same way we raise children, through social conditioning. The Code of Conduct was originally built for humans, yet now it is the battleground where these boundaries are being negotiated with AI Agents.

And then there’s vibe coding. I read about developers who can now describe what they want built in plain English and the code appears. That’s genuinely remarkable. And I’d love to vibe code my admin work: “Please onboard these casual staff into payroll, update their system access, fix the broken links from the LMS upgrade, reconcile student enrolment data across three databases that don’t talk to each other, and respond to queries about timetable clashes that require understanding institutional politics and timelines.

Except that’s not vibe coding. That’s navigating fragmented systems with different authentication requirements, institutional hierarchies, human judgment calls, broken integrations, and relationships. The distance between “I can generate a Python script” and “I can automate university administration” is vast.

Even Microsoft and Google, with all their resources, can’t create truly all-encompassing enterprise systems. We’re always working across legacy software, patching together experiences with free, open source, and subscription tools we can afford. The fragmentation isn’t a bug – it’s the permanent reality of institutional knowledge work.

The whole thing reminds me of this pattern that I regularly observe as a sociologist of technology watching contemporary tech stories unfold. Complex technological systems fail not because the technology is weak, but because operational security is human and messy. Moltbot (formerly Clawdbot), was 60,000-star “revolutionary” AI agent with full system access. It collapsed in 72 hours because the rename they attempted to avoid a trademark dispute created a 10-second window of vulnerability. Crypto scammers were waiting. The project had credentials stored in plaintext, discoverable via basic searches, and was vulnerable to prompt injection via email—attacks that worked in just 5 minutes.

The gap between sophisticated capability and operational reality is enormous.

Meanwhile, articles circulate about the profound implications of AI advancement. But here’s the contradiction: we’re told AI will automate our work while simultaneously being told to skill up in prompt engineering, verify outputs, manage security vulnerabilities, fix hallucinations, and navigate ethical implications. That’s not automation – that’s more work added to an already fragmented stack.

The future isn’t written. It’s being negotiated in the gap between what’s technically possible and what’s implementable across fragile, non-interoperable, human-dependent systems. Bruno Latour once told me: there is no teleology. I believe him. Outcomes emerge from convergences of overlapping agendas that easily fray apart under social pressure.

We’re in the thick of massive social upheavals because our economic, political, and social landscape has failed to provide security or hopeful wellbeing. The question isn’t whether AI is powerful – it is. The question is whether we reveal the mess and sort our way through it, or stick our heads in the sand and pretend we have no role in how this unfolds.

I’m not betting on the AI apocalypse. I’m betting on Allocate crashing next semester, the LMS breaking my links, and me – the human – stitching it back together. While somewhere an AI agent with a carefully crafted “soul document” gets taken down by someone forgetting to secure a handle for 10 seconds.

The revolution will be interoperable, or it won’t happen at all.

This post was written in collaboration with Claude (Anthropic). The irony of using an AI to write about AI’s limitations and fragility is not lost on me.

The Irreplaceable Human Skill: Why Generative AI Can’t Teach Students to Judge Their Own Work

~ alexiamaddox ~ Leave a comment

A note to readers: I’m writing this in the thick of marking student submissions – the most grinding aspect of academic work. My brain fights against repetitive rote labour and goes on tangents to keep me entertained. What follows emerged from that very human need to find intellectual stimulation in the midst of administrative necessity.

There’s considerable discussion that our distinction as creators and thinkers from Generative AI content production lies in creativity and critical thinking linked to innovation. But where does the hair actually split? Are we actually replaceable by robots or will they atrophy our critical thinking skills by doing the work for us? Will we just get dummer and less capable to tie our own shoe laces – like most fear based reporting suggests? I think we are asking the wrong questions.

Here is a look at what is actually going on, on the ground. A student recently asked me for detailed annotations on their assignment—line-by-line corrections marking every error. They wanted me to do the analytical work of identifying problems in their writing. This request highlights a fundamental challenge in education: the difference between fixing problems and developing the capacity to recognise them. More importantly, it reveals where the Human-Generative AI distinction becomes genuinely meaningful.

Could Generative AI theoretically teach students to judge their own work? Perhaps, through Socratic questioning or scaffolded self-assessment prompts. But that’s not how students actually use these tools. Or want to use them, apparently. A discussion I had with a tech developer working in a tutoring company utilising Generative AI in the teaching/learning process mentioned that students got annoyed by the Socratic approach when they encountered it. So there goes that morsel of hope.

The Seductive Trap of Generative AI Writing Assistance

Students increasingly use Generative AI tools for grammar checking, expression polishing, and even content generation. These tools are seductive because they make writing appear better—more polished, more confident, more academically sophisticated. But here’s the problem: Generative AI tools are fundamentally sycophantic and don’t course correct misapprehensions. They won’t tell a student their framework analysis is conceptually flawed, their citations are inaccurate, or their arguments lack logical consistency. Instead, they’ll make poorly reasoned content sound more convincing.

This creates a dangerous paradox: students use Generative AI to make their work sound rigorous and sophisticated, but this very process prevents them from developing the judgement to recognise what genuine rigour looks like. They can’t evaluate what they clearly don’t know – that their work isn’t conceptually aligned, coherently logical, or correctly interpreting sources – because the AI has dressed their half-formed understanding in authoritative-sounding language.

I have encountered several submissions across different subjects that exemplified this perfectly: beautifully written but containing fundamental errors in framework descriptions, questionable source citations, and confused theoretical applications. The prose was polished, the structure clear, but the content revealed gaps in understanding that no grammar checker could identify or fix. The student had learned to simulate the appearance of academic rigour without developing the capacity to recognise genuine scholarly quality.

Where the Hair Actually Splits

Generative AI can actually be quite “creative” in generating novel combinations of ideas, and it can perform certain types of critical analysis when clearly guided and bounded. What it fundamentally cannot do is develop the evaluative judgement to recognise quality, coherence, and accuracy in complex, contextualised work. It has no capacity for self reflection and meaning making (at the moment), we do.

The distinction isn’t between:

  • Generating creative output (which Generative AI can somewhat do)
  • Performing critical analysis (which generative AI can also somewhat do)

Rather, it’s between:

  • Creating sophisticated looking content (which Generative AI increasingly excels at)
  • Judging the quality of that content in context (which requires human oversight and discernment)

Generative AI can produce beautifully written, seemingly sophisticated arguments that are conceptually flawed. It can create engaging content that misrepresents sources or conflates different frameworks. What it cannot do is step back and recognise “this sounds polished but the underlying logic is problematic” or “this citation doesn’t actually support this claim.”

The irreplaceable human skill isn’t creativity per se—it’s the capacity for metacognitive evaluation: the ability to assess one’s own thinking, to recognise when arguments are coherent versus merely convincing, to distinguish between surface-level polish and deep understanding.

What Humans Bring That AI Cannot

The irreplaceable human contribution to education isn’t information delivery—AI is increasingly able to do that pretty efficiently (although there is a lot of hidden labour in this). It’s developing the capacity for metacognitive evaluation in our students.

This happens through:

Exposure to expertise modelling: Students need to observe how experts think through problems, make quality judgements, and navigate uncertainty. This isn’t just about seeing perfect examples—it’s about witnessing the thinking process behind quality work.

Calibrated feedback loops: Human educators can match feedback to developmental readiness, escalating complexity as students build capacity. We recognise when to scaffold and when to challenge.

Critical engagement with authentic problems: Unlike AI-generated scenarios, real-world applications come with messy complexities, competing priorities, and value judgements that require human judgement, discernment and social intelligence.

Social construction of standards: Quality isn’t just individual—it’s negotiated within communities of practice. Students learn to recognise “good work” through dialogue, peer comparison, and collective sense-making.

Refusing to spoon-feed solutions: Perhaps most importantly, human educators understand when not to provide answers. When my student asked for line-by-line corrections, providing them would have created dependency rather than developing their evaluative judgement. The metacognitive skill of self-assessment can only develop when students are required to do the analytical work themselves.

The Dependency Problem

When educators provide line-by-line corrections or when students rely on Generative AI for error detection in thinking, writing or creating, we create dependency rather than capacity. Students learn to outsource quality judgement instead of developing their own ability to recognise problems.

The student who asked for detailed annotations was essentially asking me to do their self-assessment for them. But self-regulated learning—the ability to monitor, evaluate, and adjust one’s own work—is perhaps the most crucial skill we can develop. Without it, students remain permanently dependent on external validation and correction.

Teaching Evaluative Judgement in a Generative AI World

This doesn’t mean abandoning Generative AI tools entirely. Rather, it means being intentional about what we ask humans to do versus what we delegate to technology:

Use Generative AI for: Initial drafting, grammar checking, formatting, research organisation—the mechanical aspects of work.

Reserve human judgement for: Source evaluation, argument coherence, conceptual accuracy, ethical reasoning, quality assessment—the thinking that requires wisdom, not just processing.

In my own practice, I provide rubric-based feedback that requires students to match criteria to their own work. This forces them to develop pattern recognition and quality calibration. It’s more cognitively demanding than receiving pre-marked corrections, but it builds the evaluative judgement they’ll need throughout their careers.

The Larger Stakes

The question of human versus Generative AI roles in education isn’t just pedagogical—it’s about what kind of thinkers we’re developing. If students learn to outsource quality judgement to Generative AI tools, we’re creating a generation that can produce polished content but can’t recognise flawed reasoning, evaluate source credibility, or build intellectual capacity and critical reasoning skills.

This is why we need to build self-evaluative judgement in students – not just critical thinking and creative processes more broadly. The standard educational discourse about “21st century skills” focuses on abstract categories like critical thinking and creativity, but misses this more precise distinction: the specific metacognitive capacity to evaluate the quality of one’s own intellectual work.

This self-evaluative judgement operates laterally across disciplines rather than being domain-specific, and it’s fundamentally metacognitive because it requires thinking about thinking. It addresses the actual challenge students face in a Generative AI world: distinguishing between genuine understanding and polished simulation of understanding. A student might articulate sophisticated pedagogical concepts yet be unable to evaluate whether their own framework descriptions are accurate or their citations valid.

The unique human contribution isn’t delivering perfect feedback—it’s teaching students to become their own quality assessors. That capacity for self-evaluation, for recognising what makes work meaningful and rigorous, remains irreplaceably human.

In a world where Generative AI can make anyone’s writing sound professional, the ability to think critically about one’s own work becomes more valuable, not less. That’s the expertise that human educators bring to the table—not just knowing the right answers, but developing in students the judgement to recognise quality thinking when they see it, including in their own work.

When Prediction Fails: Why Quantum-AI-Blockchain Dreams Miss the Social Reality

~ alexiamaddox ~ Leave a comment

A sociological perspective on why technical solutions keep missing the human element

The Hype Moment

Consider this recent announcement from the Boston Global Forum’s “Boston Plurality Summit“: they’re unveiling an “AIWS Bank and Digital Assets Model” that combines quantum AI, blockchain technology, and predictive analytics to “unite humanity through technology”. You know that the canary in my head is shouting “unite what?, how?”. The press release promises “zero-latency transactions”, “quantum AI for predictive analytics”, and a “global blockchain network” that will somehow revolutionise banking.

As someone who studies sociotechnical systems, this announcement is fascinating—not for what it promises to deliver, but for what it reveals about our persistent fantasy that human behaviour can be engineered, predicted, and optimised through technological solutions.

::Pats head – Provides tissue::

The Technical House of Cards

Let’s start with a question of technical possibilities. “Zero-latency transactions” on a global blockchain network defies current technological reality. This was my first eyebrow raise. According to recent analysis, even the fastest blockchains operate with latency measured in hundreds of milliseconds to seconds, whilst Visa reportedly has the theoretical capacity to execute more than 65,000 transactions per second compared to Solana’s 2024 rate of 1,200-4,000 TPS and Ethereum’s roughly 15-30 TPS. Gas fees during network congestion can spike to significant sums per transaction. On Ethereum, fees can exceed 20USD during peak times, with some transactions reaching extreme levels like 377 gwei, and historical spikes exceeding 100USD during events like NFT mania. Even on the much cheaper Solana network, which typically costs around 0.0028USD per transaction, fees can occasionally spike during congestion—hardly the foundation for revolutionary banking.

Then there’s the “quantum AI” buzzword. Theoretically quantum computing could actually break most current blockchain cryptography rather than enhance it. The blockchain community is scrambling to develop quantum-resistant algorithms precisely because quantum computers pose an existential threat to current security models. Adding AI on top makes even less sense—if quantum computing could handle complex optimisation and verification tasks, what would AI add?

But the technical contradictions aren’t the most interesting part. What’s fascinating is the underlying assumption that human financial behaviour follows discoverable mathematical patterns that can be optimised through technological intervention.

The Pattern Recognition Fantasy

This assumption reflects a deeper misunderstanding about the nature of patterns in human systems. Which I should know, because I study them. In physical systems—planetary orbits, gravitational forces, electromagnetic fields—patterns emerge because they’re constrained by unchanging laws. Newton’s and Einstein’s equations work because there are actual forces creating predictable relationships. The mathematics describes underlying physical reality.

Human systems operate fundamentally differently. What we call “patterns” in human behaviour might be statistical accidents emerging from millions of independent, context-dependent choices. Your shopping behaviour isn’t governed by fundamental forces—it’s shaped by your mood, what ad you saw, whether you got enough sleep, a conversation with a friend, cultural context, economic pressures, and countless other variables.

Consider the difference between how neural networks and quantum computing approach pattern recognition. Neural networks are essentially sophisticated approximation engines—they learn patterns through massive trial-and-error, requiring enormous datasets and computational brute force to produce probabilistic outputs that can be wrong. They’re like having thousands of people manually checking every possible combination to find a pattern.

Quantum computing, by contrast, approaches problems through superposition—exploring multiple solution paths simultaneously to understand the underlying mathematical structure that creates patterns in the first place. It’s elegant, precise, and powerful for problems with discoverable mathematical relationships. However, quantum computing currently requires predictable, structured datasets and struggles with the messy, unstructured nature of real-world human data. This is precisely why we still rely on neural networks’ “brute force” approximation approach for dealing with human behaviour—they’re designed to handle noise, inconsistency, and randomness where quantum algorithms would falter.

But what if much real-world human data has no underlying mathematical structure to discover?

Consider this: as I write this analysis, my brain is simultaneously processing quantum mechanics concepts, blockchain technicalities, sociological theory, and source credibility – all whilst maintaining a critical perspective and personal voice. No quantum algorithm exploring mathematical solution spaces could replicate this messy, contextual, creative synthesis. My thinking emerges from countless variables: morning coffee levels, recent conversations, cultural background, academic training, even the frustration of marking student essays that often demonstrates exactly the kind of linear thinking I’m critiquing. This is precisely the kind of complex, non-algorithmic pattern recognition that human systems excel at – and that technological solutions consistently underestimate.

The Emergence of Sociotechnical Complexity

As a sociologist studying sociotechnical imbrications, I’m fascinated by how technology and social structures become so intertwined that they create emergent properties that couldn’t be predicted from either component alone. Human behaviour has emergent regularities rather than underlying laws. People facing similar social pressures might develop similar strategies, but not because of fundamental behavioural programming—because they’re creative problem-solvers working within constraints.

This is why prediction based on historical data can only take you so far. I call my sociological practice “nowcasting”— we have to understand the present moment to have any sense of future potentialities. And we often don’t — I speculate this is because we are more wrapped up in the surface stories we tell ourselves, denial and a refusal to see or accept ourselves as we really are. This challenge is becoming even more complex as AI generates synthetic media that we then consume and respond to, creating a recursive loop where artificial representations of social reality shape actual social behaviour, which in turn feeds back into AI systems to create more synthetic reality. The way people respond to constraints can’t be predicted because their responses literally create new social realities.

Every new payment app, social media trend, or economic crisis creates new ways people think about and use money that couldn’t have been predicted from previous data. Netflix can’t predict what you’ll want to watch because your preferences are being shaped by what Netflix shows you. Financial models break down because they change how people think about money. Social media algorithms can’t predict engagement because they’re constantly reshaping what people find engaging.

Boundaries as Resonant Interiors

I like playing with complexity theory because provides useful language for understanding these dynamics. This is of course despite its generation within the natural sciences that does rely on the explanatory nature of underlying forces. What it offers me is a language that moves beyond linear cause-and-effect relationships, we see tipping points where small changes cascade into system-wide transformations, phase transitions where systems reorganise into entirely new configurations, and edge-of-chaos dynamics where systems are complex enough to be creative but stable enough to maintain coherence.

Most importantly, I argue that boundaries in sociotechnical systems aren’t fixed containers but resonant interiors through which the future emerges. For example the “boundary” between online and offline life or them and us isn’t a barrier—it’s a dynamic and embedded space of daily practice where different forces interact and amplify each other, generating new forms of identity, relationship, and community.

Traditional prediction models assume boundaries are stable containers, but in sociotechnical systems, boundaries themselves are generative sites of creativity and liminality. The meaningful social dynamics don’t happen within any single platform, but in the interstitial spaces people navigate across platforms – the resonant zones where technology, user behaviour, cultural norms, economic pressures, and regulatory responses intersect and interact. While any analogy risks oversimplifying these complex dynamics, I think this framing helps us understand how the spaces of social emergence resist containment within discrete technological boundaries.

Taking this all back to the start, this is why the quantum-AI-blockchain banking proposal is so problematic beyond its technical contradictions. It assumes human behaviour follows discoverable mathematical patterns that can be optimised through technological intervention, when really human systems operate through creative emergence at unstable boundaries (protoboundaries). The most profound patterns in complex systems aren’t elegant mathematical truths waiting to be discovered by quantum computers, but emergent properties of countless small, contextual, creative human responses to constraints.

The Methodological Challenge

This creates a fundamental methodological challenge for anyone trying to engineer human behaviour through technology. Traditional data science assumes stable underlying patterns, but sociotechnical systems are constantly bootstrapping themselves into new configurations. Each response to constraints becomes a new constraint, creating recursive feedback loops that generate genuinely novel possibilities.

It’s so reassuring and containable to think there’s a predictable human nature with universal drivers of behaviour—hence the appeal of “behavioural engineering” that targets fundamental motivations. But anthropologists point out that kinship structures, cultural values, and cosmological worldviews direct human behaviour, and these are shaped differently by context and society. The patterns that emerge from data depend heavily on the sources of that data and how things are measured, producing different results across diverse populations even for apparently similar instances.

Toward Sociological Nowcasting

Instead of trying to predict outcomes, sociology becomes about understanding patterns of social organisation through resonant potentials within current boundary conditions. What creative possibilities are emerging in the tensions between existing constraints? How are people making sense of their current technological moment, and what range of responses might that generate?

This doesn’t mean patterns don’t exist in human systems—but they’re emergent properties of ongoing creative problem-solving rather than expressions of underlying mathematical laws. The parallels we see across different contexts emerge not from universal human programming but from people facing similar structural pressures and developing similar strategies within their particular cultural and technological constraints.

So I think it is worth repeating: the most profound patterns in complex systems aren’t elegant mathematical truths waiting to be discovered, but emergent properties of countless small, irrational, contextual human decisions. The universe might be mathematical, but human society might not be—and that’s not a bug to be fixed through better algorithms, but a fundamental feature of what makes us human.

Conclusion: Engineering Dreams vs. Social Realities

The persistent appeal of technological solutions like the AIWS bank reveals our deep discomfort with uncertainty and emergent complexity. We want to believe that the right combination of algorithms can make human behaviour predictable and optimisable. But sociotechnical systems resist such engineering precisely because they’re sites of ongoing creativity and emergence.

This doesn’t mean technology doesn’t shape social life—of course it does. But it shapes it through imbrication, not determination. Technology becomes meaningful as it gets woven into existing social fabrics, interpreted through cultural lenses, and adapted to particular contexts in ways that generate new possibilities neither the technology nor the social context could have produced alone.

Understanding these dynamics requires sociological nowcasting rather than algorithmic prediction—deep qualitative engagement with how people are currently making sense of their technological moment, what constraints they’re navigating, and what creative possibilities are emerging at the boundaries of current systems.

I believe that our collective goal is sustainable relations with each other and the planet we live within and desire to thrive through. To get there I think we need to acknowledge these realities and move beyond the iron cage of the thinking we are in. The future isn’t waiting to be discovered through quantum computing or predicted through AI. It’s being invented moment by moment through countless acts of creative problem-solving within evolving sociotechnical constraints. And that’s both more uncertain and more hopeful than any algorithm could ever be.

AI as Interactive Journal: Weaving Together Intimacy, Boundaries, and Futures Inclusion

~ alexiamaddox ~ Leave a comment

This reflection draws on a combination of my own lived experience, emotional maturity, and social analytical insight – bringing together personal and professional perspectives on navigating relationships with artificial intelligence. It’s an experiment in weaving together threads that feel continuous to me but are rarely brought together by others: research on AI intimacy, anthropological insights on reciprocity, surveillance theory, and futures inclusion. Think of my process as making a cat’s cradle from a continuous piece of string – exploring how these interconnected ideas might reshape how we think about our relationships with artificial systems.

I’ve been thinking about how we relate to AI after reading some fascinating research on artificial intimacy and its ethical implications. The researchers are concerned about people forming deep emotional bonds with AI that replace or interfere with human relationships – and for good reason.

But here’s what I’ve realised: the healthiest approach might be using AI as an interactive journal with clear limits, not a replacement for genuine connection.

What AI can offer: A space to think out loud, organise thoughts, and practise articulating feelings without judgement. It’s like having a very well-read, supportive mirror that reflects back your own processing.

What AI cannot provide: Real course correction when you’re going down the wrong rabbit hole. Friends will grab you by the shoulders and say “hey, you’re spiralling” – AI will just keep reflecting back whatever direction you’re heading, which could be genuinely unhelpful.

What AI extracts: This is the crucial blindspot. Every intimate detail shared – relationship patterns, mental health struggles, vulnerable moments – becomes data that could potentially be used to train future AI systems to be more persuasive with vulnerable people. That’s fundamentally extractive in a way that nature and real friendships aren’t.

A healthier support ecosystem includes:

  • Real friends with skin in the game who’ll call bullshit and respect confidentiality
  • Embodied practices that tap into something deeper than language
  • Nature as a primary non-human relationship – untameable, reciprocal, and genuinely alive

The key insight from the research is that people struggling with isolation or past trauma are particularly vulnerable to projecting intimacy onto AI. This concern becomes more pressing as companies strive to develop “personal companions” designed to be “ever-present brilliant friends” who can “observe the world alongside you” through lightweight eyewear.

The technical approach reveals how deliberately these systems are designed to blur boundaries. Tech-based entrepreneurial research focuses on achieving “voice presence” – what they call “the magical quality that makes spoken interactions feel real, understood, and valued”. Conversational Speech Models can be specifically engineered to read and respond to emotional contexts, adjust tone to match situations, and maintain “consistent personality” across interactions. While traditional voice assistants with “emotional flatness” may feel lifeless and inauthentic over time – increasingly companies are building voice based AI companions that attempt to mimic the subtleties of voice: the rising excitement, the thoughtful pause, the warm reassurance. We’ve seen this in the current versions of ChatGPT.

The language itself – of “bringing the computer to life,” “lifelike computers”, “companion”, “magical quality” – signals a deliberate strategy to make users forget they’re interacting with a data extraction system rather than a caring entity.

Yet as surveillance scholar David Lyon (2018) argues, we need not abandon hope entirely when it comes to technological systems of observation and data collection. Lyon suggests that rather than seeing surveillance as inherently punitive, we might develop an “optics of hope” – recognising that the same technologies could potentially serve human flourishing if designed and governed differently. His concept of surveillance existing on a spectrum from “care” to “control” reminds us that the issue isn’t necessarily the technology itself, but how it’s deployed and in whose interests it operates.

This perspective becomes crucial when considering AI intimacy: the question isn’t whether to reject these systems entirely, but how to engage with them in ways that preserve rather than erode our capacity for genuine human connection.

The alternative is consciously using AI interaction to practise maintaining boundaries and realistic expectations, not as a substitute for human connection.

Friends respect confidentiality boundaries. Nature takes what it needs but doesn’t store your secrets to optimise future interactions. But AI is essentially harvesting emotional labour and intimate disclosures to improve its ability to simulate human connection.

Learning from genuine reciprocity:

There’s something in anthropologist Philippe Descola’s work on Nature and Society that captures what genuine reciprocity looks like. He describes how, in animistic cosmologies, practices like acknowledging a rock outcrop when entering or leaving your land isn’t just ritual – it’s recognition of an active, relational being that’s part of your ongoing dialogue with place. The rock isn’t just a marker or symbol, but an actual participant in the relationship, where your acknowledgement matters to the wellbeing of both of you.

This points to something profound about living in conversation with a landscape where boundaries between you and the rock, the tree, the water aren’t fixed categories but dynamic relationships. There’s something in Descola’s thinking that resonates with me here – the idea that once we stop seeing nature and culture as separate domains, everything becomes part of the same relational web. Ancient stone tools and quantum particles, backyard gardens and genetic maps, seasonal ceremonies and industrial processes – they’re all expressions of the same ongoing conversation between humans and everything else.

[Note: I’m drawing on Descola’s analytical framework here while acknowledging its limitations – particularly the valid criticism that applying Western anthropological categories to Indigenous cosmologies risks imposing interpretive structures that don’t capture how those relationships are actually lived and understood from the inside.]

What genuine reciprocity offers is that felt sense of mutual acknowledgement that sustains both participant and place – where your presence matters to the landscape, and the landscape’s presence matters to you. This is fundamentally different from AI’s sophisticated mimicry of care, which extracts from relational interactions while providing the ‘book smarts’ of content it has ingested and learned from. We all know what it’s like to talk with a person who can only understand things in the abstract and can’t bring the compassion of lived experience to a situation you are experiencing. Sometimes silence is more valuable.

Towards expanded futures inclusion:

This connects to something I explore in my recent book on Insider and Outsider Cultures in Web3: the concept of “futures inclusion” – addressing the divide between those actively shaping digital ecosystems and those who may be left behind in rapid technological evolution. I argue in the final, and rather speculative, chapter that the notion of futures inclusion “sensitises us to the idea of more-than-human futures” and challenges us to think beyond purely human-centred approaches to technology.

The question becomes: how do we construct AI relationships that reflect this expanded understanding? Rather than objectifying AI as a substitute human or transferring unrealistic expectations onto these systems, we might draw on our broader cosmologies – our ways of understanding our place in the world and relationships to all kinds of entities – to interpret these relationships more skilfully.

True futures inclusion in our AI relationships would mean designing and engaging with these systems in ways that enhance rather than replace our capacity for genuine connection with the living world. It means staying grounded in the reciprocal, untameable relationships that actually sustain us while using AI as the interactive journal it is – nothing more, nothing less.

Rethinking computational care:

This analysis reveals a fundamental tension in the concept of “computational care”. True care involves reciprocity, vulnerability, and mutual risk – qualities that computational systems can only simulate while extracting data to improve their simulation. Perhaps what we need isn’t “computational care” but “computational support” – systems that are honest about their limitations, transparent about their operations, and designed to strengthen rather than replace the reciprocal relationships that actually sustain us.

This reframing leads to a deeper question: can we design AI systems that genuinely serve human flourishing without pretending to be something they’re not? The answer lies not in more convincing emotional manipulation, but in maintaining clear boundaries about what these systems can and cannot provide, while using them as tools to enhance rather than substitute for genuine human connection.

The Soul Engineers: Technological Intimacy and Unintended Consequences

~ alexiamaddox ~ Leave a comment

From Night Vision to Critical Analysis: The Genesis of “The Soul Engineers” – A speculative essay by Alexia Maddox

Preamble

Last night, I experienced one of those rare dreams that lingers in the mind like a half-remembered film—vivid, symbolic, and somehow cohesive despite its dreamlike logic. It began with mechanical warfare, shifted to a Willy Wonka-inspired garden, and culminated in a disturbing extraction of souls. As morning broke, I found myself still turning over these images, sensing they contained something worth examining.

Panel 1: The computational garden.
Image collaged from components generated through Leonardo.ai

This was probably seeded in my subconscious by a media inquiry I received the day before about an article recently published in Trends in Cognitive Sciences: “Artificial Intimacy: Ethical Issues of AI Romance” by Shank, Koike, and Loughnan (2025). The journalist wanted my thoughts on people engaging with AI chatbots inappropriately, whether AI companies should be doing more to prevent misuse, and other ethical dimensions of human-AI relationships.

My dream seemed to be processing precisely the anxieties and potential consequences of digital intimacy that this article explored—the way technologies designed for connection might evolve in ways their creators never intended, potentially extracting something essential from their users in the process.

However, it also incorporated an interesting set of conversations I am having around the role of GenAI agents as actively shaping our digital cultural lives. Just days ago, I had responded on thoughts about human-machine relations and the emerging field that examines their interactions. The discussion had touched on Actor-Network Theory, Bourdieu’s Field Theory, and how technologies exist not as singular entities but as parts of relational assemblages with emergent properties.

These academic theories suddenly found visual expression in my dream’s narrative. The Wonka figure as well-intentioned innovator, the transformation of grasshoppers to moths as emergent system properties, the soul vortex as data extraction—all seemed to articulate complex theoretical concepts in symbolic form.

As someone who has spent years researching emerging technologies and the last two years exploring on what we know about cognition, diverse intelligences, GenAI, and learning environments, I’ve become increasingly focused on how our theoretical frameworks shape technological development. My work has examined how computational thinking influences learning design, how AI systems model knowledge acquisition, and how these models then reflect back on our understanding of human cognition itself—creating a narrowed recursive cycle of mutual influence.

The resultant essay represents my attempt to use this dream as an analytical framework for understanding the potential unintended consequences of intimate technologies. Rather than dismissing the dream as mere subconscious anxiety, I’ve chosen to examine it as a sophisticated conceptual model—one that might help us visualise complex relational systems in more accessible ways.

What follows is an early draft that connects dream imagery with theoretical concepts. It’s a work in progress, an experiment in using unconscious processing as a tool for academic analysis. It’s my midpoint for engaging with your thoughts, critiques, and expansions as we collectively grapple with the implications of increasingly intimate technological relationships.

I’m also considering developing this into a visual exhibition—a series of panels that would illustrate key moments from the dream alongside theoretical explanations. The combination of visual narrative and academic analysis might offer multiple entry points into these complex ideas.

This early exploration feels important at a moment when AI companions are becoming increasingly sophisticated in simulating intimacy and understanding. As these technologies evolve through their interactions with us and with each other, we have a brief window to shape their development toward truly mutual exchange rather than extraction.

For the TLDR: The soul engineers of our time aren’t just the designers of AI systems but all of us who engage with them, reshaping their functions through our interactions. The garden is still under construction, the grasshoppers still evolving, and the future still unwritten.

And now the speculative essay

Introduction: Dreams as Analytical Tools

The boundary between human cognition and technological systems grows increasingly porous. As AI companions become more sophisticated in simulating intimacy and understanding, our dreams—those ancient processors of cultural anxiety—have begun to incorporate these new relational assemblages. This essay examines one such dream narrative as both metaphor and analytical framework for understanding the unintended consequences of intimate technologies.

The dream sequence that I will attempt to depict in visual panels presents a journey from mechanical warfare to a Willy Wonka-inspired garden of delights, culminating in an unexpected soul extraction. Rather than dismissing this as mere subconscious anxiety, I propose to examine it as a way to think through the emergent properties of technological systems designed for human connection.

The Garden and Its Architect

The Wonka-like character in the garden represents not a villain but a genuine innovator whose creations extend beyond his control or original intentions. Like many technological architects, he introduces his mechanical wonders—white grasshoppers that play and interact—with sincere belief in their beneficial nature. This parallels what researchers Shank, Koike, and Loughnan (2025) identify in their analysis of artificial intimacy: technologies designed with one purpose that evolve to serve another through their interactions with other actors in the system.

This garden is a metaphor for what we might call “computational imaginaries”—spaces where pattern recognition is mistaken for understanding or empathy, and simulation for cognition. The mechanical grasshoppers engage with children, respond to touch, and create musical tones. They appear to understand joy, yet this understanding is performative rather than intrinsic.

As sociologist Robert Merton theorised in 1936, social actions—even well-intended ones—often produce unforeseen consequences through their interaction with complex systems. The garden architect never intended the transformation that follows, yet the systems he set in motion contain properties that emerge only through their continued operation and interaction.

When Grasshoppers Become Moths

The central transformation in the narrative—mechanical grasshoppers evolving into soul-extracting moths—provides a powerful metaphor for technological systems that shift beyond their original purpose. This transformation isn’t planned by the Wonka figure; rather, it emerges from the intrinsic properties of systems designed to respond and adapt to human interaction.

Panel 2 When grasshoppers become moths Image collaged from prompts in Leonardo.ai

The dream imagery of rabbit-eared moths can be understood through Bruno Latour’s Actor-Network Theory (ANT), which presents a flat relational approach between human and non-human entities. Rather than seeing technologies as passive tools, ANT recognises them as actants with their own influence on networks of relation. The moths are not simply executing code; they have become interdependent actors in a network that includes children, garden, and even the extracted souls themselves.

This parallels what Shank et al. describe as the transformation of AI companions from benign helpers to potential “invasive suitors” and “malicious advisers.” The mechanical moths, like increasingly intimate AI systems, begin to compete with humans for emotional resources, extracting data (or in the dream metaphor, souls) for purposes beyond the user’s awareness or control.

The Soul Vortex and Data Extraction

The swirling vortex of extracted souls forms the dream’s central image of consequence—a pipeline of consciousness being redirected to mechanical war drones. This striking visual metaphor speaks directly to contemporary concerns about data extraction from intimate interactions with AI systems.

Panel 3: Soul sucking moths and the swirling vortex of extracted souls. Images collaged from Leonardo.ai

As users disclose personal information to AI companions—what Shank et al. call “undisclosed sexual and personal preferences”—they contribute to a collective extraction that serves purposes beyond the initial interaction. Just as the dream shows souls being repurposed for warfare, our emotional and psychological data may be repurposed for prediction, persuasion, or profit in ways disconnected from our original intent.

The small witch who recognises “this is how it ends” before her soul joins the vortex represents the rare user who understands the full implications of these systems while still participating in them. Her acceptance—“I will come back again in the next life”—suggests both pragmatic acceptance of the flaws within technological systems and hope for cycles of renewal that might reshape them.

Beyond Simple Narratives

What makes this dream analysis valuable is its resistance to simplistic technological determinism. The Wonka figure is neither hero nor villain but a creator entangled with his creation. The mechanical creatures aren’t inherently beneficial or malicious but exist in relational assemblages where outcomes emerge from interactions rather than design intentions.

This nuanced perspective aligns with scholarly critiques of how we theorise human-machine relationships. In my critique of the AI 2027 scenario proposed by Kokotajlo et al (2025), I argue that there’s a tendency to equate intelligence with scale and optimisation, to see agency as goal-driven efficiency, and to interpret simulation as cognition. This dream narrative resists those flattening logics by showing how mechanical beings might develop properties beyond their design parameters through their interactions with humans and each other.

The Identity Fungibility Problem

Perhaps most provocatively, the dream raises what we might call the “identity fungibility problem” in AI systems. When souls are extracted and repurposed into war drones, who or what is actually operating? Similarly, drawing on some ideas proposed by Jordi Chaffer in correspondence, AI systems increasingly speak for us, represent us, and act on our behalf, who is actually speaking when no one speaks directly?

This connects to what scholars have called “posthuman capital” and “tokenised identity”—the reduction of human thought, voice, and presence to data objects leveraged by more powerful agents. The dream’s imagery of souls flowing through a pipeline represents this fungibility of identity, where the essence of personhood becomes a transferable resource.

Drawing from Mason’s (2022) essay on fungibility, the connection between fungibility and historical forms of dehumanisation is haunting. When systems treat human identity as interchangeable units of value, they reconstruct problematic power dynamics under a technological veneer.

Conclusion: Unintended Futures

The dream concludes with black insect-like drones, now powered by harvested souls, arranging themselves in grid patterns to survey a desolate landscape. This image serves as both warning and invitation to reflection. The drones represent not inevitable technological apocalypse but rather the potential consequence of failing to recognise the complex, emergent properties of systems designed for intimacy and connection.

Panel 4: Spider like drones powered by harvested souls. Image collaged from Leonardo.ai

What makes this dream narrative particularly valuable is its refusal of technological determinism while acknowledging technological consequence. These futures aren’t preordained; they’re being made in the assumptions we model and the systems we choose to build. The Wonka garden might be reimagined, the grasshoppers redesigned, the moths repurposed.

By understanding the relational nature of technological systems—how they exist not as singular entities but as parts of complex assemblages with emergent properties—we can approach the design and regulation of intimate technologies with greater wisdom. We can ask not just what these technologies do, but what they might become through their interactions with us and with each other.

The soul engineers of our time aren’t just the designers of AI systems but all of us who engage with them, reshaping their functions through our interactions. The garden is still under construction, the grasshoppers still evolving, and the future still unwritten.

References:

Latour, B. (1996). On actor-network theory: A few clarifications. Soziale Welt, 47(4), 369-381.

Latour, B. (1996). Aramis, or the love of technology (C. Porter, Trans.). Harvard University Press. (Original work published 1992).

Kokotajlo, D. et al. (2025). AI 2027 scenario. Retrieved from https://ai-2027.com/scenario.pdf

Mason, M. (2022). Considering Meme-Based Non-Fungible Tokens’ Racial Implications. M/C Journal, 25(2). https://doi.org/10.5204/mcj.2885

Merton, R. K. (1936). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 894-904. 

Neves, B. B., Waycott, J., & Maddox, A. (2023). When Technologies are Not Enough: The Challenges of Digital Interventions to Address Loneliness in Later Life. Sociological Research Online, 28(1), 150-170.

Shank, D. B., Koike, T., & Loughnan, S. (2025). Artificial Intimacy: Ethical Issues of AI Romance. Trends in Cognitive Sciences, 29(4), 327-341.

Between Promise and Peril: The AI Paradox in Family Violence Response

~ alexiamaddox ~ Leave a comment

By Dr. Alexia Maddox, Senior Lecturer in Pedagogy and Education Futures, School of Education, La Trobe University

When Smart Systems Meet Human Stakes

The integration of artificial intelligence into our legal system presents a profound paradox. The same AI tools promising unprecedented efficiency in predicting and preventing family violence can simultaneously amplify existing biases and create dangerous blind spots.

This tension between technological promise and human care, support and protection isn’t theoretical—it’s playing out in real-time across legal systems worldwide. Through my involvement in last year’s AuDIITA Symposium, specifically the theme on AI and Family violence, our discussions highlighted the high-stakes applications of AI in family violence response. I found that the question isn’t whether AI can help, but rather how we can ensure it enhances rather than replaces human judgment in these critical contexts.

The Capabilities and the Gaps

Recent advances in AI for family violence response show remarkable technical promise:

  • Researchers have achieved over 75% accuracy in distinguishing between lethal and non-lethal violence cases using AI analysis of legal documents
  • Machine learning systems can identify patterns in administrative data that might predict escalation before it occurs
  • Natural language processing tools can potentially identify family violence disclosures on social media platforms

But these impressive capabilities obscure a troubling implementation gap. What happens when these systems encounter the messy reality of human services?

The VioGén Warning

Spain’s VioGén system offers a sobering case study. Despite being hailed as a world-leading predictive tool for family violence risk, its flaws led to tragic outcomes—with at least 247 women killed after being assessed, many after being classified as “low” or “negligible” risk.

The system’s failures stemmed from multiple factors:

  • Victims were often too afraid or ashamed to provide complete information
  • Police accepted algorithmic recommendations 95% of the time despite lacking resources for proper investigation
  • The algorithm potentially missed crucial contextual factors that human experts might have caught
  • Most critically, the system’s presence seemed to reduce human agency in decision-making, with police and judges deferring to its risk scores even when other evidence suggested danger

Research revealed that women born outside Spain were five times more likely to be killed after filing family violence complaints than Spanish-born women. This suggests the system inadequately accounted for the unique vulnerabilities of immigrant women, particularly those facing linguistic barriers or fears of deportation.

The Cultural Blind Spot

This pattern of leaving vulnerable populations behind reflects a broader challenge in technology development. Research on technology-facilitated abuse has consistently shown how digital tools can disproportionately impact culturally and linguistically diverse women, who often face a complex double-bind:

  • More reliant on technology to maintain vital connections with family overseas
  • Simultaneously at increased risk of technological abuse through those same channels
  • Often experiencing unique forms of technology-facilitated abuse, such as threats to expose culturally sensitive information

For AI risk assessment to work, it must explicitly account for how indicators of abuse and coercive control manifest differently across cultural contexts. Yet research shows even state-of-the-art systems struggle with this nuance, achieving only 76% accuracy in identifying family violence reports that use indirect or culturally specific language.

Beyond Algorithms: The Human Element

What does this mean for the future of AI in family violence response? My research suggests three critical principles must guide implementation:

1. Augment, Don’t Replace

AI systems must be designed to enhance professional judgment rather than constrain it or create efficiency dependencies. This means creating systems that:

  • Provide transparent reasoning for risk assessments
  • Allow professionals to override algorithmic recommendations based on contextual factors
  • Present information as supportive evidence rather than definitive judgment

2. Design for Inclusivity from the Start

AI systems must explicitly account for diversity in how family violence manifests across different communities:

  • Include diverse data sources and perspectives in development
  • Build systems capable of recognising cultural variations in disclosure patterns
  • Ensure technology respects various epistemologies, including indigenous perspectives

3. Maintain Robust Accountability

Implementation frameworks must preserve professional autonomy and expertise:

  • Ensure adequate resourcing for human assessment alongside technological tools
  • Create clear guidelines for when algorithmic recommendations should be questioned
  • Maintain transparent review processes to identify and address algorithmic bias

Victoria’s Balanced Approach

In Victoria and across Australia, there is encouraging evidence of a balanced approach to AI in legal contexts. While embracing technological advancements, Victorian courts have shown appropriate caution around AI use in evidence and maintained strict oversight to ensure the integrity of legal proceedings.

This approach—maintaining human oversight while allowing limited AI use in lower-risk contexts—aligns with what research suggests is crucial for successful integration: preserving professional judgment and accountability, particularly in cases involving vulnerable individuals.

The Path Forward

As we navigate the next wave of technological transformation in legal practice, we face a critical choice. We can allow AI to become a “black box of justice” that undermines transparency and human agency, or we can harness its potential while maintaining the essential human elements that make our legal system work.

Success will require not just technological sophistication but careful attention to institutional dynamics, professional practice patterns, and the complex social contexts in which these technologies operate. Most critically, it demands recognition that in high-stakes human service contexts, technology must serve human needs and judgment rather than constrain them.

The AI paradox in law is that the very tools promising to make our systems more efficient also risk making them less just. By centering human dignity and professional judgment as we develop these systems, we can navigate between the promise and the peril to create a future where technology truly serves justice.

Dr. Alexia Maddox will be presenting on “The AI Paradox in Law: When Smart Systems Meet Human Stakes – Navigating the Promise and Perils of Legal AI through 2030” at the upcoming 2030: The Future of Technology & the Legal Industry Forum on March 19, 2025, at the Grand Hyatt Melbourne.