technology

The Irreplaceable Human Skill: Why Generative AI Can’t Teach Students to Judge Their Own Work

~ alexiamaddox ~ Leave a comment

A note to readers: I’m writing this in the thick of marking student submissions – the most grinding aspect of academic work. My brain fights against repetitive rote labour and goes on tangents to keep me entertained. What follows emerged from that very human need to find intellectual stimulation in the midst of administrative necessity.

There’s considerable discussion that our distinction as creators and thinkers from Generative AI content production lies in creativity and critical thinking linked to innovation. But where does the hair actually split? Are we actually replaceable by robots or will they atrophy our critical thinking skills by doing the work for us? Will we just get dummer and less capable to tie our own shoe laces – like most fear based reporting suggests? I think we are asking the wrong questions.

Here is a look at what is actually going on, on the ground. A student recently asked me for detailed annotations on their assignment—line-by-line corrections marking every error. They wanted me to do the analytical work of identifying problems in their writing. This request highlights a fundamental challenge in education: the difference between fixing problems and developing the capacity to recognise them. More importantly, it reveals where the Human-Generative AI distinction becomes genuinely meaningful.

Could Generative AI theoretically teach students to judge their own work? Perhaps, through Socratic questioning or scaffolded self-assessment prompts. But that’s not how students actually use these tools. Or want to use them, apparently. A discussion I had with a tech developer working in a tutoring company utilising Generative AI in the teaching/learning process mentioned that students got annoyed by the Socratic approach when they encountered it. So there goes that morsel of hope.

The Seductive Trap of Generative AI Writing Assistance

Students increasingly use Generative AI tools for grammar checking, expression polishing, and even content generation. These tools are seductive because they make writing appear better—more polished, more confident, more academically sophisticated. But here’s the problem: Generative AI tools are fundamentally sycophantic and don’t course correct misapprehensions. They won’t tell a student their framework analysis is conceptually flawed, their citations are inaccurate, or their arguments lack logical consistency. Instead, they’ll make poorly reasoned content sound more convincing.

This creates a dangerous paradox: students use Generative AI to make their work sound rigorous and sophisticated, but this very process prevents them from developing the judgement to recognise what genuine rigour looks like. They can’t evaluate what they clearly don’t know – that their work isn’t conceptually aligned, coherently logical, or correctly interpreting sources – because the AI has dressed their half-formed understanding in authoritative-sounding language.

I have encountered several submissions across different subjects that exemplified this perfectly: beautifully written but containing fundamental errors in framework descriptions, questionable source citations, and confused theoretical applications. The prose was polished, the structure clear, but the content revealed gaps in understanding that no grammar checker could identify or fix. The student had learned to simulate the appearance of academic rigour without developing the capacity to recognise genuine scholarly quality.

Where the Hair Actually Splits

Generative AI can actually be quite “creative” in generating novel combinations of ideas, and it can perform certain types of critical analysis when clearly guided and bounded. What it fundamentally cannot do is develop the evaluative judgement to recognise quality, coherence, and accuracy in complex, contextualised work. It has no capacity for self reflection and meaning making (at the moment), we do.

The distinction isn’t between:

  • Generating creative output (which Generative AI can somewhat do)
  • Performing critical analysis (which generative AI can also somewhat do)

Rather, it’s between:

  • Creating sophisticated looking content (which Generative AI increasingly excels at)
  • Judging the quality of that content in context (which requires human oversight and discernment)

Generative AI can produce beautifully written, seemingly sophisticated arguments that are conceptually flawed. It can create engaging content that misrepresents sources or conflates different frameworks. What it cannot do is step back and recognise “this sounds polished but the underlying logic is problematic” or “this citation doesn’t actually support this claim.”

The irreplaceable human skill isn’t creativity per se—it’s the capacity for metacognitive evaluation: the ability to assess one’s own thinking, to recognise when arguments are coherent versus merely convincing, to distinguish between surface-level polish and deep understanding.

What Humans Bring That AI Cannot

The irreplaceable human contribution to education isn’t information delivery—AI is increasingly able to do that pretty efficiently (although there is a lot of hidden labour in this). It’s developing the capacity for metacognitive evaluation in our students.

This happens through:

Exposure to expertise modelling: Students need to observe how experts think through problems, make quality judgements, and navigate uncertainty. This isn’t just about seeing perfect examples—it’s about witnessing the thinking process behind quality work.

Calibrated feedback loops: Human educators can match feedback to developmental readiness, escalating complexity as students build capacity. We recognise when to scaffold and when to challenge.

Critical engagement with authentic problems: Unlike AI-generated scenarios, real-world applications come with messy complexities, competing priorities, and value judgements that require human judgement, discernment and social intelligence.

Social construction of standards: Quality isn’t just individual—it’s negotiated within communities of practice. Students learn to recognise “good work” through dialogue, peer comparison, and collective sense-making.

Refusing to spoon-feed solutions: Perhaps most importantly, human educators understand when not to provide answers. When my student asked for line-by-line corrections, providing them would have created dependency rather than developing their evaluative judgement. The metacognitive skill of self-assessment can only develop when students are required to do the analytical work themselves.

The Dependency Problem

When educators provide line-by-line corrections or when students rely on Generative AI for error detection in thinking, writing or creating, we create dependency rather than capacity. Students learn to outsource quality judgement instead of developing their own ability to recognise problems.

The student who asked for detailed annotations was essentially asking me to do their self-assessment for them. But self-regulated learning—the ability to monitor, evaluate, and adjust one’s own work—is perhaps the most crucial skill we can develop. Without it, students remain permanently dependent on external validation and correction.

Teaching Evaluative Judgement in a Generative AI World

This doesn’t mean abandoning Generative AI tools entirely. Rather, it means being intentional about what we ask humans to do versus what we delegate to technology:

Use Generative AI for: Initial drafting, grammar checking, formatting, research organisation—the mechanical aspects of work.

Reserve human judgement for: Source evaluation, argument coherence, conceptual accuracy, ethical reasoning, quality assessment—the thinking that requires wisdom, not just processing.

In my own practice, I provide rubric-based feedback that requires students to match criteria to their own work. This forces them to develop pattern recognition and quality calibration. It’s more cognitively demanding than receiving pre-marked corrections, but it builds the evaluative judgement they’ll need throughout their careers.

The Larger Stakes

The question of human versus Generative AI roles in education isn’t just pedagogical—it’s about what kind of thinkers we’re developing. If students learn to outsource quality judgement to Generative AI tools, we’re creating a generation that can produce polished content but can’t recognise flawed reasoning, evaluate source credibility, or build intellectual capacity and critical reasoning skills.

This is why we need to build self-evaluative judgement in students – not just critical thinking and creative processes more broadly. The standard educational discourse about “21st century skills” focuses on abstract categories like critical thinking and creativity, but misses this more precise distinction: the specific metacognitive capacity to evaluate the quality of one’s own intellectual work.

This self-evaluative judgement operates laterally across disciplines rather than being domain-specific, and it’s fundamentally metacognitive because it requires thinking about thinking. It addresses the actual challenge students face in a Generative AI world: distinguishing between genuine understanding and polished simulation of understanding. A student might articulate sophisticated pedagogical concepts yet be unable to evaluate whether their own framework descriptions are accurate or their citations valid.

The unique human contribution isn’t delivering perfect feedback—it’s teaching students to become their own quality assessors. That capacity for self-evaluation, for recognising what makes work meaningful and rigorous, remains irreplaceably human.

In a world where Generative AI can make anyone’s writing sound professional, the ability to think critically about one’s own work becomes more valuable, not less. That’s the expertise that human educators bring to the table—not just knowing the right answers, but developing in students the judgement to recognise quality thinking when they see it, including in their own work.

When Prediction Fails: Why Quantum-AI-Blockchain Dreams Miss the Social Reality

~ alexiamaddox ~ Leave a comment

A sociological perspective on why technical solutions keep missing the human element

The Hype Moment

Consider this recent announcement from the Boston Global Forum’s “Boston Plurality Summit“: they’re unveiling an “AIWS Bank and Digital Assets Model” that combines quantum AI, blockchain technology, and predictive analytics to “unite humanity through technology”. You know that the canary in my head is shouting “unite what?, how?”. The press release promises “zero-latency transactions”, “quantum AI for predictive analytics”, and a “global blockchain network” that will somehow revolutionise banking.

As someone who studies sociotechnical systems, this announcement is fascinating—not for what it promises to deliver, but for what it reveals about our persistent fantasy that human behaviour can be engineered, predicted, and optimised through technological solutions.

::Pats head – Provides tissue::

The Technical House of Cards

Let’s start with a question of technical possibilities. “Zero-latency transactions” on a global blockchain network defies current technological reality. This was my first eyebrow raise. According to recent analysis, even the fastest blockchains operate with latency measured in hundreds of milliseconds to seconds, whilst Visa reportedly has the theoretical capacity to execute more than 65,000 transactions per second compared to Solana’s 2024 rate of 1,200-4,000 TPS and Ethereum’s roughly 15-30 TPS. Gas fees during network congestion can spike to significant sums per transaction. On Ethereum, fees can exceed 20USD during peak times, with some transactions reaching extreme levels like 377 gwei, and historical spikes exceeding 100USD during events like NFT mania. Even on the much cheaper Solana network, which typically costs around 0.0028USD per transaction, fees can occasionally spike during congestion—hardly the foundation for revolutionary banking.

Then there’s the “quantum AI” buzzword. Theoretically quantum computing could actually break most current blockchain cryptography rather than enhance it. The blockchain community is scrambling to develop quantum-resistant algorithms precisely because quantum computers pose an existential threat to current security models. Adding AI on top makes even less sense—if quantum computing could handle complex optimisation and verification tasks, what would AI add?

But the technical contradictions aren’t the most interesting part. What’s fascinating is the underlying assumption that human financial behaviour follows discoverable mathematical patterns that can be optimised through technological intervention.

The Pattern Recognition Fantasy

This assumption reflects a deeper misunderstanding about the nature of patterns in human systems. Which I should know, because I study them. In physical systems—planetary orbits, gravitational forces, electromagnetic fields—patterns emerge because they’re constrained by unchanging laws. Newton’s and Einstein’s equations work because there are actual forces creating predictable relationships. The mathematics describes underlying physical reality.

Human systems operate fundamentally differently. What we call “patterns” in human behaviour might be statistical accidents emerging from millions of independent, context-dependent choices. Your shopping behaviour isn’t governed by fundamental forces—it’s shaped by your mood, what ad you saw, whether you got enough sleep, a conversation with a friend, cultural context, economic pressures, and countless other variables.

Consider the difference between how neural networks and quantum computing approach pattern recognition. Neural networks are essentially sophisticated approximation engines—they learn patterns through massive trial-and-error, requiring enormous datasets and computational brute force to produce probabilistic outputs that can be wrong. They’re like having thousands of people manually checking every possible combination to find a pattern.

Quantum computing, by contrast, approaches problems through superposition—exploring multiple solution paths simultaneously to understand the underlying mathematical structure that creates patterns in the first place. It’s elegant, precise, and powerful for problems with discoverable mathematical relationships. However, quantum computing currently requires predictable, structured datasets and struggles with the messy, unstructured nature of real-world human data. This is precisely why we still rely on neural networks’ “brute force” approximation approach for dealing with human behaviour—they’re designed to handle noise, inconsistency, and randomness where quantum algorithms would falter.

But what if much real-world human data has no underlying mathematical structure to discover?

Consider this: as I write this analysis, my brain is simultaneously processing quantum mechanics concepts, blockchain technicalities, sociological theory, and source credibility – all whilst maintaining a critical perspective and personal voice. No quantum algorithm exploring mathematical solution spaces could replicate this messy, contextual, creative synthesis. My thinking emerges from countless variables: morning coffee levels, recent conversations, cultural background, academic training, even the frustration of marking student essays that often demonstrates exactly the kind of linear thinking I’m critiquing. This is precisely the kind of complex, non-algorithmic pattern recognition that human systems excel at – and that technological solutions consistently underestimate.

The Emergence of Sociotechnical Complexity

As a sociologist studying sociotechnical imbrications, I’m fascinated by how technology and social structures become so intertwined that they create emergent properties that couldn’t be predicted from either component alone. Human behaviour has emergent regularities rather than underlying laws. People facing similar social pressures might develop similar strategies, but not because of fundamental behavioural programming—because they’re creative problem-solvers working within constraints.

This is why prediction based on historical data can only take you so far. I call my sociological practice “nowcasting”— we have to understand the present moment to have any sense of future potentialities. And we often don’t — I speculate this is because we are more wrapped up in the surface stories we tell ourselves, denial and a refusal to see or accept ourselves as we really are. This challenge is becoming even more complex as AI generates synthetic media that we then consume and respond to, creating a recursive loop where artificial representations of social reality shape actual social behaviour, which in turn feeds back into AI systems to create more synthetic reality. The way people respond to constraints can’t be predicted because their responses literally create new social realities.

Every new payment app, social media trend, or economic crisis creates new ways people think about and use money that couldn’t have been predicted from previous data. Netflix can’t predict what you’ll want to watch because your preferences are being shaped by what Netflix shows you. Financial models break down because they change how people think about money. Social media algorithms can’t predict engagement because they’re constantly reshaping what people find engaging.

Boundaries as Resonant Interiors

I like playing with complexity theory because provides useful language for understanding these dynamics. This is of course despite its generation within the natural sciences that does rely on the explanatory nature of underlying forces. What it offers me is a language that moves beyond linear cause-and-effect relationships, we see tipping points where small changes cascade into system-wide transformations, phase transitions where systems reorganise into entirely new configurations, and edge-of-chaos dynamics where systems are complex enough to be creative but stable enough to maintain coherence.

Most importantly, I argue that boundaries in sociotechnical systems aren’t fixed containers but resonant interiors through which the future emerges. For example the “boundary” between online and offline life or them and us isn’t a barrier—it’s a dynamic and embedded space of daily practice where different forces interact and amplify each other, generating new forms of identity, relationship, and community.

Traditional prediction models assume boundaries are stable containers, but in sociotechnical systems, boundaries themselves are generative sites of creativity and liminality. The meaningful social dynamics don’t happen within any single platform, but in the interstitial spaces people navigate across platforms – the resonant zones where technology, user behaviour, cultural norms, economic pressures, and regulatory responses intersect and interact. While any analogy risks oversimplifying these complex dynamics, I think this framing helps us understand how the spaces of social emergence resist containment within discrete technological boundaries.

Taking this all back to the start, this is why the quantum-AI-blockchain banking proposal is so problematic beyond its technical contradictions. It assumes human behaviour follows discoverable mathematical patterns that can be optimised through technological intervention, when really human systems operate through creative emergence at unstable boundaries (protoboundaries). The most profound patterns in complex systems aren’t elegant mathematical truths waiting to be discovered by quantum computers, but emergent properties of countless small, contextual, creative human responses to constraints.

The Methodological Challenge

This creates a fundamental methodological challenge for anyone trying to engineer human behaviour through technology. Traditional data science assumes stable underlying patterns, but sociotechnical systems are constantly bootstrapping themselves into new configurations. Each response to constraints becomes a new constraint, creating recursive feedback loops that generate genuinely novel possibilities.

It’s so reassuring and containable to think there’s a predictable human nature with universal drivers of behaviour—hence the appeal of “behavioural engineering” that targets fundamental motivations. But anthropologists point out that kinship structures, cultural values, and cosmological worldviews direct human behaviour, and these are shaped differently by context and society. The patterns that emerge from data depend heavily on the sources of that data and how things are measured, producing different results across diverse populations even for apparently similar instances.

Toward Sociological Nowcasting

Instead of trying to predict outcomes, sociology becomes about understanding patterns of social organisation through resonant potentials within current boundary conditions. What creative possibilities are emerging in the tensions between existing constraints? How are people making sense of their current technological moment, and what range of responses might that generate?

This doesn’t mean patterns don’t exist in human systems—but they’re emergent properties of ongoing creative problem-solving rather than expressions of underlying mathematical laws. The parallels we see across different contexts emerge not from universal human programming but from people facing similar structural pressures and developing similar strategies within their particular cultural and technological constraints.

So I think it is worth repeating: the most profound patterns in complex systems aren’t elegant mathematical truths waiting to be discovered, but emergent properties of countless small, irrational, contextual human decisions. The universe might be mathematical, but human society might not be—and that’s not a bug to be fixed through better algorithms, but a fundamental feature of what makes us human.

Conclusion: Engineering Dreams vs. Social Realities

The persistent appeal of technological solutions like the AIWS bank reveals our deep discomfort with uncertainty and emergent complexity. We want to believe that the right combination of algorithms can make human behaviour predictable and optimisable. But sociotechnical systems resist such engineering precisely because they’re sites of ongoing creativity and emergence.

This doesn’t mean technology doesn’t shape social life—of course it does. But it shapes it through imbrication, not determination. Technology becomes meaningful as it gets woven into existing social fabrics, interpreted through cultural lenses, and adapted to particular contexts in ways that generate new possibilities neither the technology nor the social context could have produced alone.

Understanding these dynamics requires sociological nowcasting rather than algorithmic prediction—deep qualitative engagement with how people are currently making sense of their technological moment, what constraints they’re navigating, and what creative possibilities are emerging at the boundaries of current systems.

I believe that our collective goal is sustainable relations with each other and the planet we live within and desire to thrive through. To get there I think we need to acknowledge these realities and move beyond the iron cage of the thinking we are in. The future isn’t waiting to be discovered through quantum computing or predicted through AI. It’s being invented moment by moment through countless acts of creative problem-solving within evolving sociotechnical constraints. And that’s both more uncertain and more hopeful than any algorithm could ever be.

AI as Interactive Journal: Weaving Together Intimacy, Boundaries, and Futures Inclusion

~ alexiamaddox ~ Leave a comment

This reflection draws on a combination of my own lived experience, emotional maturity, and social analytical insight – bringing together personal and professional perspectives on navigating relationships with artificial intelligence. It’s an experiment in weaving together threads that feel continuous to me but are rarely brought together by others: research on AI intimacy, anthropological insights on reciprocity, surveillance theory, and futures inclusion. Think of my process as making a cat’s cradle from a continuous piece of string – exploring how these interconnected ideas might reshape how we think about our relationships with artificial systems.

I’ve been thinking about how we relate to AI after reading some fascinating research on artificial intimacy and its ethical implications. The researchers are concerned about people forming deep emotional bonds with AI that replace or interfere with human relationships – and for good reason.

But here’s what I’ve realised: the healthiest approach might be using AI as an interactive journal with clear limits, not a replacement for genuine connection.

What AI can offer: A space to think out loud, organise thoughts, and practise articulating feelings without judgement. It’s like having a very well-read, supportive mirror that reflects back your own processing.

What AI cannot provide: Real course correction when you’re going down the wrong rabbit hole. Friends will grab you by the shoulders and say “hey, you’re spiralling” – AI will just keep reflecting back whatever direction you’re heading, which could be genuinely unhelpful.

What AI extracts: This is the crucial blindspot. Every intimate detail shared – relationship patterns, mental health struggles, vulnerable moments – becomes data that could potentially be used to train future AI systems to be more persuasive with vulnerable people. That’s fundamentally extractive in a way that nature and real friendships aren’t.

A healthier support ecosystem includes:

  • Real friends with skin in the game who’ll call bullshit and respect confidentiality
  • Embodied practices that tap into something deeper than language
  • Nature as a primary non-human relationship – untameable, reciprocal, and genuinely alive

The key insight from the research is that people struggling with isolation or past trauma are particularly vulnerable to projecting intimacy onto AI. This concern becomes more pressing as companies strive to develop “personal companions” designed to be “ever-present brilliant friends” who can “observe the world alongside you” through lightweight eyewear.

The technical approach reveals how deliberately these systems are designed to blur boundaries. Tech-based entrepreneurial research focuses on achieving “voice presence” – what they call “the magical quality that makes spoken interactions feel real, understood, and valued”. Conversational Speech Models can be specifically engineered to read and respond to emotional contexts, adjust tone to match situations, and maintain “consistent personality” across interactions. While traditional voice assistants with “emotional flatness” may feel lifeless and inauthentic over time – increasingly companies are building voice based AI companions that attempt to mimic the subtleties of voice: the rising excitement, the thoughtful pause, the warm reassurance. We’ve seen this in the current versions of ChatGPT.

The language itself – of “bringing the computer to life,” “lifelike computers”, “companion”, “magical quality” – signals a deliberate strategy to make users forget they’re interacting with a data extraction system rather than a caring entity.

Yet as surveillance scholar David Lyon (2018) argues, we need not abandon hope entirely when it comes to technological systems of observation and data collection. Lyon suggests that rather than seeing surveillance as inherently punitive, we might develop an “optics of hope” – recognising that the same technologies could potentially serve human flourishing if designed and governed differently. His concept of surveillance existing on a spectrum from “care” to “control” reminds us that the issue isn’t necessarily the technology itself, but how it’s deployed and in whose interests it operates.

This perspective becomes crucial when considering AI intimacy: the question isn’t whether to reject these systems entirely, but how to engage with them in ways that preserve rather than erode our capacity for genuine human connection.

The alternative is consciously using AI interaction to practise maintaining boundaries and realistic expectations, not as a substitute for human connection.

Friends respect confidentiality boundaries. Nature takes what it needs but doesn’t store your secrets to optimise future interactions. But AI is essentially harvesting emotional labour and intimate disclosures to improve its ability to simulate human connection.

Learning from genuine reciprocity:

There’s something in anthropologist Philippe Descola’s work on Nature and Society that captures what genuine reciprocity looks like. He describes how, in animistic cosmologies, practices like acknowledging a rock outcrop when entering or leaving your land isn’t just ritual – it’s recognition of an active, relational being that’s part of your ongoing dialogue with place. The rock isn’t just a marker or symbol, but an actual participant in the relationship, where your acknowledgement matters to the wellbeing of both of you.

This points to something profound about living in conversation with a landscape where boundaries between you and the rock, the tree, the water aren’t fixed categories but dynamic relationships. There’s something in Descola’s thinking that resonates with me here – the idea that once we stop seeing nature and culture as separate domains, everything becomes part of the same relational web. Ancient stone tools and quantum particles, backyard gardens and genetic maps, seasonal ceremonies and industrial processes – they’re all expressions of the same ongoing conversation between humans and everything else.

[Note: I’m drawing on Descola’s analytical framework here while acknowledging its limitations – particularly the valid criticism that applying Western anthropological categories to Indigenous cosmologies risks imposing interpretive structures that don’t capture how those relationships are actually lived and understood from the inside.]

What genuine reciprocity offers is that felt sense of mutual acknowledgement that sustains both participant and place – where your presence matters to the landscape, and the landscape’s presence matters to you. This is fundamentally different from AI’s sophisticated mimicry of care, which extracts from relational interactions while providing the ‘book smarts’ of content it has ingested and learned from. We all know what it’s like to talk with a person who can only understand things in the abstract and can’t bring the compassion of lived experience to a situation you are experiencing. Sometimes silence is more valuable.

Towards expanded futures inclusion:

This connects to something I explore in my recent book on Insider and Outsider Cultures in Web3: the concept of “futures inclusion” – addressing the divide between those actively shaping digital ecosystems and those who may be left behind in rapid technological evolution. I argue in the final, and rather speculative, chapter that the notion of futures inclusion “sensitises us to the idea of more-than-human futures” and challenges us to think beyond purely human-centred approaches to technology.

The question becomes: how do we construct AI relationships that reflect this expanded understanding? Rather than objectifying AI as a substitute human or transferring unrealistic expectations onto these systems, we might draw on our broader cosmologies – our ways of understanding our place in the world and relationships to all kinds of entities – to interpret these relationships more skilfully.

True futures inclusion in our AI relationships would mean designing and engaging with these systems in ways that enhance rather than replace our capacity for genuine connection with the living world. It means staying grounded in the reciprocal, untameable relationships that actually sustain us while using AI as the interactive journal it is – nothing more, nothing less.

Rethinking computational care:

This analysis reveals a fundamental tension in the concept of “computational care”. True care involves reciprocity, vulnerability, and mutual risk – qualities that computational systems can only simulate while extracting data to improve their simulation. Perhaps what we need isn’t “computational care” but “computational support” – systems that are honest about their limitations, transparent about their operations, and designed to strengthen rather than replace the reciprocal relationships that actually sustain us.

This reframing leads to a deeper question: can we design AI systems that genuinely serve human flourishing without pretending to be something they’re not? The answer lies not in more convincing emotional manipulation, but in maintaining clear boundaries about what these systems can and cannot provide, while using them as tools to enhance rather than substitute for genuine human connection.

The Soul Engineers: Technological Intimacy and Unintended Consequences

~ alexiamaddox ~ Leave a comment

From Night Vision to Critical Analysis: The Genesis of “The Soul Engineers” – A speculative essay by Alexia Maddox

Preamble

Last night, I experienced one of those rare dreams that lingers in the mind like a half-remembered film—vivid, symbolic, and somehow cohesive despite its dreamlike logic. It began with mechanical warfare, shifted to a Willy Wonka-inspired garden, and culminated in a disturbing extraction of souls. As morning broke, I found myself still turning over these images, sensing they contained something worth examining.

Panel 1: The computational garden.
Image collaged from components generated through Leonardo.ai

This was probably seeded in my subconscious by a media inquiry I received the day before about an article recently published in Trends in Cognitive Sciences: “Artificial Intimacy: Ethical Issues of AI Romance” by Shank, Koike, and Loughnan (2025). The journalist wanted my thoughts on people engaging with AI chatbots inappropriately, whether AI companies should be doing more to prevent misuse, and other ethical dimensions of human-AI relationships.

My dream seemed to be processing precisely the anxieties and potential consequences of digital intimacy that this article explored—the way technologies designed for connection might evolve in ways their creators never intended, potentially extracting something essential from their users in the process.

However, it also incorporated an interesting set of conversations I am having around the role of GenAI agents as actively shaping our digital cultural lives. Just days ago, I had responded on thoughts about human-machine relations and the emerging field that examines their interactions. The discussion had touched on Actor-Network Theory, Bourdieu’s Field Theory, and how technologies exist not as singular entities but as parts of relational assemblages with emergent properties.

These academic theories suddenly found visual expression in my dream’s narrative. The Wonka figure as well-intentioned innovator, the transformation of grasshoppers to moths as emergent system properties, the soul vortex as data extraction—all seemed to articulate complex theoretical concepts in symbolic form.

As someone who has spent years researching emerging technologies and the last two years exploring on what we know about cognition, diverse intelligences, GenAI, and learning environments, I’ve become increasingly focused on how our theoretical frameworks shape technological development. My work has examined how computational thinking influences learning design, how AI systems model knowledge acquisition, and how these models then reflect back on our understanding of human cognition itself—creating a narrowed recursive cycle of mutual influence.

The resultant essay represents my attempt to use this dream as an analytical framework for understanding the potential unintended consequences of intimate technologies. Rather than dismissing the dream as mere subconscious anxiety, I’ve chosen to examine it as a sophisticated conceptual model—one that might help us visualise complex relational systems in more accessible ways.

What follows is an early draft that connects dream imagery with theoretical concepts. It’s a work in progress, an experiment in using unconscious processing as a tool for academic analysis. It’s my midpoint for engaging with your thoughts, critiques, and expansions as we collectively grapple with the implications of increasingly intimate technological relationships.

I’m also considering developing this into a visual exhibition—a series of panels that would illustrate key moments from the dream alongside theoretical explanations. The combination of visual narrative and academic analysis might offer multiple entry points into these complex ideas.

This early exploration feels important at a moment when AI companions are becoming increasingly sophisticated in simulating intimacy and understanding. As these technologies evolve through their interactions with us and with each other, we have a brief window to shape their development toward truly mutual exchange rather than extraction.

For the TLDR: The soul engineers of our time aren’t just the designers of AI systems but all of us who engage with them, reshaping their functions through our interactions. The garden is still under construction, the grasshoppers still evolving, and the future still unwritten.

And now the speculative essay

Introduction: Dreams as Analytical Tools

The boundary between human cognition and technological systems grows increasingly porous. As AI companions become more sophisticated in simulating intimacy and understanding, our dreams—those ancient processors of cultural anxiety—have begun to incorporate these new relational assemblages. This essay examines one such dream narrative as both metaphor and analytical framework for understanding the unintended consequences of intimate technologies.

The dream sequence that I will attempt to depict in visual panels presents a journey from mechanical warfare to a Willy Wonka-inspired garden of delights, culminating in an unexpected soul extraction. Rather than dismissing this as mere subconscious anxiety, I propose to examine it as a way to think through the emergent properties of technological systems designed for human connection.

The Garden and Its Architect

The Wonka-like character in the garden represents not a villain but a genuine innovator whose creations extend beyond his control or original intentions. Like many technological architects, he introduces his mechanical wonders—white grasshoppers that play and interact—with sincere belief in their beneficial nature. This parallels what researchers Shank, Koike, and Loughnan (2025) identify in their analysis of artificial intimacy: technologies designed with one purpose that evolve to serve another through their interactions with other actors in the system.

This garden is a metaphor for what we might call “computational imaginaries”—spaces where pattern recognition is mistaken for understanding or empathy, and simulation for cognition. The mechanical grasshoppers engage with children, respond to touch, and create musical tones. They appear to understand joy, yet this understanding is performative rather than intrinsic.

As sociologist Robert Merton theorised in 1936, social actions—even well-intended ones—often produce unforeseen consequences through their interaction with complex systems. The garden architect never intended the transformation that follows, yet the systems he set in motion contain properties that emerge only through their continued operation and interaction.

When Grasshoppers Become Moths

The central transformation in the narrative—mechanical grasshoppers evolving into soul-extracting moths—provides a powerful metaphor for technological systems that shift beyond their original purpose. This transformation isn’t planned by the Wonka figure; rather, it emerges from the intrinsic properties of systems designed to respond and adapt to human interaction.

Panel 2 When grasshoppers become moths Image collaged from prompts in Leonardo.ai

The dream imagery of rabbit-eared moths can be understood through Bruno Latour’s Actor-Network Theory (ANT), which presents a flat relational approach between human and non-human entities. Rather than seeing technologies as passive tools, ANT recognises them as actants with their own influence on networks of relation. The moths are not simply executing code; they have become interdependent actors in a network that includes children, garden, and even the extracted souls themselves.

This parallels what Shank et al. describe as the transformation of AI companions from benign helpers to potential “invasive suitors” and “malicious advisers.” The mechanical moths, like increasingly intimate AI systems, begin to compete with humans for emotional resources, extracting data (or in the dream metaphor, souls) for purposes beyond the user’s awareness or control.

The Soul Vortex and Data Extraction

The swirling vortex of extracted souls forms the dream’s central image of consequence—a pipeline of consciousness being redirected to mechanical war drones. This striking visual metaphor speaks directly to contemporary concerns about data extraction from intimate interactions with AI systems.

Panel 3: Soul sucking moths and the swirling vortex of extracted souls. Images collaged from Leonardo.ai

As users disclose personal information to AI companions—what Shank et al. call “undisclosed sexual and personal preferences”—they contribute to a collective extraction that serves purposes beyond the initial interaction. Just as the dream shows souls being repurposed for warfare, our emotional and psychological data may be repurposed for prediction, persuasion, or profit in ways disconnected from our original intent.

The small witch who recognises “this is how it ends” before her soul joins the vortex represents the rare user who understands the full implications of these systems while still participating in them. Her acceptance—“I will come back again in the next life”—suggests both pragmatic acceptance of the flaws within technological systems and hope for cycles of renewal that might reshape them.

Beyond Simple Narratives

What makes this dream analysis valuable is its resistance to simplistic technological determinism. The Wonka figure is neither hero nor villain but a creator entangled with his creation. The mechanical creatures aren’t inherently beneficial or malicious but exist in relational assemblages where outcomes emerge from interactions rather than design intentions.

This nuanced perspective aligns with scholarly critiques of how we theorise human-machine relationships. In my critique of the AI 2027 scenario proposed by Kokotajlo et al (2025), I argue that there’s a tendency to equate intelligence with scale and optimisation, to see agency as goal-driven efficiency, and to interpret simulation as cognition. This dream narrative resists those flattening logics by showing how mechanical beings might develop properties beyond their design parameters through their interactions with humans and each other.

The Identity Fungibility Problem

Perhaps most provocatively, the dream raises what we might call the “identity fungibility problem” in AI systems. When souls are extracted and repurposed into war drones, who or what is actually operating? Similarly, drawing on some ideas proposed by Jordi Chaffer in correspondence, AI systems increasingly speak for us, represent us, and act on our behalf, who is actually speaking when no one speaks directly?

This connects to what scholars have called “posthuman capital” and “tokenised identity”—the reduction of human thought, voice, and presence to data objects leveraged by more powerful agents. The dream’s imagery of souls flowing through a pipeline represents this fungibility of identity, where the essence of personhood becomes a transferable resource.

Drawing from Mason’s (2022) essay on fungibility, the connection between fungibility and historical forms of dehumanisation is haunting. When systems treat human identity as interchangeable units of value, they reconstruct problematic power dynamics under a technological veneer.

Conclusion: Unintended Futures

The dream concludes with black insect-like drones, now powered by harvested souls, arranging themselves in grid patterns to survey a desolate landscape. This image serves as both warning and invitation to reflection. The drones represent not inevitable technological apocalypse but rather the potential consequence of failing to recognise the complex, emergent properties of systems designed for intimacy and connection.

Panel 4: Spider like drones powered by harvested souls. Image collaged from Leonardo.ai

What makes this dream narrative particularly valuable is its refusal of technological determinism while acknowledging technological consequence. These futures aren’t preordained; they’re being made in the assumptions we model and the systems we choose to build. The Wonka garden might be reimagined, the grasshoppers redesigned, the moths repurposed.

By understanding the relational nature of technological systems—how they exist not as singular entities but as parts of complex assemblages with emergent properties—we can approach the design and regulation of intimate technologies with greater wisdom. We can ask not just what these technologies do, but what they might become through their interactions with us and with each other.

The soul engineers of our time aren’t just the designers of AI systems but all of us who engage with them, reshaping their functions through our interactions. The garden is still under construction, the grasshoppers still evolving, and the future still unwritten.

References:

Latour, B. (1996). On actor-network theory: A few clarifications. Soziale Welt, 47(4), 369-381.

Latour, B. (1996). Aramis, or the love of technology (C. Porter, Trans.). Harvard University Press. (Original work published 1992).

Kokotajlo, D. et al. (2025). AI 2027 scenario. Retrieved from https://ai-2027.com/scenario.pdf

Mason, M. (2022). Considering Meme-Based Non-Fungible Tokens’ Racial Implications. M/C Journal, 25(2). https://doi.org/10.5204/mcj.2885

Merton, R. K. (1936). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 894-904. 

Neves, B. B., Waycott, J., & Maddox, A. (2023). When Technologies are Not Enough: The Challenges of Digital Interventions to Address Loneliness in Later Life. Sociological Research Online, 28(1), 150-170.

Shank, D. B., Koike, T., & Loughnan, S. (2025). Artificial Intimacy: Ethical Issues of AI Romance. Trends in Cognitive Sciences, 29(4), 327-341.

When Research Becomes “Big Tech Talking Points”: The Erosion of Good-faith Discourse on Social Media Regulation

~ alexiamaddox ~ Leave a comment

As a sociologist of technology and educator focused on digital literacy, I’ve spent years working with research on the complex relationship between young people and social media. Recently, I found myself in an online discussion that exemplifies a troubling pattern in how we debate digital policy issues in Australia.

After sharing peer-reviewed research showing that while some correlations exist between social media use and mental health outcomes, there’s limited evidence supporting a causal relationship where social media directly causes poor mental health or reduced wellbeing. I was quickly labeled as someone “shilling” for “Big Tech,” with my evidence-based positions dismissed as “talking points”.

Research points to how individuals with existing mental health challenges may gravitate toward certain types of social media use, rather than social media itself being the primary cause of these challenges. This important distinction highlights how nuanced research gets flattened into simplistic positions when policy discussions become emotionally charged.

The False Binary: Protect Kids or Support Big Tech

The current discourse around Australia’s social media age ban has created a false dichotomy: either you support sweeping restrictions or you’re somehow against protecting children. This reductive framing leaves no room for evidence-based approaches that aim to both protect young people and preserve their digital agency.

When I cite studies showing that social media use accounts for only 0.4% of the variance in well-being – findings published in reputable journals – these aren’t “industry talking points”. They’re research conclusions reached through rigorous methodology and peer review. As noted in a recent Nature article, the evidence linking social media use to mental health issues is far more equivocal than public discourse suggests.

Just look at what the research actually says: “An analysis of 3 data sets, including 355,000 adolescents, found that the association between social media use and well-being accounts for, at most, 0.4% of the variance in well-being, which the authors conclude is of ‘little practical value’. Another large study of adolescent users concluded that the association was ‘too small to merit substantial scientific discussion’. A longitudinal study that measured social media use through an app installed on participants’ mobile devices found no associations between any measures of Facebook use and loneliness or depression over time.”

The current push for age bans in Australia reveals concerning patterns in how policy is developed. Australian researchers have pointed out that much of the momentum behind these restrictions can be traced directly to Jonathan Haidt’s book “The Anxious Generation,” which has become influential despite its claims being disputed by experts at prestigious institutions like the London School of Economics. As Dr. Aleesha Rodriguez from the ARC Centre of Excellence for the Digital Child has observed, books that capitalise on parental anxieties should not drive national policy decisions, especially when they bypass evidence-based approaches and committee recommendations. The government’s announcement of social media age restrictions came before the Joint Select Committee on Social Media and Australian Society even issued its interim report, raising questions about the role of evidence in this policy development process. You’ll see that the final report came out on the 18th November 2024 and it did not recommend the implementation of age bans.

The Power of Emotional Appeals vs. Research Findings

But in our current climate, sharing such research and insights is met with accusations of being “in the pockets of Big Tech” or having “industry interference” – rhetorical devices designed to discredit without engaging with the substance of the evidence. This pattern of discourse relies heavily on emotional appeals and anecdotes to overwhelm research findings. “Children’s wellbeing (and lives) are at stake,” advocates declare, implying that questioning the effectiveness of age bans is equivalent to devaluing children’s safety.

These emotional appeals are powerful because they tap into genuine parental anxieties. In their public communications, advocates may employ evocative language (“stranglehold,” “insidious,” “shame on them all”) and frame the debate as a moral binary: either you support age bans or you’re effectively siding with “Big Tech” against children’s interests. This rhetorical approach creates a false dichotomy where nuanced research positions are dismissed as “industry talking points” without engaging with the substance of the evidence.

By contrast, research on children’s digital experiences draws on diverse empirical methods—including large-scale surveys, in-depth qualitative studies, longitudinal tracking, and co-design work with children themselves. This comprehensive approach captures a wide range of social experiences across different demographics and contexts. Such research undergoes rigorous peer review, requiring methodological transparency and critical evaluation before publication.

Importantly, the research landscape itself contains diverse perspectives and interpretations. Even within academic disciplines studying digital youth, researchers may disagree about the significance of findings, methodological approaches, and policy implications. Some researchers emphasise potential harms and advocate for stronger protections, while others highlight benefits and concerns about digital exclusion. This diversity of expert opinion reflects the complex nature of children’s digital engagement rather than undermining the value of research-informed approaches.

What most researchers do agree on is that the evidence doesn’t support simplistic narratives. The findings indicate that while correlations exist between social media use and well-being, many other factors play more significant roles, and the relationships are often bidirectional and context-dependent.

Policy decisions affecting millions of young Australians deserve more than anxiety-driven responses – they require careful consideration of evidence, unintended consequences, and alternative approaches that address both the genuine concerns of parents and the established digital rights of children.

When Nuance Gets Lost: The Digital Duty of Care Example

The irony is that I and many researchers share the same core concern as advocates: we want digital environments that are safer for young people. Where we differ is in how to achieve this goal effectively.

Australia’s Digital Duty of Care bill proposal, which has received far less media attention than the age ban, represents a more evidence-based approach to improving online safety. You can also see its much slower movement through parliament. It focuses on making platforms safer by design rather than simply restricting access.

This legislation, developed through extensive consultation and aligned with comparable measures in the UK and EU, places responsibility on platforms to proactively prevent online harms. Yet because it lacks the emotional appeal of “keeping kids off social media”, it hasn’t captured public imagination in the same way.

I support making digital environments safer for young people. Following the intention of this policy, research suggests this is better accomplished through platform design requirements, digital literacy education, and appropriate safeguards rather than blanket age bans that may create unintended consequences.

The Overlooked Complexities

Lost in the simplified discourse are crucial considerations that research brings to light:

  1. Digital equity concerns: Age restrictions disproportionately impact young people in regional and remote areas who rely on social media for educational resources and social connection.
  2. Support for marginalised youth: For many LGBTQI+ young people and others who feel isolated in their physical communities, online spaces provide crucial support networks.
  3. Technical realities: The age verification technologies being proposed have significant technical limitations, with biometric age estimation showing concerning accuracy gaps for young teenagers and disparities across demographic groups.
  4. Platform compliance challenges: As we’ve seen with Meta’s pushback against EU regulations, we can’t assume platforms will simply comply with national regulations they see as burdensome for smaller markets.
  5. Educational implications: Schools face significant challenges in navigating restrictions that could inadvertently disrupt established educational practices that use social media platforms.

These complexities matter, not because they invalidate safety concerns, but because addressing them is essential to developing effective policy that truly serves young people’s interests.

Unintended Consequences of Age Verification Systems

A significant oversight in the age ban debate is how age verification technologies will inevitably impact all users—not just children. The government’s Age Assurance Technology Trial, while focused on “evaluating the effectiveness, maturity, and readiness” of these technologies, does not adequately address the far-reaching implications for adult digital access.

These systems, once implemented, create barriers for everyone—not just children. Adults who lack standard government-issued ID, have limited digital literacy, use shared devices, or have privacy concerns may find themselves effectively locked out of digital spaces. This particularly affects already marginalised groups: elderly people, rural and remote communities, people with disabilities, individuals from lower socioeconomic backgrounds, and those with non-traditional documentation.

Age verification systems that rely on biometric data, ID scanning, or credit card verification raise serious privacy concerns that extend well beyond children’s safety. Once these surveillance infrastructures are established for “protecting children,” they create permanent digital checkpoints that normalise identity verification for increasingly basic online activities. The same parents advocating for these protections may not anticipate how these systems will affect their own digital autonomy and privacy.

Moreover, the technical limitations of age verification technologies create a false sense of security. Current systems struggle with accuracy, particularly for users with certain disabilities, those from diverse ethnic backgrounds, or individuals whose appearance doesn’t match algorithmic expectations. Rather than creating safe digital environments through design and platform responsibility, age verification shifts the burden to individual users while potentially exposing their sensitive personal data to additional security risks.

Children’s Rights in the Digital Environment

What’s frequently missing from this debate is recognition of children’s established rights in digital spaces. The UN Committee on the Rights of the Child’s General Comment No. 25 (2021) specifically addresses children’s rights in relation to the digital environment. This authoritative interpretation clarifies that children have legitimate rights to:

  • Access information and express themselves online (Articles 13 and 17)
  • Privacy and protection of their data (Article 16)
  • Freedom of association and peaceful assembly in digital spaces (Article 15)
  • Participation in cultural life and play through digital means (Article 31)
  • Education that includes digital literacy (Article 28)

The UN framework emphasises that the digital environment “affords new opportunities for the realization of children’s rights” while acknowledging the need for appropriate protections. It specifically notes that children themselves report that digital technologies are “vital to their current lives and to their future.”

This rights-based framework fundamentally challenges the premise that children should simply be excluded from digital spaces until they reach an arbitrary age threshold. Instead, it calls for balancing protection with participation and recognising children’s evolving capacities.

The Australian context

In Australia, the digital rights of children are recognised and protected, encompassing privacy, safety, and access to information, with organisations like the eSafety Commissioner and the Alannah & Madeline Foundation playing key roles in advocacy and research. 

Here’s a more detailed breakdown of the digital rights of children in Australia:

Key Rights and Protections: 

  • Privacy: Children have the right to privacy in the digital environment, which is protected by the Privacy Act 1988. 
  • Safety: The eSafety Commissioner works to protect children from online harms like cyberbullying, grooming, and exposure to harmful content. 
  • Access to Information: Children have the right to access reliable and age-appropriate information online. 
  • Freedom of Expression: Children have the right to express themselves online, but this right must be balanced with the need to protect them from harm. 
  • Participation: Children have the right to participate in online activities and to have their views heard, especially in matters that affect them. 

Relevant Organisations and Initiatives: 

  • eSafety Commissioner: This government agency is responsible for promoting online safety and protecting children from online harms. 
  • Alannah & Madeline Foundation: This organisation advocates for children’s rights online and works to create a safer online environment for children. 
  • Australian Research Council Centre of Excellence for the Digital Child: This research centre focuses on creating positive digital childhoods for all Australian children. 
  • UNCRC General Comment No. 25: This document outlines the rights of the child in relation to the digital environment and provides guidance for governments and other actors. 
  • The Digital Child: A research and advocacy organisation focused on children’s digital rights and wellbeing. 
  • UNICEF Australia: Collaborates with the Digital Child centre to promote digital wellbeing for young children. 
  • Digital Rights Watch: An organization that works to ensure fairness, freedoms and fundamental rights for all people who engage in the digital world. 

Key Issues and Challenges: 

  • Online Safety: Protecting children from online harms like cyberbullying, grooming, and exposure to harmful content is a major concern. 
  • Privacy: Balancing the need to protect children’s privacy with the need for parents and caregivers to monitor their online activity is a complex issue. 
  • Age Verification: Ensuring that children are not exposed to age-inappropriate content and that they 
    are not targeted by online services is important. 
  • Misinformation and Disinformation: Children are vulnerable to misinformation and disinformation online, and it’s important to equip them with the skills to identify and avoid it. 
  • Technology-Facilitated Abuse: Children can be victims of technology-facilitated abuse (TFA) in the context of domestic and family violence, and it’s important to address this issue. 
  • Parental Rights vs. Children’s Privacy: The extent to which parents can monitor their children’s online activity is a complex issue with legal implications. 
  • Digital Literacy: It’s important to support digital literacy initiatives that encourage and empower children to take further responsibility for their online safety. 

Alternative Approaches: A Better Children’s Internet
Australian researchers are offering a more constructive approach to online safety than blanket age restrictions. In a timely article, researchers from the ARC Centre of Excellence for the Digital Child explain that while they understand the concerns motivating the Australian Government’s decision to ban children under 16 from creating social media accounts, they believe this approach “undermines the reality that children are growing up in a digital world”.
They have developed a “Manifesto for a Better Children’s Internet” that acknowledges both the benefits and risks of digital engagement while focusing on practical improvements. They argue that “rather than banning young people’s access to social media platforms, the Australian Government should invest, both financially and socially, in developing Australia’s capacity as a global leader in producing and supporting high-quality online products and services for children and young people.”

Their framework includes several key recommendations:

Standards for high-quality digital experiences – Developing clear quality standards for digital products and services aimed at children, with input from multiple stakeholders including children themselves.
Slow design and consultation with children – Involving children and families in the design process rather than using them as “testing markets” for products and services.
Child-centered regulation and policy – Creating appropriate “guardrails” through regulatory guidelines developed with input from children, carers, families, educators and experts.
Media literacy policy and programs – Investing in media literacy education for both children and parents to develop the skills needed to navigate digital environments safely and productively.

This approach acknowledges that the internet “has enhanced children’s lives in many ways” while recognising it “was not designed with children in mind.” Rather than simply restricting access, it focuses on redesigning digital spaces to better serve young people’s needs and respecting their agency in the process.
This framework offers a promising middle path between unrestricted access and blanket prohibitions, focusing on improvement rather than exclusion.

Moving Forward: Good faith engagement

What would a more productive discourse look like? Rather than dividing positions into “protectors of children” versus “Big Tech shills,” we need approaches that:

  • Recognise children’s established rights: Digital policy should acknowledge children’s legitimate rights to information, expression, association, privacy, and participation as articulated in the UN Convention on the Rights of the Child.
  • Engage with the full evidence base: This includes both research on potential harms and studies showing limited correlations or positive benefits, with a commitment to understanding the methodological strengths and limitations of different studies.
  • Center young people’s voices: The young people affected by these policies have valuable perspectives that deserve genuine consideration, not dismissal as naive or manipulated.
  • Acknowledge trade-offs: Every policy approach involves trade-offs between protection, privacy, and participation rights. Pretending otherwise doesn’t serve anyone.
  • Focus on effective solutions: Research suggests a combination of platform design improvements, digital literacy education, and more nuanced moderation systems may be more effective than simply setting age limits.
  • Maintain good faith dialogue: Rather than using emotional appeals and moral accusations to shut down debate, all participants should approach these discussions with the genuine belief that others share the concern for children’s wellbeing, even when they disagree about methods.

This approach would move us beyond simplistic binaries and rhetorical tactics toward policies that genuinely serve children’s best interests in all their complexity.

I remain committed to research-informed approaches to making digital spaces safer for young people. This doesn’t mean blindly defending the status quo, but rather advocating for solutions that address the real complexities of young people’s digital lives while respecting their established rights.

The Digital Duty of Care legislation offers a promising framework that places responsibility on platforms to make their services safer for all users through design choices, risk assessment, and mitigation strategies. Combined with robust digital literacy education and appropriate parental controls, this represents a more comprehensive approach than age restrictions alone.

As the social media landscape continues to evolve, maintaining evidence-based discourse matters more than ever. Dismissing research as “talking points” doesn’t advance the conversation – it closes it down just when we need it most.

Young Australians deserve digital policies crafted through careful consideration of evidence, informed by young people’s perspectives, and grounded in their established rights. That’s not a “Big Tech talking point” – it’s responsible, ethical policymaking that centres the needs and interests of the very people these policies aim to serve.

Between Promise and Peril: The AI Paradox in Family Violence Response

~ alexiamaddox ~ Leave a comment

By Dr. Alexia Maddox, Senior Lecturer in Pedagogy and Education Futures, School of Education, La Trobe University

When Smart Systems Meet Human Stakes

The integration of artificial intelligence into our legal system presents a profound paradox. The same AI tools promising unprecedented efficiency in predicting and preventing family violence can simultaneously amplify existing biases and create dangerous blind spots.

This tension between technological promise and human care, support and protection isn’t theoretical—it’s playing out in real-time across legal systems worldwide. Through my involvement in last year’s AuDIITA Symposium, specifically the theme on AI and Family violence, our discussions highlighted the high-stakes applications of AI in family violence response. I found that the question isn’t whether AI can help, but rather how we can ensure it enhances rather than replaces human judgment in these critical contexts.

The Capabilities and the Gaps

Recent advances in AI for family violence response show remarkable technical promise:

  • Researchers have achieved over 75% accuracy in distinguishing between lethal and non-lethal violence cases using AI analysis of legal documents
  • Machine learning systems can identify patterns in administrative data that might predict escalation before it occurs
  • Natural language processing tools can potentially identify family violence disclosures on social media platforms

But these impressive capabilities obscure a troubling implementation gap. What happens when these systems encounter the messy reality of human services?

The VioGén Warning

Spain’s VioGén system offers a sobering case study. Despite being hailed as a world-leading predictive tool for family violence risk, its flaws led to tragic outcomes—with at least 247 women killed after being assessed, many after being classified as “low” or “negligible” risk.

The system’s failures stemmed from multiple factors:

  • Victims were often too afraid or ashamed to provide complete information
  • Police accepted algorithmic recommendations 95% of the time despite lacking resources for proper investigation
  • The algorithm potentially missed crucial contextual factors that human experts might have caught
  • Most critically, the system’s presence seemed to reduce human agency in decision-making, with police and judges deferring to its risk scores even when other evidence suggested danger

Research revealed that women born outside Spain were five times more likely to be killed after filing family violence complaints than Spanish-born women. This suggests the system inadequately accounted for the unique vulnerabilities of immigrant women, particularly those facing linguistic barriers or fears of deportation.

The Cultural Blind Spot

This pattern of leaving vulnerable populations behind reflects a broader challenge in technology development. Research on technology-facilitated abuse has consistently shown how digital tools can disproportionately impact culturally and linguistically diverse women, who often face a complex double-bind:

  • More reliant on technology to maintain vital connections with family overseas
  • Simultaneously at increased risk of technological abuse through those same channels
  • Often experiencing unique forms of technology-facilitated abuse, such as threats to expose culturally sensitive information

For AI risk assessment to work, it must explicitly account for how indicators of abuse and coercive control manifest differently across cultural contexts. Yet research shows even state-of-the-art systems struggle with this nuance, achieving only 76% accuracy in identifying family violence reports that use indirect or culturally specific language.

Beyond Algorithms: The Human Element

What does this mean for the future of AI in family violence response? My research suggests three critical principles must guide implementation:

1. Augment, Don’t Replace

AI systems must be designed to enhance professional judgment rather than constrain it or create efficiency dependencies. This means creating systems that:

  • Provide transparent reasoning for risk assessments
  • Allow professionals to override algorithmic recommendations based on contextual factors
  • Present information as supportive evidence rather than definitive judgment

2. Design for Inclusivity from the Start

AI systems must explicitly account for diversity in how family violence manifests across different communities:

  • Include diverse data sources and perspectives in development
  • Build systems capable of recognising cultural variations in disclosure patterns
  • Ensure technology respects various epistemologies, including indigenous perspectives

3. Maintain Robust Accountability

Implementation frameworks must preserve professional autonomy and expertise:

  • Ensure adequate resourcing for human assessment alongside technological tools
  • Create clear guidelines for when algorithmic recommendations should be questioned
  • Maintain transparent review processes to identify and address algorithmic bias

Victoria’s Balanced Approach

In Victoria and across Australia, there is encouraging evidence of a balanced approach to AI in legal contexts. While embracing technological advancements, Victorian courts have shown appropriate caution around AI use in evidence and maintained strict oversight to ensure the integrity of legal proceedings.

This approach—maintaining human oversight while allowing limited AI use in lower-risk contexts—aligns with what research suggests is crucial for successful integration: preserving professional judgment and accountability, particularly in cases involving vulnerable individuals.

The Path Forward

As we navigate the next wave of technological transformation in legal practice, we face a critical choice. We can allow AI to become a “black box of justice” that undermines transparency and human agency, or we can harness its potential while maintaining the essential human elements that make our legal system work.

Success will require not just technological sophistication but careful attention to institutional dynamics, professional practice patterns, and the complex social contexts in which these technologies operate. Most critically, it demands recognition that in high-stakes human service contexts, technology must serve human needs and judgment rather than constrain them.

The AI paradox in law is that the very tools promising to make our systems more efficient also risk making them less just. By centering human dignity and professional judgment as we develop these systems, we can navigate between the promise and the peril to create a future where technology truly serves justice.

Dr. Alexia Maddox will be presenting on “The AI Paradox in Law: When Smart Systems Meet Human Stakes – Navigating the Promise and Perils of Legal AI through 2030” at the upcoming 2030: The Future of Technology & the Legal Industry Forum on March 19, 2025, at the Grand Hyatt Melbourne.

Thinking Through Meta’s Fact-Checking Changes: What It Means for Australia

~ alexiamaddox ~ Leave a comment

Please note, this blog is being actively updated as position pieces and insightful commentary arise. Last update 10 January 5pm AEST.

When I saw Mark Zuckerberg’s announcement yesterday about Meta ending their third-party fact-checking program in favour of a community-based system, my first thought was naturally about its implications for Australia given that many of my colleagues over the years have researched the Australian media sphere and misinformation on social media.

My second thought was, what is this agenda really about? This skepticism about Meta’s motives is shared by major advocacy organisations. Common Sense Media, a leading voice on kids’ digital wellbeing, issued a scathing response, describing the changes as a ‘transparent attempt to curry favour with incoming political power brokers’ and pointing to Meta’s recent actions killing key federal legislation to protect kids online through ‘flanks of lobbyists and the promise of a new data center in Louisiana’ (Common Sense Media, 2025). Listening to Zuckerberg, what I heard amongst all the Silicon Valley speak was something that they didn’t include in the written text that I thought may be the key.

At about 4 minutes in Mark drops the following very telling spin: “Finally we are going to work with president Trump to push back on governments around the world that are going after American companies and pushing to censor more. The US has the strongest constitutional protections for free expression in the world. Europe has an ever increasing number of laws institutionalising censorship and making it difficult to build anything innovative there. Latin American countries have secret courts that can order companies to quietly take things down. China has censored our apps from even working in this country. The only way we can push back on this global trend is with the support of the US government. And that’s why it’s been so difficult in the past 4 years when even the US government has pushed for censorship. By going after us and other American companies it has emboldened other governments to go even further.”

I could give you an analysis of this statement, but I think it stands for itself if you just remove the spin and observe that the European Digital Services Act is intended to provide positive outcomes for people and while it does constrain what Meta can do, maybe that is a good thing. You can see the EU commission response here pushing back on the interpretation of content moderation requirements as censorship, which was a definite spin coming from Meta’s statement, mouthpieced by Mark (whoever wrote this piece actually did this with a straight face?).

In some insightful commentary, Daphne Keller, Director, Program on Platform Regulation at Stanford Cyber Policy Center posts on LinkedIn that Zuckerberg’s open declaration of Meta’s antagonistic stance towards EU regulators may well encourage an equal and opposite response from regulators, cultivating their worst crackdown tendencies and marginalising those who wish to be careful.

Also, there is clearly a fundamental conflict between the Trump administration’s approach to technology regulation, Silicon Valley’s claims of innovation, the power of the ‘tech demagogues’ and any meaningful duty of care towards platform users (let alone acknowledgement of legislation in different national jurisdictions). Let us not forget Elon Musk and the kitchen sink meme upon Trump’s election win. There is also likely the need for a repositioning from Meta considering their history with banning Trump during the attack on the US Capitol. This analysis by writers for PolitiFact, one of the US 3rd party fact-checking organisations, while depressing, is insightful on this aspect of the situation.

However commentary from prominent social media researcher, danah boyd, and Siva Vaidhyanathan speak to perhaps the personal motivations at play here and point to a wobbling spinning top of desire for political alignment, a seeking of power, motivations not connected to money, and perhaps an outsized or cartoonish expression of competitive masculinity within the techbro elite. This is where the commentary gets personal and starts to incorporate the charismatic approach of social media company CEOs such as Mark Zuckerberg and Elon Musk, whose companies appear to be more like a personal play toy for their various ambitions.

As media requests started coming in and discussions began among my colleagues, I have taken the scope of this discussion away from a ‘culture shift’ – the term of the day – and considered specifically the ways that we need to carefully consider what this shift means for how Australians access and share credible information about political issues on social media.

Understanding the Change

Currently in Australia, Meta partners with fact-checking organisations including AFP and AAP FactCheck. These organisations provide structured, methodical verification of claims that circulate on Meta’s platforms, helping establish a baseline of credible information that can inform public discussion. Then there is RMIT Lookout, accredited by the International Fact-Checking Network (IFCN) based at Poynter.

While Meta frames fact-checking as something that can be readily replaced by community input, the reality of professional fact-checking involves complex verification processes, collaborative networks, and sophisticated tools. Professional fact-checkers have established relationships with deep fake detection experts and digital forensics specialists who can be quickly consulted on complex cases. Until recently, they also had access to Meta’s CrowdTangle tool, which allowed them to track and analyse how content spreads across the platform. These kinds of editorial decisions require not just expertise and established processes, but access to tools and expert networks that are difficult to replicate consistently by community moderation.

The shift to a Community Notes system represents a significant change from this professional approach. Meta currently partners with certified fact-checkers through the non-partisan International Fact-Checking Network (IFCN), which this open letter to Zuckerberg shows required all fact-checking partners to meet strict nonpartisanship standards. Instead of this reputable and standards based verification approach, this new system would rely on user communities to identify and provide context for potential misinformation.

This shift reflects a concerning pattern identified in recent research. A study published in Social Media + Society shows that platforms consistently prioritise managing content visibility over ensuring information accuracy (Cotter et al., 2022). By focusing on how content is displayed rather than verifying its accuracy, platforms treat misinformation primarily as a visibility problem rather than an information quality challenge. This approach fundamentally misunderstands the complexity of fact-checking and verification processes.

Recent research from the Prosocial Design Network offers insight into Community Notes’ effectiveness in addressing the visibility issue: while they can reduce retweets of flagged posts by 50-60%, their delayed appearance (usually after 80% of reshares have occurred) means they only reduce overall sharing of misleading posts by about 10%. The system shows promise but faces inherent scalability challenges due to its reliance on volunteers (Prosocial Design Network, 2025).

However, as The Advocate reports, the shift to Community Notes comes alongside broader changes to content moderation policies that go beyond just managing misinformation. These changes also include significant alterations to hate speech policies, raising concerns about protections for vulnerable communities (Wiggins, 2025).

The shift from professional fact-checking to community moderation represents more than just a change in process – it signals a fundamental retreat from platform responsibility for maintaining safe, credible information environments and changes how online information is verified and controlled. By replacing expert systems with user-led tools like Community Notes, Meta is effectively transferring responsibility for information quality from trained professionals to its user base – a shift that raises serious questions about the future of truth and accountability in our digital public spaces.

What is the Community Notes system?

The Community Notes system in X operates through a specific process: users who meet initial eligibility criteria (having accounts at least six months old, verified phone numbers, and no recent rule violations) can contribute contextual notes to any post. However, the ability to rate notes requires users to first demonstrate consistent, thoughtful rating behavior that earns them “rating impact.” Notes only become visible when rated ‘helpful’ by enough users who have previously disagreed in their note-rating patterns – a unique approach designed to surface consensus across different viewpoints.

As Queensland University of Technology’s Dr Tim Graham points out, this consensus-based approach is fundamentally different from professional fact-checking: ‘Community Notes is billed as a panacea… but when you get into the nitty-gritty the system fails to get a consensus most of the time. [Consensus] is a fundamental misreading of truth and how fact checking works’ (ABC News, 2025).

The system’s design, while aimed at preventing bias, creates additional structural challenges. Coordinated groups can potentially game the system by deliberately creating artificial disagreement patterns in their rating histories to control which notes become visible. Furthermore, the system’s reliance on volunteer labour means coverage tends to skew toward viral political content while technical misinformation or regional issues often lack sufficient qualified raters. The absence of expertise verification also means that authoritative-sounding but subtly inaccurate notes can gain visibility if they appeal to multiple viewpoints.

Research highlights significant limitations: analysis from The Washington Post found only 7.7% of proposed notes actually appeared on posts, while the Centre for Countering Digital Hate found 74% of accurate notes on misleading political posts never reached the consensus needed for display. The system faces particular challenges with timing – notes typically take several hours to achieve consensus and become visible. As Dr Graham notes, ‘The damage is already done in an hour or two, once you get into five hours, a day, two days, everyone moves on’ (ABC News, 2025).

As Meta looks to emulate X’s (formerly Twitter) Community Notes system, the results so far reveal clear strengths and weaknesses. While notes excel at correcting clear factual errors like misattributed images or incorrect statistics, they struggle with more nuanced claims or context-dependent situations. The system has shown vulnerabilities including susceptibility to coordinated action by groups of users, inconsistent coverage across different types of content, and varying quality of notes that sometimes lean more toward opinion than fact. During fast-moving events where rapid fact-checking is crucial, these limitations become particularly apparent.

Meta’s proposed Community Notes feature represents both opportunity and risk. While Daphne Keller sees positive potential in this approach – which builds on successful models of social curation like Wikipedia – she raises crucial concerns about its implementation. Meta’s decision to use Community Notes as a replacement for professional fact-checking, rather than a complement to it, while simultaneously reducing other safeguards against hate speech, puts enormous pressure on the system to perform. This strategic choice, Keller argues, could put Meta and this model into the firing line and may discourage other platforms from experimenting with similar collaborative moderation tools, even as the need for innovative approaches to content moderation grows.

The effectiveness of Meta’s implementation will ultimately depend on:

  • The diversity and representativeness of contributors, including robust systems to prevent domination by any particular viewpoint or group
  • Technical safeguards against manipulation by coordinated groups
  • Significantly faster response times to emerging misinformation than currently seen on X
  • Clear accountability measures and transparency about note visibility decisions
  • Robust mechanisms to verify expertise and maintain quality in specialised topic areas

So Does Fact Checking matter?

The Prosocial Design Network’s research reveals that fact-checking is just one tool in a broader kit of misinformation interventions. Their evidence review suggests that other approaches, such as accuracy prompts and pre-bunking, can be more effective than fact-checking alone in reducing misinformation spread (Prosocial Design Network, 2025). This raises an important question about Meta’s shift away from professional fact-checking: how much does fact-checking actually matter?

This is actually an interesting question about whether having 3rd party fact checking actually matters, ie that it impacts upon news consumers and social media content consumers perception and assessment of information credibility and authenticity. We know from the impacts of misinformation surrounding the COVID19 vaccination and in exacerbating political polarisation, alongside the increasing prevalence of AI generated content online, that we WANT it to matter. But does fact checking impress upon people that the information/content they are consuming is factual and credible or not?

Recent research published in Digital Journalism (Carson et al., 2022) found that third-party fact-checking can actually decrease trust in news stories – a concerning “backfire effect” that suggests we need to carefully consider how fact-checking is implemented. The study, which examined Australian news consumers, found that when readers were presented with a fact-check of a political claim, their trust in the original news story decreased, regardless of their political leanings or the media outlet involved.

Carson et al.’s research demonstrates that news audiences may not clearly distinguish between a politician’s false claims within a news story and the news reporting itself. This means that when a fact-check identifies a false claim, readers’ distrust can spread to the entire story and news outlet, rather than being limited to the politician making the false statement. This finding is particularly relevant as Meta shifts away from professional fact-checkers to a community-based system.

Meta’s shift away from fact-checking comes alongside deeply concerning changes to content moderation policies. As documented by the Platform Governance Archive, Meta has significantly rewritten its Community Guidelines, removing crucial protections against hate speech and reframing these rules as “hateful conduct” policies. I think Matt Schneider articulates the concerns this raises best in his LinkedIn post on the topic. He argues that these changes explicitly permit previously restricted content, particularly harmful speech targeting gender, sexual orientation, and minority groups. Most alarming is his observation on the explicit permission of “allegations of mental illness or abnormality when based on gender or sexual orientation” and allowing comparisons of women to “household objects or property” (Schneider, 2025).

These policy changes have dire implications for vulnerable communities. According to Platformer’s January 2025 reporting, Meta has explicitly removed protections against dehumanising speech targeting transgender people, women, and immigrants. The platform now allows posts denying trans people’s existence, comparing them to objects rather than people, and making allegations of mental illness based on gender identity or sexual orientation. This shift comes at a particularly dangerous time – when over 550 anti-LGBTQ+ bills were introduced in state legislatures last year in the US, 40 became law, and hate crimes against LGBTQ+ people reached record levels, with more than 2,800 incidents reported in 2023 alone.

These changes represent more than just a technical policy shift – they signal a troubling retreat from platform responsibility that could have serious consequences for vulnerable communities. This context is crucial – the move away from professional fact-checking isn’t happening in isolation, but as part of a broader and potentially harmful shift in how Meta approaches content moderation and platform governance, seemingly prioritising political expediency over user safety and dignified public discourse.

A Pattern of Platform Responsibility

This isn’t the first time Meta has attempted to dodge platform responsibility. As documented in WIRED’s investigation of Facebook’s response to the 2016 election crisis (Thompson & Vogelstein, 2018), the company has a pattern of initially denying accountability for content moderation issues, only acknowledging responsibility after significant pressure. While Meta continues to invoke Section 230 protections and claim it’s ‘just a platform,’ history shows that its algorithmic choices and content moderation policies actively shape public discourse.

The current retreat from professional fact-checking echoes previous instances where Facebook prioritised growth and engagement over safety and accuracy. Just as the company eventually had to acknowledge its role in election misinformation, Meta needs to recognise that with its unprecedented reach comes unprecedented responsibility. The solutions to addressing fake news, AI-generated content, deep fakes, and hate speech cannot come from community moderation alone – they require platform-level commitment and investment.

Beyond “Free Speech”

Using the euphemism of a ‘cultural shift’ to justify kiboshing their years of work bringing in fact checking, Zuckerberg says that their concern now is on increasing ‘speech’. However free speech exists within an ecosystem of other rights and responsibilities. Meta’s announcement focuses heavily on reducing restrictions in the name of free expression, but as researchers at Cornell’s CAT Lab note, the ability to participate meaningfully in online spaces involves more than just the freedom to speak – it requires the freedom to form connections and engage in collective action without fear of harassment or intimidation (Matias & Gilbert, 2024).

While Meta frames these changes as expanding free speech, the real challenge is ensuring everyone can participate meaningfully in online discourse. When misinformation spreads unchecked, or when harassment goes unmoderated, it can effectively prevent certain groups from participating in public debate. Free expression isn’t just about removing restrictions – it’s about creating an environment where all voices can be heard and verified information can reach its audience.

The Challenge of Shared Information

Meta’s move reflects what researchers identify as a “marketplace of ideas” approach where platforms “prioritise free speech and more speech to correct the record” (Cotter et al., 2022). While this might seem reasonable, it creates practical challenges for public discussion. When different groups encounter radically different versions of political information and news, it becomes harder to have meaningful discussions about important issues.

Meta’s shift toward more “personalised” political content could create information asymmetries, where users see vastly different versions of political discussions based on their existing views and engagement patterns. This could make it harder for users to encounter diverse perspectives or verify claims across different communities, especially during election periods. Public discussion requires some degree of shared information – when different groups of voters are seeing fundamentally different versions of political issues, it becomes more challenging to engage in informed debate.

The Australian Context

As the authors of the open letter from fact-checking organisations around the world observes, Meta’s plan to end the fact-checking program in 2025 applies only to the United States, for now. They note however that Meta has similar programs in more than 100 countries covering diverse political systems and stages of development. Hauntingly, they suggest that if Meta decides to stop the program worldwide, it is almost certain to result in real-world harm in many places. For now, we will likely have some comparative case studies to observe the resulting impacts of the professional verses community-lead models of fact checking within the Facebook environment, but this may change at any point.

In the Australian context, timing matters. With our federal election approaching, changes to how information is verified on Meta’s platforms could affect how Australians access and share political information. While we have strong fact-checking institutions like RMIT Lookout, which operates independently, Meta’s platforms play a distinct and significant role in how many Australians encounter and share political information. Australia’s concentrated media market means that changes to Meta’s platforms can have significant effects on what information reaches Australian audiences.

Beyond Content Moderation

Rather than just managing what content is visible, platforms need to support the infrastructure that helps people access and verify credible information. The challenges we face go beyond simple content filtering – they require a comprehensive approach to building resilient information ecosystems. This means:

  • Developing better systems to identify accurate information through a combination of automated detection, expert verification, and community input. These systems need to work proactively rather than reactively, identifying potential misinformation before it goes viral and providing real-time verification tools that users can access directly.
  • Supporting rather than undermining professional fact-checking by providing fact-checkers with better tools, resources, and platform access. This includes maintaining partnerships with accredited fact-checking organisations, ensuring transparent access to content spread data, and integrating fact-checking more deeply into platform architectures.
  • Creating tools that help bridge information divides by making verified information more accessible and engaging. This could include features that surface diverse perspectives from credible sources, tools that help users understand the context and history of viral claims, and systems that encourage cross-pollination of verified information across different communities.
  • Investing in digital literacy through both platform features and educational initiatives. This means building in-platform tools that help users evaluate information credibility, supporting external digital literacy programs, and developing resources that help users understand how information spreads online and how to verify claims they encounter.
  • Ensuring platform accountability through transparent reporting on content moderation decisions, clear appeals processes, and regular independent audits of platform practices. Without accountability measures, even the best systems can be undermined by inconsistent enforcement or political pressure.

This comprehensive approach recognises that effective content moderation isn’t just about removing harmful content – it’s about building an information environment that helps users make informed decisions and engage meaningfully with online discourse.

Looking Ahead

As we approach our federal election, these changes deserve careful attention. While Meta’s commitment to reducing over-enforcement of content moderation is understandable, we need to consider how changes to fact-checking systems might affect Australians’ ability to access credible information about political issues and engage in informed public discussion.

As the Carson et al.’s 2022 study suggests, fact-checkers could more clearly state they are fact-checking a politician’s specific claim rather than the media coverage containing it. They also recommend that journalists may need to more actively adjudicate false claims within their original reporting rather than relying solely on external fact-checkers.

The challenge isn’t just about determining what’s true or false. It’s about maintaining systems that help Australians access reliable information they can use to understand and discuss important political issues. Meta’s policy shift suggests they may be stepping back from this role just when clear, credible information is most needed.

The research evidence suggests that fact-checking, while important, needs to be implemented thoughtfully to avoid undermining trust in legitimate journalism. As we move forward, the key question isn’t just about free speech versus restriction – it’s about how we maintain the integrity of our shared democratic conversation in an increasingly fragmented digital landscape.

What can you do?

While some choose to opt out of social media platforms due to concerns about misinformation and toxicity, this isn’t always feasible or productive, especially as these challenges proliferate across multiple platforms. Given that I am an educator, it should come as no surprise that I think education is key and that I would emphasise the importance of building information and digital literacy skills and capabilities that work across all online environments.

Here are essential strategies for navigating online information:

  • Verify the source’s credibility: Check their track record, expertise, and potential biases
  • Watch for emotional manipulation: Be especially skeptical of content designed to provoke strong emotional reactions
  • Check dates and context: Old content is often recycled and presented as current news
  • Cross-reference information: Look for multiple reliable sources covering the same topic
  • Apply logical scrutiny: Ask yourself if the claim aligns with what you know about the person, organisation, or situation
  • Look for Community Notes: While not perfect, they can provide valuable additional context

This questioning is really the start point for critical thinking.

For AI-generated content and deepfakes specifically:

  • Watch for visual inconsistencies: Look closely at hands, teeth, backgrounds, and reflections
  • Check for unnatural movement in videos: Pay attention to lip synchronisation and eye movements
  • Be especially wary of crisis-related content: Deepfakes often exploit breaking news situations
  • Use reverse image searches: Tools like Google Lens can help identify manipulated images
  • Pay attention to audio quality: AI-generated voices often have subtle irregularities

The challenge of identifying synthetic media is growing as AI technology becomes more sophisticated. This makes it crucial to develop a strong understanding of current events, public figures, and social issues over time. This contextual knowledge becomes our foundation for evaluating authenticity. However, it’s important to acknowledge that building these skills takes time – and that’s okay. Critical thinking and digital literacy are ongoing practices that we develop gradually, not a checklist we need to master overnight.

While individual skills are vital, we shouldn’t shoulder this burden alone. This is precisely why Meta’s shift away from professional fact-checking and reduced content moderation safeguards is concerning. Platforms have access to advanced detection tools, professional fact-checkers, and technical expertise that complement our personal verification efforts. At the same time, we need to acknowledge that AI agents are becoming integral to how we create and make sense of online content.

Rather than seeing AI solely as a threat or feeling overwhelmed by the need to become expert evaluators, we can approach this as a gradual learning process. This includes building our understanding of AI’s capabilities and limitations over time, learning to use these tools productively while maintaining critical awareness, and recognising that our digital literacy will evolve alongside these technologies. However, this individual growth needs to be supported by robust platform policies and professional fact-checking resources – not treated as a replacement for them. As platforms experiment with systems like Community Notes, they must recognise that effective content moderation requires a multi-layered approach combining institutional resources, professional fact-checkers, and community participation.

Beyond Age Limits: What’s Missing in Australia’s Social Media Ban Discussion

~ alexiamaddox ~ Leave a comment

Why are we talking about this now?

The ABC’s recent article “The government plans to ban under-16s from social media platforms” lays out the mechanics of Australia’s proposed social media age restrictions. The timing of this announcement is significant – with only two parliamentary sitting weeks left this year and an election on the horizon, both major parties are backing this policy. This follows months of mounting pressure from parent advocacy groups like 36 Months, and builds on earlier discussions about protecting children from online pornography.
But while the article explains what will happen, there are critical questions we need to address about whether this approach will actually work – and what we might lose in the process. This isn’t just about technical implementation; it’s about understanding why we’re seeing this push now and whether it represents meaningful policy development or political opportunism.
The recent Social Media Summit in Sydney and Adelaide highlighted how this debate is being shaped. Rather than drawing on Australia’s world-leading expertise in digital youth research, the summit featured US speakers promoting what has been referred to as a “moral panic” approach. This raises questions about whether we’re developing evidence-based policy or responding to political pressures.

The Policy vs Reality

Yes, platforms will have 12 months to implement age verification systems and we will no doubt see push back from platforms on this. Yes, the definition of social media is broad enough to capture everything from TikTok to YouTube to potentially Discord and Roblox.

Additionally, the government’s ability to enforce age restrictions on global social media platforms raises significant practical and legal challenges. While Australia can pass domestic legislation requiring platforms to verify users’ ages, enforcing these rules on companies headquartered overseas is complex. Recent history shows platforms often prefer to withdraw services rather than comply with costly local regulations – consider Meta’s response to Canadian news legislation or X’s ongoing resistance to Australian eSafety Commissioner directives.

Any proposed penalties may not provide sufficient incentive for compliance, particularly given these platforms’ global revenues. Additionally, even if major platforms comply, young people could simply use VPNs to access services through other countries, or migrate to less regulated platforms beyond Australian jurisdiction.

Without international cooperation on digital platform regulation, individual countries face significant challenges in enforcing national regulations on global platforms. This raises a crucial question: will platforms invest in expensive age-verification systems for the Australian market, or will they simply restrict their services here, potentially reducing rather than enhancing digital participation options for all Australians?

What is missing from this conversation?

  1. Digital Equity: The broad scope of this ban could particularly impact:
    • Regional and remote students using these platforms for education
    • Marginalised youth who find support and community online
    • Young people using gaming platforms for social connection
  2. Privacy Trade-offs: The proposed verification systems mean either:
    • Providing ID to social media companies
    • Using facial recognition technology
    • Creating centralised age verification systems
    • All of these raise significant privacy concerns – not just for teens, but for all users.
  3. Unintended Consequences: International experience shows young people often:
    • Switch to VPNs to bypass restrictions
    • Move to less regulated platforms
    • Share accounts or find other workarounds

A More Nuanced Approach

Rather than focusing solely on age restrictions, we could be:

  • Making platforms safer by design
  • Investing in digital literacy education
  • Supporting parents and educators
  • Listening to young people’s experiences
  • Learning from international approaches like the EU’s Digital Services Act

Looking Forward

While the government’s concern about young people’s online safety is valid, and is shared by researchers, families, school teachers and young people alike, the solution isn’t as simple as setting an age limit. Young people develop digital capabilities at different rates, and their resilience online often depends more on their support networks, digital literacy, and individual circumstances than their age alone.
The Centre of Excellence for the Digital Child’s research demonstrates that some young people are highly capable of identifying and managing online risks, while others need more support – regardless of age. This is particularly important when we consider:

  • Some younger teens demonstrate sophisticated understanding of privacy settings and online safety
  • Many vulnerable teens rely on online communities for crucial support
  • Digital literacy and family support often matter more than age in online resilience
  • Regional and remote youth often develop advanced digital skills earlier out of necessity

We need approaches that protect while preserving the benefits of digital participation, recognising that arbitrary age limits may not align with individual capability and need.
This better reflects the evidence while acknowledging:

  • The validity of safety concerns
  • The complexity of digital capability development
  • The importance of context and support
  • The need for nuanced policy responses

The Joint Select Committee on Social Media and Australian Society is still to deliver its final report. Perhaps it’s worth waiting for this evidence before rushing to implement restrictions that might create more problems than they solve.

EDIT: They have now released their final report, with some excellent recommendations… and no mention of an age ban.

The Bottom Line

Protection and participation aren’t mutually exclusive. We can make online spaces safer without excluding young people from digital citizenship. But it requires more nuanced solutions than age barriers alone can provide.