Science is about making the world clearer and more understandable. By classifying the world into observable, repeatable, verifiable phenomena we move towards a shared sense of reality rather than an individual, subjective one. A cornerstone of contemporary science is how well any research stands up to the rigour of peers questioning and responding to it.
But to question, we must first know, to repeat we must first see. This requires openness and transparency. The clarity of contemporary science is giving way to more opaque systems in the digital era, where complex algorithms are guarded by intellectual property, commercial patents and secret code, and the inner workings of significant global products are hidden from us.
This is increasingly moving us towards a black box society, where we no longer see or understand the world around us, as everyday products and platforms become opaque systems of code, algorithms and AI that we can’t reach, and therefore can’t hope to comprehend. A society where we don’t know how key decisions are made, and which specific individuals or groups can be held to account for those decisions.
In some cases, these products and systems develop recommendations that are inane and trifling, like what movie to watch next, what book you might like to buy as a gift, or a lost password for a playlist. But other times they are life-changing, like a job rejection, who you should date, accurate news, a home loan application or even deportation.
We assume that most of the time, as long as these black box systems work, we don’t need to know how they function internally – like the traffic light system on our roads. Or that if something goes wrong, they will be fixed, governed by systems of accountability that are generally fair and reasonable.
But what happens when those systems cause harm, or start to act in ways that are not in the community’s best interests? What happens when we have no humans who are accountable for these systems who we can talk to, appeal to, or hold responsible if systems malfunction?
As software eats the world, we are in danger of being ruled by large, incomprehensible, and impenetrable black boxes. From convoluted government services to social media platforms, to ubiquitous digital products like Google search, and now, a generation of AI products and its large language models.
How do everyday people have agency when we are ruled by software that we are unable to detect, let alone reason with, or defend against?
Public black box harms
The public sector and government services are increasingly being digitised, from digital IDs, to Medicare and health services, to tax filing and returns, and more. There are both risks and opportunities with this. Recent events demonstrate that when black box systems aren’t properly managed, they can lead to devastating consequences.
Robodebt has been one of the worst examples of automated decision making causing significant harms to the community. At the centre of the Robodebt fiasco was an algorithm that cross-referenced fortnightly Centrelink payments with annual income from the Australian Taxation Office, fundamentally misunderstanding the flow of casual or irregular work for many Centrelink recipients. The algorithm was supposed to identify overpayments and “welfare cheats”, saving the government money. Instead it ultimately cost the government almost half a billion dollars, resulted in a Royal Commission and ruined lives.
Robodebt baked in inherent biases against vulnerable people and removed the minimum levels of human oversight and checks and balances to ensure the system wasn’t failing. It also didn’t allow for any contestability or ability for the public to challenge its findings, since there were no humans on the frontline who could explain the claims.
A significant stain in public administration that should serve as a warning, a cautionary tale, and a case study of what not to do for years to come, Robodebt is one of the worst instances of the fallout from black box systems.
There are other examples – immigration and residency being a notable one. Using algorithms for visa processing could result in systemic biases against applicants from countries that are deemed medium-to-high risk. In the UK, this flaw was revealed when a legal dispute showed the inner workings of a Home Office visa processing algorithm. Those from medium-to high-risk countries led to more rejections and “breaches”. And although this may have been legitimate at the start of the screening process, over time these breaches get flagged in the system. As the algorithm determines current applicants based on past results these breaches continually show up. This means that breaches get magnified for countries that are flagged, amplifying the rejections and causing a feedback loop against those countries.
It’s not clear whether Australia’s Department of Home Affairs has similar algorithms given these are not made transparent, but it’s not hard to imagine similar systems being in place, trialed or being investigated for future use.
As our lives grow more digital, black box systems are also at risk of getting exposed and targeted by bad actors, including for sensitive data like our health. Medisecure was formerly contracted to provide electronic prescriptions by the federal government until 2023 and was recently subject to a large-scale ransomware data breach. The investigation is ongoing at the time of writing, so details of the number of individuals and what data was exposed were unclear, except that the breach was “large-scale”.
Clearly, black box systems for public services, their design, governance and impacts are consequential for the Australian public. For the most part however, these are mostly black swan or unusual events. Generally speaking, our public services work for the majority of people, and when they don’t, it’s noticeable, and there are systems of accountability in place, with specific people who can be held responsible.
What is more concerning is the increase in private black boxes. In particular, from large-scale commercial tech companies whose products continue to infiltrate our lives.
Why private black boxes are worse
Private black box products and services owned by private companies can be much harder to manage. By nature, internal workings are closely guarded secrets, protected by commercial in confidence agreements and contracts. But what happens when those systems fail, and their leaders are not held to account?
Recent high-profile hacks, like Optus and Medibank, have alerted the country to the vulnerability of our data, and serve as a reminder that our digital footprints can be stored, maintained and used against us, even when we didn’t think we had those footprints in the first place (as many past customers of these companies discovered).
Where there are substitutes, people can simply punish these companies by walking, taking their business to a competitor. However, for the largest digital platforms, this is not possible, as they have become what the ACCC deemed “unavoidable trading partners” – virtual monopolies with no real competitors.
This is the case with social media platforms. The last few years have revealed that they have facilitated disinformation which destabilises our society and breaches our privacy. They have employed invasive practices, produced products that can harm our mental health, and created echo chambers which can divide and enrage us.
The potential harms are now well known, and the backlash against social media platforms reached fever pitch recently with the South Australian government calling to ban social media access for children under 14.
And yet the algorithms which power these platforms continue to be opaque black boxes. Only exposure from whistleblowers and journalists seem to reveal the harms, like when whistleblower Frances Haugen disclosed thousands of Facebook’s internal documents to the Securities and Exchange Commission and The Wall Street Journal in 2021. The documents pointed to Facebook executives being aware of dangers to adolescents and other users and failing to act.
Online safety regulation is gaining some ground against the platforms, but their function is mostly about compliance and mitigation of harms after the fact. The most targeted regulation against social media black boxes to date has been a lukewarm voluntary code with self-reporting annual transparency reports from platform signatories.
And for many the cost of disconnecting from these platforms is simply too high, as it means losing contact with their networks, being cut off from important personal and professional conversations, and missing out on critical news and information. So the public continue to ignore the harms and tolerate the opaque algorithms.
Google’s ubiquitous search engine has been the centre of the digital ecosystem, digital commerce and digital life in general over the past two decades, and it is one of the biggest black boxes to date.
Google’s black box has been so important and valuable that an entire cottage industry has developed around trying to help people understand and leverage it – the search engine optimisation industry.
This multi-billion dollar industry is showing signs of oversaturation however, and many are starting to notice that Google’s search engine and its results seem to have been getting progressively worse.
Google search results now seem bloated and inefficient, and its valuable front page is littered with ads and sponsored posts. A recent study from researchers at Leipzig University, Bauhaus-University Weimar, and the Center for Scalable Data Analytics and Artificial Intelligence found “a torrent of low-quality content…drowning any kind of useful information in search results”. This follows an earlier incident where other search specialists reported a spam attack lasting days.
It’s a worrying thought that Google, one of the most powerful companies in the world, owner of one of the most important digital products used globally, may have lost control of its product. Or worse, knowingly allowed its quality to deteriorate.
Google search was predicated on the idea that the world’s information should be free and available to everyone. But the world’s information should never have been allowed to be managed by a single private company. Instead of public libraries that are genuinely available to everyone, Google is a private black box that pretends to be a neutral public utility, but is anything but. We have little choice in the matter however, as there are simply no real viable alternatives to Google’s search engine.
The generative AI trend over the last couple of years appears to be the one of the few things disrupting Google’s monopoly in this space (aside from global attempts at antitrust suits), as it joins the arms race with other tech giants vying for AI supremacy.
Google’s incorporation of its AI assistant into the core search product has met with uneven results to date, with early AI-powered results giving bizarre, nonsensical and sometimes dangerous recommendations, like telling people to glue toppings on their pizza to prevent them from slipping out, to eat rocks as a source of minerals, or that pregnant women should smoke two to three cigarettes a day.
As the race for AI dominance heats up, we have little choice but to be subjected to these experiments by tech companies, as they incorporate black box products to market and release them to the public.
There are at least attempts at trying to regulate AI around the world, with regulation like the EU’s AI Act specifying more interpretability and transparency.
While some of the most popular AI large language models today claim to be open source, including Meta’s Llama and X’s Grok, most are proprietary black boxes. This includes OpenAI’s popular ChatGPT. OpenAI famously backflipped from being open source for its earlier GPT models, to a black box once it switched its original non-profit structure into a commercial one partnering with Microsoft. Google is the same, and Apple also prefers a closed system approach for its AI, with reports of its AI products processed in a “cloud black box”.
Shockingly, AI models are reportedly even black boxes to those who develop and study them. This was openly admitted by OpenAI’s Sam Altman, and Anthropic’s Dario Amodei, who said that “we do not know how to understand what’s going on inside these models”.
This is because most are set up as neural networks, attempts at emulating the processes of the human brain. Models are given vast amounts of data, and the model itself creates its own patterns and connections from the data, generating an output. How it arrived at that output, however, is a mystery. Hence, hallucinations’ and fabrications like the earlier Google examples.
AI systems began by trying to emulate human intelligence, but now we find ourselves in a situation where these AI systems process and complete tasks in ways that are completely different to how humans would do it, creating a new type of machine logic we do not comprehend. And yet we continue to deploy these models as they are.
This isn’t even the first time an AI product was released to the public and had problematic results. As early as 2016, Microsoft released an AI chatbot called Tay that Twitter users quickly tricked into making racist tweets. Microsoft had to shut it down.
While there are humans who can intervene, shutting down chatbots like Tay, or scrambling to manually update bizarre search results in Google are possible. These have been fairly isolated incidents.
But what happens when we allow these black box AI products to be rolled out on a massive scale, across the giant tech companies whose products and services are used by millions of people globally? And what happens when there are more machine-to-machine interactions within these systems that no longer need human intervention?
This may seem alarmist, but the potential for disaster will continue to be a real, plausible risk while these systems remain unexplainable and opaque, lacking proper oversight and the checks and balances needed for proper governance.
Shining a light on black boxes
As we embrace digitisation and incorporate more virtual products, AI, and complex technologies like quantum computing into our public and private sectors, we need to ensure that we continue to design these systems in ways that humans can continue to understand and oversee.
Government policy needs to build in transparency and auditability and ensure we don’t live in a society reliant on black box technology with no means of intervention or accountability.
The final report on the Robodebt Royal Commission, a whopping 1,000-plus pages, came with 57 recommendations, some of which are good proposals to consider for black box systems. Some recommendations were essentially about inserting more humans into the mix, including more face to face support for the public, and human support staff. This is a generally good idea when it comes to public services and customer/public facing systems. Many private companies have done away with human customer support services, preferring instead to have AI chatbots, or self-service wikis rather than having people respond to queries. Where humans are available, often the user must first run through the gauntlet of technology mediated Q&As, request qualifiers and automated checking, and only once when those options are exhausted does it get escalated to humans.
Another key proposal from the Robodebt report is for stronger regulation and frameworks when implementing automated decision-making projects, including clearer guidelines for processes, transparency on business rules and algorithms so that external independent scrutiny is possible. A dedicated audit body which looks at automated decision making in the public service was also among the proposals.
Other ideas include a foundational “right to reasons” proposed by the Australian Human Rights Commission, a legislated right to request clarification or an explanation behind automated decision making.
When it comes to private entities, there are considerations around intellectual property and commercial in confidence arrangements.
There are some ideas which suggest successful alternative results as a way of explaining decisions – for example, if a black box system refused your approval for a loan, it could describe an alternative scenario where you would have been successful – e.g. had you applied for a different type of loan, or less money.
By providing a scenario with a successful alternative, you don’t necessarily need to look inside the black box, thereby protecting the black box’s secrets. You only need to know that there were different actions to take for a successful outcome.
However, this still assumes that the black box algorithm is working as intended, with no baked-in biases or variables that cause harm. Ultimately the only way to guarantee this is to insert some ability to interrogate the algorithms in question.
Anthropic AI, founded by former OpenAI staff who were particularly concerned about AI safety, and which is at least open about the black box nature of their AI systems, has released research that claims to go towards understanding AI neural networks a bit more.
Using a technique called dictionary learning, it uncovers patterns of activated neurons, giving a glimpse at how those neurons interpret a concept or feature. For example, when they gave their AI a prompt talking about driving from San Francisco to Marin County, they interrupted the network, and it revealed neurons representing the Golden Gate Bridge and related concepts.
While this progress should be welcomed, it continues to demonstrate the voluntary nature of these types of experiments and disclosures. Anthropic has chosen to share its research on its black boxes, but some other commercial companies have only provided spin and excuses for recent errors.
If we are to incorporate these large AI models into critical services, there must be legislated rules about AI interpretability, explainability and auditability.
There are even more added layers of complexity when it comes to public adoption of these private black boxes – for example, if governments choose to use private AI models for government services. Then it would be critical that transparency and accountability measures are put firmly in place.
They say technology that’s suitably advanced is indistinguishable from magic. Let’s ensure that with all the new technology we’re embracing, we don’t slip into a dark age of opacity and techno-mysticism, where only the wizards who control the black boxes know and understand the forces that govern us, and the rest of us live under the shadow of blindness, hearsay and ignorance.
Jordan Guiao, director of responsible technology at Per Capita’s Centre of the Public Square. He is a technology strategy and policy leader interested in creating a safer and more inclusive online experience for all Australians. He has over 15 years’ experience as a technology leader with broadcasters including the ABC, SBS and Network Ten. Jordan is also the chief digital officer for The Intermedia Group.
This article is part of The Industry Papers publication by InnovationAus.com. Order your hard copy here. 36 Papers, 48 Authors, 65,000 words, 72 page tabloid newspaper + 32 page insert magazine.
The Industry Papers is a big undertaking and would not be possible without the assistance of our valued sponsors. InnovationAus.com would like to thank Geoscape Australia, The University of Sydney Faculty of Science, the S3B, AirTrunk, InnoFocus, ANDHealth, QIMR Berghofer, Advance Queensland and the Queensland Government.
Do you know more? Contact James Riley via Email.