AI in Government and Society

AI in government is the use of statistical models, automated decision systems and, more recently, generative AI by public bodies to administer benefits, assess risk, support courts and police, and deliver services. It differs from private-sector use because the state can impose binding decisions on people who cannot opt out, and the law places specific duties of transparency, due process and non-discrimination on public decisions.

Public bodies handle large volumes of repetitive decisions, which makes automation attractive, and they hold the data to build predictive models. The same features create the main risks: errors at scale, bias in who is flagged, and decisions that are hard to contest. The cases that shaped policy are mostly failures, so this article keeps three classes distinct:

Some of the best-known cases involve rule-based automation rather than machine learning (for example, Australia's Robodebt, below). They are included because the governance questions are the same, and the distinction is stated each time.

Governments have used statistical risk assessment for decades. A 2020 study commissioned by the Administrative Conference of the United States (ACUS) surveyed AI and machine-learning use across US federal agencies and the accountability, transparency and non-discrimination challenges they raise (Engstrom et al., 2020). Controversies followed: a Dutch tax-authority risk model and the childcare-benefits scandal (Amnesty International, 2021), Australia's Robodebt scheme (2015-2019), face-recognition accuracy disparities documented by independent academics and by NIST (Buolamwini & Gebru, 2018; Grother et al., 2019), and recidivism-prediction tools used in US courts (Dressel & Farid, 2018). Policy responded with instruments such as the EU AI Act (European Union, 2024) and US federal memoranda, which have changed repeatedly (Section 8).

Prediction versus decision. A risk model estimates $\hat{p}(x) = P(Y=1 \mid x)$, for example the probability that a person will reoffend, fail to appear or commit benefit fraud. A policy rule then turns $\hat{p}$ into an action, such as detain, flag or audit, via a threshold $\tau$. Error rates at that threshold, and who bears them, depend on the data and on the cost chosen for false positives versus false negatives.

Selective labels. In bail and fraud settings the outcome is observed only for people who were released or investigated, so evaluating a new rule is statistically hard. Kleinberg et al. (2018) addressed this with econometric strategies, including quasi-random assignment of cases to judges, and reported in a policy simulation potential crime reductions of up to 24.7% with no change in jailing rates, or jailing-rate reductions of up to 41.9% with no increase in crime. These are simulations on historical data, not outcomes of a deployed system.

Feedback loops and proxies. Models trained on enforcement records may encode past enforcement intensity rather than underlying behaviour, and features such as nationality can act as direct or indirect discriminators (see the Dutch case below).

• Benefits and tax administration: eligibility checks, fraud-risk scoring, debt recovery.
• Justice and policing: pretrial risk assessment, recidivism scores, face recognition, predictive policing.
• Border and migration management: document and biometric verification, risk flagging.
• Citizen-facing services: chat assistants, translation, document summarisation.
• Internal administration: case triage, procurement, regulatory analysis.
• Rule-based automation versus learned models: a system may be a fixed rule applied automatically (data-matching, averaging) or a model trained from data. Governance and legal remedies are similar; technical failure modes differ.

#6.1 Deployed systems and documented outcomes

• US federal agencies. A report prepared for ACUS surveyed federal use of AI/ML and interviewed officials, identifying promising uses and challenges of accountability, transparency and non-discrimination (Engstrom et al., 2020). This is a 2020 snapshot; current agency inventories are published under later policy instruments.
• Dutch childcare benefits. According to Amnesty International (2021), the Dutch tax authority used a risk-classification model that treated non-Dutch nationality as a risk factor, assigning higher scores to non-Dutch nationals; tens of thousands of parents and caregivers, disproportionately from ethnic minorities, were falsely accused of fraud, and the scandal contributed to the government's resignation in January 2021. This account is from a human-rights NGO (Tier 4); official Dutch inquiry and Data Protection Authority findings were not retrieved [VERIFICATION NEEDED: official primary sources for the model's design and the regulator's findings].
• Robodebt (Australia, rule-based automation). The Royal Commission's report of July 2023 found that the scheme, which ran from July 2015 to November 2019, relied on averaging annual income across fortnights to raise debts and reversed the onus of proof; secondary reports state that about 1.73 billion Australian dollars in unlawful debts were raised against more than 400,000 people and that the Commission made 57 recommendations, calling the scheme "crude and cruel" (Law Society of NSW Journal, 2023) [VERIFICATION NEEDED: figures from a secondary report; the Royal Commission report itself was not retrieved]. It was an automated data-matching and calculation scheme, not a machine-learning model.
• Face recognition. The NIST Face Recognition Vendor Test, Part 3 (demographic effects), published on 2019-12-19, found that false-positive differentials across demographic groups are much larger than false-negative differentials and exist for many but not all of the algorithms tested, with the most accurate algorithms showing smaller differentials (Grother et al., 2019; figures as described in NIST summaries, full report not re-read). An earlier academic audit reported intersectional accuracy disparities in commercial gender classification (Buolamwini & Gebru, 2018).

#6.2 Validated results and independent evaluations

• Recidivism tools. A study in Science Advances compared a widely used commercial tool (COMPAS) with predictions by people with little criminal-justice expertise and with simple models using few features, and examined accuracy and fairness (Dressel & Farid, 2018) [VERIFICATION NEEDED: findings stated at the level of the paper's subject; abstract not fully read].
• Bail decisions. See section 4 (Kleinberg et al., 2018), a retrospective simulation.

#6.3 Research demonstrations and emerging use

Generative AI pilots in public administration (drafting, summarising, citizen chat) were not evaluated with peer-reviewed studies in the sources used for this draft; claims about their savings are unverified.

Advantages. Consistency, speed and potential accuracy gains in high-volume decisions (Kleinberg et al., 2018, in simulation); the ability to audit a model's behaviour more systematically than individual human discretion.

Limitations.

• Scale of error. Automated errors reach many people quickly (Robodebt; Dutch benefits).
• Discrimination. Disparities in face-recognition error rates (Grother et al., 2019; Buolamwini & Gebru, 2018) and use of nationality as a risk factor (Amnesty International, 2021).
• Weak incremental accuracy. A commercial recidivism tool was compared with non-experts and simple models, raising the question of what complexity adds (Dressel & Farid, 2018).
• Contestability. Affected people may not know a model was used or be able to challenge it.
• Evidence gaps. Public-sector deployment often lacks prospective evaluation.

Unresolved questions. What standard of accuracy and fairness should be required before use; whether human review meaningfully constrains automated recommendations; how to define and measure fairness when criteria conflict; liability and redress.

As of 2026-10-15, the legal frameworks are in flux and dated claims must be checked.

• EU. The AI Act, Regulation (EU) 2024/1689, classifies as high-risk certain systems used in access to essential public services and benefits, law enforcement, migration and border control, and administration of justice, and prohibits certain practices, including social scoring (European Union, 2024) [VERIFICATION NEEDED: Annex III point numbering and the exact wording of Article 5 were recalled, not re-read in this session]. The Digital Omnibus on AI, Regulation (EU) 2026/1744, published in the Official Journal on 2026-07-24, defers high-risk obligations for stand-alone Annex III systems to 2027-12-02 and for Annex I product-embedded systems to 2028-08-02 (Art. 113(c) as amended; checked against the Official Journal text on EUR-Lex; see GOV-EU-001).
• United States. Executive Order 14110 was revoked on 2025-01-20. OMB memorandum M-25-21 (2025-04-03), implementing Executive Order 14179, rescinds and replaces M-24-10 on federal agency use of AI and shifts emphasis toward innovation while keeping some safeguards, including for presumptively high-impact uses (OMB, 2025, per the memorandum and secondary summaries). Whether M-25-21 remains in force and unamended on 2026-10-15 was not verified [VERIFICATION NEEDED]. See GOV-US-001.

Evidence classification: documented harms from automated systems in benefits and policing are Well Supported; net benefit of specific deployed predictive tools is Contested; prospective evaluation of generative AI in public services is lacking.

Near-term (0-3 years): application of EU high-risk rules to public bodies on the revised timetable; agency inventories and impact assessments in the US under current memoranda; more generative-AI pilots. Medium-term (3-10 years): standards for audit and redress; evidence from evaluations. Long-term claims about automated governance are speculative and not rated.

• Foundational Concepts:
  • GOV-LAW-001 AI Governance: General regulatory frameworks.
• Related Methods & Technologies:
  • GOV-EU-001 The EU AI Act: High-risk classes for public bodies.
  • GOV-US-001 United States AI Policy: Executive and OMB instruments.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Error behaviour relevant to generative pilots.
• Advanced Implications & Future Trajectories:
  • APP-PHYS-001 AI in Industry, Robotics and Autonomous Vehicles: Regulation of physical autonomous systems.
  • APP-HEALTH-001 AI in Healthcare and Medicine: Bias in health-system algorithms.
  • SAFE-ALIGN-001 AI Alignment: Values and objectives in automated decisions.