---
id: PHIL-CIV-001
title: "Civilization and Advanced Intelligence"
alternative_names:
  - "AI and the Future of Civilization"
  - "Long-Run Consequences of Advanced AI"
category: "Philosophy → Civilization and the Long Term"
volume: XI
chapter: 62
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "ASI-001"
related_entries:
  - "ASI-EXP-001"
  - "SAFE-XRISK-001"
  - "SAFE-ALIGN-001"
  - "ECON-LABOR-001"
  - "PHIL-MIND-001"
  - "PHIL-COSMOS-001"
  - "AGI-001"
---

# Civilization and Advanced Intelligence

## 1. Concise Definition
This article examines philosophical and social-scientific arguments about how advanced artificial intelligence, including systems more capable than humans in many domains, could affect human civilization over decades to millennia: its economy, institutions, vulnerability to catastrophe and long-run trajectory. It is a survey of arguments and scenarios, not a prediction. No system with the capabilities described in the speculative scenarios exists as of 2026-10-15.

## 2. Overview & Significance
Most of the encyclopedia deals with systems that exist or are under active research. This article concerns long-run implications that depend on assumptions about future capabilities, discussed in [ASI-001] and [AGI-001]. Because the evidential quality varies sharply, every section separates three layers, labelled throughout:

- **[Established science / history]**: documented past facts and well-tested analysis, such as how earlier waves of automation affected employment.
- **[Philosophical argument]**: reasoned positions on values and priorities that evidence cannot settle by itself (for example, how much to weight the far future). Rated **[Contested]** unless stated otherwise.
- **[Speculation]**: scenarios about systems that do not currently exist. Rated **[Hypothetical]** under the Evidential Framework; they are not forecasts.

The article makes no probability estimates. The few quantified expert-opinion sources it points to are cited as opinion, not as evidence of outcomes.

## 3. Historical Development
The idea that machine intelligence could transform or end human dominance has a long lineage. I. J. Good (1966) wrote about a "first ultraintelligent machine", and Vernor Vinge (1993) described a coming "technological singularity" in a symposium paper for NASA Lewis Research Center; both are speculative essays **[Hypothetical]** (see [LIT-SOURCES-001]). Bostrom (2014) developed a systematic argument about paths to and risks of superintelligence. Russell (2019) argued that the standard model of building systems to optimize fixed objectives is problematic for highly capable systems.

In parallel, longer-term ethics developed. Bostrom's (2003) *Astronomical Waste* argued that, on certain utilitarian assumptions, delaying technological development has a very large opportunity cost, which implies that "we ought to maximize its safety" rather than its pace (abstract). Ord (2020) and MacAskill (2022) are book-length treatments of existential risk and long-term thinking whose arguments were not re-read for this draft. [VERIFICATION NEEDED: page-level claims from Ord (2020) and MacAskill (2022) before any are quoted]

Empirical economic analysis of automation is older: Autor (2015) and Acemoglu and Restrepo (2019) analyze how automation affects labor, described in section 4.1.

## 4. Technical / Conceptual Explanation

### 4.1 Layer one: established analysis of technological change **[Established science / history]**
Autor (2015) states that automation has not eliminated most jobs over past decades and centuries because it "also complements labor, raises output in ways that leads to higher demand for labor, and interacts with adjustments in labor supply", while also changing "the types of jobs available and what those jobs pay", with a "polarization" of the labor market in recent decades. Acemoglu and Restrepo (2019) present a task-based framework in which automation has a displacement effect that reduces labor's share, countered by a reinstatement effect from new tasks. These are analyses of past technologies. Whether they extend to systems that perform most cognitive tasks is **[Contested]**, and is treated in [ECON-LABOR-001].

### 4.2 Layer two: philosophical arguments **[Philosophical argument]**
1. **Opportunity-cost argument.** On assumptions stated in Bostrom (2003), the potential value lost by delay is very large, so safety dominates speed. The argument depends on contested premises about aggregating welfare across enormous future populations; "person-affecting" views accept only a modified conclusion, as the abstract notes. **[Contested]**.
2. **Vulnerable world argument.** Bostrom (2019) introduces "a vulnerable world", one in which "there is some level of technological development at which civilization almost certainly gets devastated by default". The paper analyses "counterfactual historical and speculative future vulnerabilities" arranged in a typology. It is an analytic framework, not an empirical demonstration that such a level of development exists. **[Contested]**. It is relevant to AI because advanced systems could be one source of such vulnerabilities; whether they are is **[Hypothetical]**.
3. **Control argument.** Russell (2019) argues for a different design principle for AI objectives. **[Contested]**; see [SAFE-ALIGN-001].
4. **Moral circle expansion.** If digital minds could be moral patients, civilization may include new kinds of beings with claims on resources (Shulman & Bostrom, 2021); see [PHIL-MIND-001]. **[Contested]**/**[Hypothetical]**.

### 4.3 Layer three: speculative scenarios **[Speculation]**
- **Intelligence explosion and singularity.** Good (1966) and Vinge (1993) speculate about rapid self-improvement and a break in human predictability. The mechanism is treated in [ASI-EXP-001]. **[Hypothetical]**.
- **Gradual disempowerment.** Kulveit et al. (2025) develop a concept of "gradual disempowerment", contrasting it with "abrupt takeover scenarios". They argue that "even incremental improvements in AI capabilities can undermine human influence over large-scale systems that society depends on, including the economy, culture, and nation-states". It is a conceptual argument in a preprint by five authors. **[Hypothetical]**, **[Preliminary]** as an argument; it has no empirical test yet.
- **Emulated minds.** Hanson (2016) analyses a hypothetical economy of brain emulations. The scenario assumes technology that does not exist. **[Hypothetical]**.

## 5. Types, Components & Classifications
| Domain of civilizational effect | Established base | Contested or speculative extension |
| :--- | :--- | :--- |
| Labor and economy | Past automation analyses (Autor, 2015; Acemoglu & Restrepo, 2019) | Full-scale cognitive automation (see ECON-LABOR-001) **[Contested]** |
| Catastrophic risk | Typology of vulnerabilities (Bostrom, 2019) as an analytic frame | AI as a source of civilizational vulnerability **[Hypothetical]** |
| Institutions and power | No verified source in this draft | Gradual disempowerment (Kulveit et al., 2025) **[Hypothetical]** |
| Long-term value | None (normative) | Opportunity cost of delay (Bostrom, 2003) **[Contested]** |
| Moral community | None (normative) | Digital minds (Shulman & Bostrom, 2021) **[Hypothetical]** |

## 6. Applications & Concrete Examples
The arguments are used in discussions of AI safety priorities and governance; see [SAFE-XRISK-001] and [GOV-LAW-001]. The International AI Safety Report (Bengio et al., 2025) is an example of a multi-author assessment of risks produced for policymakers; this article does not summarize its findings. The speculative scenarios are not implemented anywhere and have no deployed applications.

## 7. Advantages, Limitations & Unresolved Questions
**Value of the exercise.** Long-run analysis structures priorities and identifies assumptions that policy depends on.

**Limitations.**
- **Weak evidence base.** The more consequential a claim is for the far future, the less empirical support exists for it.
- **Premise sensitivity.** Conclusions of the opportunity-cost and vulnerable-world arguments depend on moral and empirical premises that reasonable people reject.
- **Selection of sources.** The sources cited are largely from a small research community concerned with existential risk and long-term ethics; critical perspectives are underrepresented in this draft. [VERIFICATION NEEDED: verified critical sources on long-termism and singularity scenarios]
- **Time horizon.** Scenarios spanning centuries cannot be validated against data.

**Unresolved questions.** Whether advanced AI shifts economic power toward capital or labor; whether societies can maintain control of institutions as AI substitutes for human participation (Kulveit et al., 2025); how to weigh the far future; whether any threshold of technology exists beyond which civilization is unstable (Bostrom, 2019).

## 8. Current State of Knowledge
As of 2026-10-15: past-automation results are well documented, though their interpretation varies **[Established science / history]**. The philosophical arguments exist in published form and are debated **[Contested]**. The speculative scenarios have no empirical confirmation **[Hypothetical]**. No source cited here shows that civilization-scale effects from advanced AI have occurred. For survey evidence on expert beliefs see Grace et al. (2024), which reports views of 2,778 researchers; the survey is opinion, not evidence of outcomes, and its figures are not reproduced here (see [AGI-TIME-001]).

## 9. Future Trajectory & Strategic Outlook
No registered forecasts are made. Near term (0 to 3 years): continued economic measurement and policy debate, which are observable trends but whose conclusions are not predicted here. Medium term (3 to 10 years) and long term (10+ years): the scenarios in 4.3 are the principal speculative candidates. Each is **[Hypothetical]**. Anyone converting these into forecasts should follow the record format in the Evidential Framework, including an operational definition and review date; none is attempted here. See [FCST-SCEN-001] for scenario-based treatment.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[ASI-001] Artificial Superintelligence](../vol06_asi/ASI-001_artificial_superintelligence.md): Capability assumptions behind the scenarios.
  - [[AGI-001] Artificial General Intelligence](../vol05_agi/AGI-001_artificial_general_intelligence.md): Preceding capability threshold.
- **Related Methods & Technologies**:
  - [[ASI-EXP-001] Intelligence Explosion](../vol06_asi/ASI-EXP-001_intelligence_explosion.md): The self-improvement mechanism.
  - [[SAFE-XRISK-001] Existential Risk from AI](../vol08_safety_governance/SAFE-XRISK-001_existential_risk.md): Risk framing.
  - [[SAFE-ALIGN-001] AI Alignment](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md): Technical approaches to control.
  - [[ECON-LABOR-001] AI, Productivity and Labor Markets](../vol09_economy/ECON-LABOR-001_ai_productivity_and_labor_markets.md): Economic analysis.
- **Advanced Implications & Future Trajectories**:
  - [[PHIL-MIND-001] Moral Status and Personhood of AI](./PHIL-MIND-001_moral_status_and_personhood_of_ai.md): Digital minds.
  - [[PHIL-COSMOS-001] Computational Limits and AI Beyond Earth](./PHIL-COSMOS-001_computational_limits_and_ai_beyond_earth.md): Physical limits on long-run scenarios.
  - [[FCST-SCEN-001] Medium- and Long-Term Scenarios](../vol12_forecasts/FCST-SCEN-001_medium_and_long_term_scenarios.md): Scenario forecasting.
  - [[AGI-TIME-001] AGI Forecasting](../vol05_agi/AGI-TIME-001_agi_forecasting.md): Expert timelines.

## 11. References & Foundational Literature
1. Acemoglu, D., & Restrepo, P. (2019). *Automation and New Tasks: How Technology Displaces and Reinstates Labor*. Journal of Economic Perspectives, 33(2), 3-30. https://doi.org/10.1257/jep.33.2.3. [Task-based framework with displacement and reinstatement effects.]
2. Autor, D. H. (2015). *Why Are There Still So Many Jobs? The History and Future of Workplace Automation*. Journal of Economic Perspectives, 29(3), 3-30. https://doi.org/10.1257/jep.29.3.3. [Why automation has not eliminated most jobs.]
3. Bengio, Y., Mindermann, S., Privitera, D., et al. (2025). *International AI Safety Report* (96 authors). arXiv:2501.17805. [Example of a multi-author risk assessment; not summarized.]
4. Bostrom, N. (2003). *Astronomical Waste: The Opportunity Cost of Delayed Technological Development*. Utilitas, 15(3), 308-314. https://doi.org/10.1017/S0953820800004076. [Opportunity-cost argument.]
5. Bostrom, N. (2014). *Superintelligence: Paths, Dangers, Strategies*. Oxford University Press. ISBN 978-0199678112. [Systematic argument about superintelligence.]
6. Bostrom, N. (2019). *The Vulnerable World Hypothesis*. Global Policy, 10(4), 455-476. https://doi.org/10.1111/1758-5899.12718. [Concept of a vulnerable world.]
7. Good, I. J. (1966). *Speculations Concerning the First Ultraintelligent Machine*. Advances in Computers, 31-88. https://doi.org/10.1016/S0065-2458(08)60418-0. [Early intelligence-explosion speculation.]
8. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B., Brauner, J., & Korzekwa, R. C. (2024). *Thousands of AI Authors on the Future of AI*. arXiv:2401.02843. [Survey of 2,778 researchers; opinion.]
9. Hanson, R. (2016). *The Age of Em*. Oxford University Press. ISBN 978-0198754626. [Scenario analysis of an economy of brain emulations; book not re-read.]
10. Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025). *Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development*. arXiv:2501.16946. [Conceptual argument; preprint.]
11. MacAskill, W. (2022). *What We Owe the Future*. Basic Books. ISBN 978-1541618626. [Book on long-term ethics; arguments not re-read.]
12. Ord, T. (2020). *The Precipice: Existential Risk and the Future of Humanity*. Hachette Books. ISBN 978-0316484916. [Book on existential risk; arguments not re-read.]
13. Russell, S. (2019). *Human Compatible: Artificial Intelligence and the Problem of Control*. Viking. ISBN 978-0525558613. [Argument about the design of AI objectives.]
14. Shulman, C., & Bostrom, N. (2021). Sharing the World with Digital Minds. In *Rethinking Moral Status*. Oxford University Press. https://doi.org/10.1093/oso/9780192894076.003.0018. [Moral and policy issues of digital minds.]
15. Vinge, V. (1993). *The coming technological singularity: How to survive in the post-human era*. NASA Lewis Research Center, Vision 21 symposium. https://ntrs.nasa.gov/citations/19940022856. [Singularity essay.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Past automation has not eliminated most jobs; it changes job types and pay | Well Supported (economic analysis) | Autor (2015) | Moderate to high for past periods |
| Task-based model of displacement and reinstatement | Well Supported (published framework) | Acemoglu & Restrepo (2019) | Moderate to high |
| Delay has a very large opportunity cost, so safety outweighs speed | Contested (philosophical argument) | Bostrom (2003) | Premise-dependent |
| A vulnerable-world typology | Contested (analytic framework) | Bostrom (2019) | Premise-dependent |
| Gradual disempowerment | Hypothetical (conceptual preprint) | Kulveit et al. (2025) | Low; untested |
| Intelligence explosion / singularity | Hypothetical | Good (1966); Vinge (1993) | Not empirical |
| Emulation economy | Hypothetical | Hanson (2016) | Not empirical |
| Expert opinion on AI progress | Preliminary (survey of opinion) | Grace et al. (2024) | Moderate; not outcome evidence |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.