AI in Software and Computing

AI in software engineering is the use of machine-learning models, chiefly large language models trained on source code and text, to assist or automate programming tasks: completing code, writing functions from descriptions, explaining and refactoring code, generating tests, reviewing changes and, in agentic form, resolving issues in whole repositories. This article concerns how these tools are used and what has been measured about their effects; the underlying model techniques are in TECH-CODE-001 and the capability benchmarks in CAP-CODE-001.

Software development is an early large-scale setting for generative AI because its outputs can be checked mechanically (compilers, tests) and because developers adopt tools quickly. It is also a setting where empirical productivity studies have been run, and where the results disagree in instructive ways. Three evidence classes are separated throughout:

Benchmarks measure what a model can do on a defined task set; productivity trials measure what happens when people use it; neither alone shows effects on software quality, maintenance cost or delivery outcomes.

Program synthesis and code completion existed before neural language models. The transformer era began for code with Codex, a GPT model fine-tuned on public GitHub code, reported in 2021; a distinct production version powered the first GitHub Copilot, and the paper introduced the HumanEval benchmark (Chen et al., 2021). In 2022 AlphaCode reported an average ranking in the top 54.3% in simulated Codeforces contests (Li et al., 2022). Early field work on Copilot examined perceived productivity (Ziegler et al., 2022), a controlled experiment (Peng et al., 2023) and security of suggestions (Pearce et al., 2022). In 2023 the SWE-bench benchmark moved evaluation from isolated functions to resolving real repository issues (Jimenez et al., 2024), and in 2024 SWE-agent showed that the interface given to a model matters (Yang et al., 2024). Field experiments at firms and a randomised trial on experienced open-source developers followed (Cui et al., 2026; Becker et al., 2025).

Code completion and generation. A language model defines $p_\theta(t_k \mid t_{<k})$ over code and text tokens. Given a prefix and surrounding context, the tool samples or ranks continuations.

Functional-correctness evaluation. Because code can be executed, a standard metric is pass@$k$: the probability that at least one of $k$ sampled programs passes the unit tests. With $n$ samples per problem of which $c$ pass, the unbiased estimator is

Chen et al. (2021) reported that a Codex model solved 28.8% of HumanEval problems with a single sample, and 70.2% with 100 samples per problem, illustrating how sampling plus verification can raise success when tests exist.

Repository-level agents. An agent receives an issue, explores files, edits code, runs tests and iterates. SWE-bench frames this as: given a codebase and an issue description, produce a patch that makes hidden tests pass (Jimenez et al., 2024). SWE-agent reported that a custom agent-computer interface improved performance on that benchmark (Yang et al., 2024).

Why scores change quickly. The original SWE-bench paper reported that the best model resolved 1.96% of issues (Jimenez et al., 2024). Scores on such benchmarks have since changed by large factors as models and scaffolds improved, so current numbers are not recorded here; use the Living Reference collection MODEL-001 and CAP-CODE-001. Benchmark saturation, test-suite weakness and contamination of public repositories in training data are standing concerns (see CAP-BENCH-001).

• Inline completion: low-latency suggestions in an editor.
• Chat and explanation assistants: question answering about code, documentation, refactoring.
• Test and review assistants: generating unit tests, flagging defects in pull requests.
• Agentic coding: multi-step systems operating on a repository or terminal (see TECH-AGENT-001 and CAP-AGENTEVAL-001).
• Competition-style synthesis: large-scale sampling and filtering for contest problems (Li et al., 2022); a research setting rather than everyday practice.

#6.1 Deployed products

Commercial assistants are widely used by professional developers. The first major product was GitHub Copilot, powered by a production version of Codex (Chen et al., 2021). Product names and adoption statistics change often and are vendor-reported; they are not tabulated here.

#6.2 Validated results (experiments and field studies)

These results are not contradictory once settings are compared: the first is a short greenfield task; the second a large multi-firm study with an outcome of completed tasks; the third a small trial of experienced maintainers working on their own large codebases. The third study is small (16 developers); its authors examine 20 candidate explanations for the slowdown and caution that experimental artifacts cannot be entirely ruled out, and the result should not be read as applying to all software work. Each measures a different construct, and none measures defect rates or long-term maintenance cost.

#6.3 Security evidence

• Prompting Copilot in 89 scenarios relevant to high-risk weakness categories produced 1,689 programs, of which approximately 40% were found vulnerable (Pearce et al., 2022). This tests an early (2021) version of the tool.
• In a user study, participants with access to an assistant based on a Codex model wrote significantly less secure code than those without it, and were more likely to believe their code was secure (Perry et al., 2023).

#6.4 Research demonstrations

Competition-level synthesis (Li et al., 2022), HumanEval results and SWE-bench results are demonstrations of capability under benchmark conditions. They do not by themselves predict productivity effects in practice.

Advantages. Faster completion of well-specified, bounded tasks (Peng et al., 2023); larger measured gains for less experienced developers (Cui et al., 2026); executable verification makes some outputs checkable.

Limitations.

• Productivity is task- and population-dependent. An RCT with experienced developers working in familiar large codebases found slower completion with AI, even though the developers believed AI had sped them up (Becker et al., 2025). Self-reported productivity can diverge from measured time.
• Security and over-trust. Evidence from 2021-22 assistants suggests insecure suggestions and misplaced confidence (Pearce et al., 2022; Perry et al., 2023). These results predate current tools and should be re-measured, not assumed unchanged.
• Benchmark validity. Hidden-test success is not equivalent to a mergeable, maintainable change; test suites can be incomplete.
• Evidence gaps. Few studies follow code quality, incident rates or maintenance burden over time [VERIFICATION NEEDED: no such long-run study was confirmed during drafting].

Unresolved questions. How effects scale with model capability and agentic autonomy; whether novice gains reflect learning or dependence (cf. APP-EDU-001); how review workload shifts to humans; what happens to demand for different software roles (see ECON-LABOR-001).

As of 2026-10-15: AI coding assistants are in routine professional use (Well Supported, adoption figures vendor-reported). The sign and size of productivity effects depend on task type, developer experience and tool generation (Contested for experienced developers on large familiar codebases; Well Supported that gains appear on short, well-specified tasks). Published evidence on security is dated and mixed with respect to current tools (Preliminary). Capability benchmarks have advanced rapidly; any figure quoted without a date should be treated as stale.

Near-term (0-3 years): expansion of agentic workflows with human review; better-designed trials measuring defects and maintenance. Medium-term (3-10 years): shifts in task mix and review practice; open questions on verification at scale. Long-term claims about full automation of software engineering are speculative (see AGI-PATH-001 for the broader discussion).

• Foundational Concepts:
  • TECH-CODE-001 AI Code Generation: Model techniques for code.
  • TECH-LLM-001 Large Language Models: The model class behind assistants.
• Related Methods & Technologies:
  • TECH-AGENT-001 Autonomous AI Agents: Agentic coding systems.
  • CAP-CODE-001 Software Engineering Capability and Evaluation: Benchmarks and capability measurement.
  • CAP-BENCH-001 AI Evaluation: Benchmark validity and contamination.
  • APP-BIZ-001 AI in Business and Finance: Field experiments in other knowledge work.
• Advanced Implications & Future Trajectories:
  • ECON-LABOR-001 AI, Productivity and Labor Markets: Labour-market implications.