---
id: HIST-012
title: "Natural Language Processing: A History"
alternative_names:
  - "History of Computational Linguistics"
  - "History of Language Technology"
category: "History → Language"
volume: II
chapter: 8, 13
entry_type: "Historical Record"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "HIST-003"
related_entries:
  - "HIST-007"
  - "HIST-010"
  - "TECH-LLM-001"
  - "TECH-ATTN-001"
  - "TECH-TOK-001"
---

# Natural Language Processing: A History

## 1. Concise Definition
**Natural language processing (NLP)** is the field concerned with computational methods for analysing, understanding and generating human language. Its history is conventionally divided into rule-based, statistical, neural and pretrained-model phases, each defined by how linguistic knowledge is obtained: written by hand, estimated from corpora, learned as dense representations, or acquired by large-scale pretraining.

## 2. Overview & Significance
Language was among the first domains proposed for machine intelligence, since fluent use of language is a prominent feature of human cognition and a practical need (translation, search, dialogue). NLP has also been the area in which the central methodological shifts of AI have been most visible: from hand-coded grammars to statistics, from statistics to neural networks, and from task-specific networks to general pretrained models that now underlie large language models ([[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md)). The phases overlap, and techniques from earlier phases persist in current systems (tokenisation, evaluation metrics, parsing resources).

## 3. Historical Development

### 3.1 Machine translation and early optimism (1950s-1960s)
Machine translation (MT) was the first major NLP application. Hutchins (2004) describes the Georgetown-IBM experiment of January 1954, an early public demonstration of Russian-to-English translation by computer, which attracted attention and funding. [VERIFICATION NEEDED: the vocabulary and rule counts of the Georgetown-IBM system are widely quoted but were not checked against Hutchins (2004) text and are omitted.] Expectations of rapid progress were not met. The 1966 report of the Automatic Language Processing Advisory Committee (ALPAC), titled *Language and Machines*, evaluated the state of MT and the need for computational linguistics (ALPAC, 1966), and is widely cited as having contributed to a sharp reduction in US funding for MT. [VERIFICATION NEEDED: the report's specific conclusions and the extent of its funding effect were not checked in this draft and are described only in general terms.] This episode is one of the cases discussed in [[HIST-004] The AI Winters: Causes and Lessons](./HIST-004_ai_winters.md).

### 3.2 Dialogue and language understanding programs (1960s-1970s)
Weizenbaum (1966) described ELIZA, a program that conducted text conversations by matching patterns in the user's input and applying scripted transformation rules. ELIZA is commonly cited as an example of how readily people attribute understanding to conversational software. [VERIFICATION NEEDED: this interpretation is common in secondary literature and was not checked against the text of Weizenbaum (1966).] Winograd (1972) described SHRDLU, a system that understood and acted on English commands in a small simulated world of blocks by integrating syntax, semantics and reasoning about the scene. SHRDLU showed deep integration in a restricted domain but did not scale to open-ended language.

### 3.3 The statistical turn (1980s-1990s)
Statistical methods entered NLP through speech recognition and MT. Brown et al. (1990) described a statistical approach to MT in which probabilistic models are estimated from parallel text in two languages. The shift was supported by annotated corpora: Marcus, Santorini, and Marcinkiewicz (1993) described the Penn Treebank, a large corpus annotated with part-of-speech tags and syntactic structure that became a standard resource for training and evaluating parsers. Papineni et al. (2002) introduced BLEU, an automatic metric of MT quality based on n-gram overlap with reference translations, which allowed fast comparison of systems, though it only approximates human judgement. The general machine-learning context is covered in [[HIST-007] Statistical Machine Learning, 1980s-2000s](./HIST-007_statistical_machine_learning_1980s_2000s.md).

### 3.4 Neural language models and word embeddings (2000s-2014)
Bengio et al. (2003) proposed a neural probabilistic language model that learns a distributed representation for each word jointly with the probability function for word sequences, which lets the model generalise across similar words. Collobert et al. (2011) trained a unified neural architecture to perform several tagging tasks with minimal task-specific feature engineering. Mikolov et al. (2013) introduced efficient architectures for learning word vectors from very large corpora and reported state-of-the-art performance on syntactic and semantic word-similarity tests with training taking less than a day on a 1.6-billion-word dataset. Pennington, Socher, and Manning (2014) proposed GloVe, which learns word vectors from global word co-occurrence statistics. The embedding idea and its descendants are described in [[TECH-TOK-001] Tokenization and Embeddings](../vol03_technical/TECH-TOK-001_tokenization_and_embeddings.md).

### 3.5 Sequence-to-sequence models and attention (2014-2016)
Sutskever, Vinyals, and Le (2014) showed that an encoder-decoder network of recurrent units could map an input sequence to an output sequence, applying it to MT. Bahdanau, Cho, and Bengio (2014) introduced an attention mechanism that lets the decoder consult different source positions while generating each output word, removing the bottleneck of a single fixed-length vector. Wu et al. (2016) described Google's production neural MT system, a deep LSTM encoder-decoder with attention and sub-word units, which moved neural MT from research into widespread service. See [[TECH-ATTN-001] Attention Mechanisms](../vol03_technical/TECH-ATTN-001_attention_mechanisms.md).

### 3.6 Transformers and pretrained models (2017 onward)
Vaswani et al. (2017) proposed the Transformer, built on attention without recurrence. Devlin et al. (2018) introduced BERT, a pretrained bidirectional Transformer fine-tuned for each task, reporting state-of-the-art results on a range of benchmarks. Brown et al. (2020) described GPT-3, a 175-billion-parameter autoregressive language model that performed many tasks from a few prompt examples. This phase is covered in [[HIST-010] The Transformer and Foundation Model Era, 2017-2022](./HIST-010_the_transformer_and_foundation_model_era_2017_2022.md) and subsequent events in [[HIST-011] The Contemporary AI Era, 2022 onward (living record)](./HIST-011_the_contemporary_ai_era_2022_onward_living_record.md).

## 4. Technical / Conceptual Explanation

### 4.1 The language modelling problem
A language model assigns a probability to a sequence of tokens $w_1,\dots,w_T$. By the chain rule,

$$P(w_1,\dots,w_T)=\prod_{t=1}^{T} P(w_t \mid w_1,\dots,w_{t-1}).$$

N-gram models approximate each conditional by the previous $n-1$ words and estimate it from counts; neural models parameterise the conditional with a network (Bengio et al., 2003), and Transformers condition on the whole context with attention (Vaswani et al., 2017).

### 4.2 Statistical translation
The noisy-channel formulation selects the target sentence $e$ that maximises $P(e \mid f) \propto P(f \mid e)\,P(e)$ for source sentence $f$, combining a translation model and a language model (Brown et al., 1990).

### 4.3 Attention in sequence models
At output step $t$, an attention-based decoder forms a context vector $c_t=\sum_{i}\alpha_{ti}h_i$ with weights $\alpha_{ti}=\operatorname{softmax}_i\big(\text{score}(s_{t-1},h_i)\big)$ over encoder states $h_i$ (Bahdanau et al., 2014).

### 4.4 Evaluation
BLEU compares n-gram overlap between system output and references (Papineni et al., 2002). Reference corpora such as the Penn Treebank (Marcus et al., 1993) supply gold-standard annotations for parsing and tagging.

## 5. Types, Components & Classifications
| Phase | Source of linguistic knowledge | Representative work |
| :--- | :--- | :--- |
| Rule-based | Hand-written rules and scripts | Georgetown-IBM (Hutchins, 2004); ELIZA (Weizenbaum, 1966); SHRDLU (Winograd, 1972) |
| Statistical | Probabilities estimated from corpora | Brown et al. (1990); Penn Treebank (Marcus et al., 1993) |
| Neural | Learned dense representations | Bengio et al. (2003); Collobert et al. (2011); Mikolov et al. (2013); Pennington et al. (2014) |
| Sequence-to-sequence with attention | End-to-end learned mapping | Sutskever et al. (2014); Bahdanau et al. (2014); Wu et al. (2016) |
| Pretrained Transformers | Large-scale self-supervised pretraining | Vaswani et al. (2017); Devlin et al. (2018); Brown et al. (2020) |

## 6. Applications & Concrete Examples
- **Machine translation**: from the 1954 demonstration (Hutchins, 2004) to neural MT in production (Wu et al., 2016).
- **Conversation**: ELIZA (Weizenbaum, 1966) as an early demonstration; current assistants are discussed in [[HIST-011] The Contemporary AI Era, 2022 onward (living record)](./HIST-011_the_contemporary_ai_era_2022_onward_living_record.md).
- **Parsing and tagging**: trained and evaluated on the Penn Treebank (Marcus et al., 1993).
- **General-purpose language tasks**: question answering, translation and cloze tasks with few-shot prompting (Brown et al., 2020).

## 7. Advantages, Limitations & Unresolved Questions
**What each phase contributed.** Rule-based systems showed what could be done in restricted domains. Statistical methods brought data-driven development and standard evaluation. Neural methods removed most feature engineering and supported transfer through shared representations. Pretrained models reduced the need for task-specific labelled data.

**Limitations.** Rule-based systems did not scale to open language (Winograd, 1972 achieved depth only in a small world). BLEU only approximates human judgement of translation quality (Papineni et al., 2002 introduce it as an approximation). Neural and pretrained models depend on large training corpora, and the sources cited in this article do not address data provenance or bias in depth; see [[TECH-DATA-001] Training Data, Contamination and Provenance](../vol03_technical/TECH-DATA-001_training_data_contamination_and_provenance.md).

**Unresolved question.** Whether large language models "understand" language or exhibit only statistical pattern use is debated; this article cites no source that settles it, and the question is treated in [[FOUND-005] Philosophy of Mind](../vol01_foundations/FOUND-005_philosophy_of_mind.md).

## 8. Current State of Knowledge
The dominant approach to most NLP tasks is the pretrained Transformer language model, adapted by fine-tuning or prompting (Devlin et al., 2018; Brown et al., 2020). Classical resources and metrics remain in use alongside it, though benchmark scores and system comparisons change rapidly and are tracked elsewhere ([[MODEL-001] Model Registry](../reference/factbook/MODEL-001_model_registry.md)). The division of history into phases is a convenience of historians; contemporary research combines elements of several.

## 9. Future Trajectory & Strategic Outlook
No forecasts are registered in this article. For registered forecasts see [[FCST-REG-001] Forecast Register](../reference/factbook/FCST-REG-001_forecast_register.md). The persistent themes of the history (data availability, evaluation validity, and the relation between statistical competence and understanding) are expected to remain relevant, but this is an observation about the past pattern and not a registered prediction.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[HIST-003] Early Symbolic AI & The Perceptron](./HIST-003_symbolic_ai_perceptron.md): symbolic AI from which rule-based NLP derives.
  - [[HIST-004] The AI Winters: Causes and Lessons](./HIST-004_ai_winters.md): funding cycles including MT.
- **Related Methods & Technologies**:
  - [[HIST-007] Statistical Machine Learning, 1980s-2000s](./HIST-007_statistical_machine_learning_1980s_2000s.md): the statistical turn.
  - [[TECH-TOK-001] Tokenization and Embeddings](../vol03_technical/TECH-TOK-001_tokenization_and_embeddings.md): word and sub-word representations.
  - [[TECH-ATTN-001] Attention Mechanisms](../vol03_technical/TECH-ATTN-001_attention_mechanisms.md): attention.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): current language models.
- **Advanced Implications & Future Trajectories**:
  - [[HIST-010] The Transformer and Foundation Model Era, 2017-2022](./HIST-010_the_transformer_and_foundation_model_era_2017_2022.md): the pretraining era.
  - [[HIST-013] Computer Vision: A History](./HIST-013_computer_vision_a_history.md): the parallel history in vision.

## 11. References & Foundational Literature
1. Hutchins, W. J. (2004). The Georgetown-IBM experiment demonstrated in January 1954. In *Machine Translation: From Real Users to Research (AMTA 2004)*, Lecture Notes in Computer Science, Springer, 102-114. https://doi.org/10.1007/978-3-540-30194-3_12. [Historical account of the 1954 demonstration.]
2. Automatic Language Processing Advisory Committee (ALPAC). (1966). *Language and Machines: Computers in Translation and Linguistics*. National Academy of Sciences, National Research Council. https://doi.org/10.17226/9547. [The ALPAC report on machine translation.]
3. Weizenbaum, J. (1966). ELIZA: A computer program for the study of natural language communication between man and machine. *Communications of the ACM*, 9(1), 36-45. https://doi.org/10.1145/365153.365168. [Describes ELIZA.]
4. Winograd, T. (1972). Understanding natural language. *Cognitive Psychology*, 3(1), 1-191. https://doi.org/10.1016/0010-0285(72)90002-3. [Describes SHRDLU.]
5. Brown, P. F., Cocke, J., Della Pietra, S. A., Della Pietra, V. J., Jelinek, F., Lafferty, J. D., Mercer, R. L., & Roossin, P. S. (1990). A statistical approach to machine translation. *Computational Linguistics*, 16(2), 79-85. https://aclanthology.org/J90-2002/. [Statistical MT.]
6. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn Treebank. *Computational Linguistics*, 19(2), 313-330. https://aclanthology.org/J93-2004/. [Describes the Penn Treebank.]
7. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. *Proceedings of ACL 2002*, 311-318. https://aclanthology.org/P02-1040/. [Introduces BLEU.]
8. Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. *Journal of Machine Learning Research*, 3, 1137-1155. https://www.jmlr.org/papers/v3/bengio03a.html. [Neural language model with learned word representations.]
9. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. *Journal of Machine Learning Research*, 12, 2493-2537. https://jmlr.org/papers/v12/collobert11a.html. [Unified neural architecture for several NLP tasks.]
10. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781. [Word vector architectures and results.]
11. Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. *Proceedings of EMNLP 2014*, 1532-1543. https://aclanthology.org/D14-1162/. [GloVe embeddings.]
12. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. arXiv:1409.3215. [Encoder-decoder LSTM applied to MT.]
13. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. [Attention mechanism for MT.]
14. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., et al. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. [GNMT production system.]
15. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. *NeurIPS 2017*. arXiv:1706.03762. [The Transformer.]
16. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional Transformers for language understanding. arXiv:1810.04805; *NAACL-HLT 2019*, https://aclanthology.org/N19-1423/. [BERT.]
17. Brown, T. B., et al. (2020). Language models are few-shot learners. arXiv:2005.14165. [GPT-3.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Georgetown-IBM demonstration occurred in January 1954 | Well Supported (historical account) | Hutchins (2004) | High for date; details unverified |
| ALPAC report existed and addressed MT | Established | ALPAC (1966) | High; consequences described only generally |
| ELIZA pattern-matching dialogue; SHRDLU in a blocks world | Established | Weizenbaum (1966); Winograd (1972) | High |
| Statistical MT from parallel text by 1990 | Established | Brown et al. (1990) | High |
| Penn Treebank and BLEU as standard resources | Established | Marcus et al. (1993); Papineni et al. (2002) | High |
| Neural language models and word embeddings | Established | Bengio et al. (2003); Mikolov et al. (2013); Pennington et al. (2014); Collobert et al. (2011) | High |
| Seq2seq and attention led to neural MT in production | Well Supported | Sutskever et al. (2014); Bahdanau et al. (2014); Wu et al. (2016) | High |
| Pretrained Transformers dominate NLP | Well Supported (as of the review date) | Vaswani et al. (2017); Devlin et al. (2018); Brown et al. (2020) | Moderate to high |
| Whether LLMs understand language | Contested | none cited | Not rated by the cited sources |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.