Early AI Development, 1950s–1960s

Early Symbolic AI (often termed Good Old-Fashioned AI, or GOFAI) was the dominant paradigm in artificial intelligence from the late 1950s through the 1980s. It was based on the Physical Symbol System Hypothesis, which asserted that intelligence is achieved through the formal manipulation of discrete symbolic representations using logical rules and heuristic search. Concurrent with early symbolic AI was the development of the Perceptron (1958), a biologically inspired, connectionist model that learned from data through iterative weight updates. The intellectual collision and subsequent funding wars between these two paradigms defined the early architecture of the AI discipline.

The history of early AI is the history of a profound philosophical and engineering schism.

1. The Symbolic Approach (Top-Down): Posited that human thought is essentially deductive logic. It attempted to program intelligence directly by giving machines explicit rules and facts about the world. It succeeded brilliantly at closed, highly structured tasks (algebra, chess, theorem proving) but failed completely at perception, learning, and navigating ambiguity.
2. The Connectionist Approach (Bottom-Up): Posited that intelligence emerges from the distributed connections of simple processing units (neurons). It attempted to teach machines to learn from raw data without explicit rules. It showed promise in pattern recognition but suffered from severe mathematical constraints and inadequate computing hardware.

The temporary victory of the Symbolic approach and the marginalization of the Perceptron in the 1970s delayed the widespread adoption of modern machine learning by decades, serving as a critical lesson in the danger of institutional orthodoxies in scientific research.

#3.1 The Triumph of Symbolic AI

Following the 1956 Dartmouth Workshop, researchers like Allen Newell and Herbert Simon at Carnegie Mellon, and John McCarthy and Marvin Minsky at MIT, drove rapid progress in symbolic reasoning. In 1959, Newell, Shaw and Simon described the General Problem Solver (GPS), a successor to the Logic Theorist (Newell, Shaw & Simon, 1959). GPS formalized Means-Ends Analysis—an algorithm that evaluates the difference between the current state and the goal state, and applies operators to reduce that difference.

The symbolic community achieved remarkable early victories:

• SHRDLU (c. 1970): Terry Winograd’s program could respond to typed English commands and questions about a small simulated blocks world (Winograd, 1972); it worked only within that restricted domain and vocabulary. [VERIFICATION NEEDED: exact 1970 date]
• Physical Symbol System Hypothesis: In their Turing Award lecture (published 1976), Newell and Simon stated the hypothesis that a physical symbol system has the necessary and sufficient means for general intelligent action (Newell & Simon, 1976) [VERIFICATION NEEDED: wording checked only via secondary sources].

#3.2 The Rise of the Perceptron

Simultaneously, at the Cornell Aeronautical Laboratory, psychologist Frank Rosenblatt developed a radically different system: the Perceptron (1958). The Perceptron was an electromechanical device designed for pattern recognition; the funding details (reported as US Office of Naval Research support) are not verified here [VERIFICATION NEEDED]. Instead of being programmed with rules, the Perceptron was fed inputs (from an array of photocells). It calculated a weighted sum of these inputs; if the sum exceeded a threshold, the unit fired. Crucially, Rosenblatt developed the Perceptron Learning Rule, which iteratively adjusted the weights based on the error of its output, allowing the machine to learn to classify simple patterns from examples. [VERIFICATION NEEDED: the specific example in Rosenblatt's reported demonstrations was distinguishing cards marked on the left from those marked on the right, per secondary sources.]

A New York Times report of 8 July 1958 on a Navy demonstration described the device as the embryo of an electronic computer that the Navy expected would "walk, talk, see, write, reproduce itself and be conscious of its existence" (New York Times, 1958); secondary sources differ on how far these expectations were Rosenblatt's own words.

#3.3 The Collision and Perceptrons (1969)

Marvin Minsky and Seymour Papert at MIT published a critical analysis of perceptrons; historians debate their motives and the book's influence (see Section 7).

In 1969, Minsky and Papert published the book Perceptrons. Through rigorous mathematical proofs, they demonstrated the severe limitations of single-layer perceptrons. The most famous proof showed that a single-layer perceptron was mathematically incapable of calculating the XOR (exclusive-or) function because the function is not linearly separable.

While Minsky and Papert acknowledged that multi-layer perceptrons might overcome these limits, they expressed doubt that an effective learning algorithm for multi-layer networks would be found [VERIFICATION NEEDED: wording and page in Minsky & Papert, 1969, not checked].

#4.1 Heuristic Search and State Spaces (Symbolic AI)

Symbolic AI conceptualizes problems as a graph of nodes (states). Solving a problem means searching for a path from the initial state to the goal state. Because brute-force search suffers from combinatorial explosion (exponential scaling), symbolic AI relies on heuristics—rules of thumb that estimate the cheapest path to the goal, significantly pruning the search tree.

#4.2 Linear Separability and the XOR Problem

A single-layer perceptron calculates an output $y$ based on inputs $x$, weights $w$, and a threshold $\theta$:

Geometrically, this equation defines a single hyperplane (a straight line in 2D) that separates inputs into two classes. The XOR problem requires inputs (0,0) and (1,1) to be Class A, and inputs (0,1) and (1,0) to be Class B. It is geometrically impossible to draw a single straight line that separates these classes, hence proving the single-layer perceptron's inability to learn XOR.

• Symbolic Reasoning: Rule-based deduction, Means-Ends Analysis, heuristic state-space search, Lisp programming.
• Connectionism (Early): Single-layer threshold logic units, Hebbian learning, the Perceptron learning rule.

• Symbolic Success: Deep Blue's 1997 match win over Garry Kasparov is often cited as a late success of search-based game-playing methods (alpha-beta search with heuristic evaluation on custom hardware; Campbell, Hoane & Hsu, 2002), although it post-dates the era covered here.
• Perceptron Legacy: While the single-layer hardware Perceptron was a dead end, its error-correcting weight-update rule is an early example of learning by iterative weight adjustment; modern optimisers descend more directly from gradient-based methods (e.g., Widrow-Hoff) [VERIFICATION NEEDED: lineage not checked against a source].

#Limitations of Early Symbolic AI

• The Knowledge Acquisition Bottleneck: Symbolic systems require humans to manually input all rules. Much of human knowledge is tacit (e.g., how to balance a bicycle) and cannot be written as explicit logical propositions.
• Moravec's Paradox: Hans Moravec observed that tasks humans find hard (e.g., chess, theorem proving) proved easier for early programs than perception and mobility, which humans find easy (Moravec, 1988). The explanation offered here, that perception relies on parallel, noisy sensory data hard to capture in explicit symbols, is one common interpretation.

#Limitations of the Minsky-Papert Critique

Perceptrons (1969) is highly controversial historically. While mathematically impeccable regarding single-layer networks, the pessimism it cast over multi-layer networks proved unwarranted in practice. Multi-layer networks can represent XOR without backpropagation; what backpropagation provided was a practical way to learn the weights. It was popularised for learning multi-layer representations in the mid-1980s (Rumelhart et al., 1986), after earlier derivations by others (not re-verified here); the deep learning era is dated later (see HIST-005).

Historians differ on the book's role: some read it as a rigorous check on inflated claims, others as a cause of reduced support for neural-network research. Historians argue that it contributed to a decline of funding and interest in neural-network research through the 1970s, although the extent and causes are debated (Nilsson, 2009 [VERIFICATION NEEDED: page]; Olazaran, 1996).

Symbolic AI has been largely displaced by deep learning as the dominant paradigm. Some researchers advocate a synthesis, drawing on the System 1 / System 2 distinction popularised by Kahneman (2011) as an analogy, and argue that neural networks struggle with explicit logical deduction (a contested claim; no survey is cited here). The pursuit of Neuro-Symbolic Integration seeks to combine the pattern recognition of the Perceptron with the logical rigor of the General Problem Solver.

• Foundational Concepts:
  • FOUND-003 Knowledge Representation: The logic structures powering GOFAI.
  • FOUND-004 Computational Complexity in AI: The combinatorial explosion that ultimately defeated GOFAI search algorithms.
• Related Methods & Technologies:
  • TECH-DL-001 Deep Learning: The multi-layer successor to the Perceptron.
• Advanced Implications & Future Trajectories:
  • HIST-004 The AI Winters: Causes and Lessons: The funding collapse that followed the limits of early Symbolic AI.

Computer game playing is the study and construction of programs that play board and card games, treated in AI as a source of well-defined problems with measurable outcomes. Its history runs from the first proposals for chess programs in 1950 through the defeat of a reigning chess world champion in 1997 to programs that learn Go, chess and shogi through self-play.

Games provide closed environments with explicit rules, clear win conditions and human reference opponents, so progress can be measured directly. Early AI researchers regarded chess as a test of machine intelligence; later milestones, such as AlphaGo, were used to demonstrate new learning methods. Game results are easy to communicate, but they are also narrow: a program that is superhuman at a game does not thereby possess general intelligence, a point stressed in the articles on general-purpose systems (AGI-001 Artificial General Intelligence).

The history shows two successive technical regimes. In the first, strength came from hand-designed evaluation functions and brute-force search with specialised hardware. In the second, strength came from learned evaluation and policy functions, trained with reinforcement learning, combined with search. The second regime connects to TECH-RL-001 Reinforcement Learning and to the deep-learning developments in HIST-005 The Deep Learning Breakthrough (2012).

#3.1 Foundations (1950-1960s)

Shannon (1950) described how a computer could be programmed to play chess: represent the position, define an evaluation function that scores positions, and search a game tree of possible moves using the minimax principle. He distinguished searching all moves to a fixed depth from selective search, and discussed the limits imposed by the size of the game tree. Samuel (1959) reported a checkers program that improved its play through machine learning, using stored-position learning ("rote learning") and adjustment of evaluation parameters through self-play. It is a frequently cited early demonstration of a program that learned to play better than the person who wrote it. [VERIFICATION NEEDED: the exact strength claims for Samuel's program were not checked against the paper's text and are not asserted here.]

#3.2 Chess programs and specialised hardware

Through the 1970s to the 1990s, chess programs improved with faster hardware, deeper search and refined evaluation functions. Campbell, Hoane, and Hsu (2002) describe the IBM Deep Blue system, which combined a parallel architecture with custom chess chips and a search guided by an expert-designed evaluation function. According to IBM's historical account, Deep Blue lost a six-game match to world champion Garry Kasparov 4-2 in Philadelphia in February 1996, winning one game, and won a rematch in New York in May 1997 by 3.5-2.5 (IBM, n.d.). IBM's account describes the 1997 result as a win over the reigning world champion in a six-game match; this article does not assert additional priority claims. The match remains discussed: Kasparov disputed aspects of it, and the sources cited here describe the system and the result but do not adjudicate those disputes. Deep Blue was a special-purpose system and did not learn from games.

#3.3 Solving games

Schaeffer et al. (2007) reported that checkers is solved: with perfect play by both sides the game is a draw. The team had worked on the problem since 1989, and the paper gives the size of the game as about $5 \times 10^{20}$ positions. Solving a game replaces heuristics with a proof of the game-theoretic value. Chess and Go have not been solved.

#3.4 Learning from self-play: backgammon

Tesauro (1995) described TD-Gammon, a neural network trained by temporal-difference learning from self-play to evaluate backgammon positions. It showed that a learned evaluation function, without human-written strategy knowledge, could reach high levels of play and influenced later work in reinforcement learning. [VERIFICATION NEEDED: the strength level reported for TD-Gammon was not checked against the source text.]

#3.5 Monte Carlo tree search and Go

Go resisted the methods that worked for chess because its branching factor is larger and positions are hard to evaluate. Browne et al. (2012) survey Monte Carlo tree search (MCTS), a family of algorithms that grow a search tree guided by random simulations of games to the end; MCTS programs raised the strength of computer Go substantially in the years before deep learning was added.

#3.6 AlphaGo and successors

Silver et al. (2016) introduced AlphaGo, which combined "policy networks" to select moves and "value networks" to evaluate positions with MCTS. The networks were trained by supervised learning from human expert games and then by reinforcement learning from self-play. The paper reports that AlphaGo won 99.8% of its games against other Go programs and defeated the European champion (a professional player) 5 games to 0, the first time a program had defeated a human professional in full-sized Go. In March 2016 AlphaGo played Lee Sedol in Seoul and won 4-1, according to DeepMind (DeepMind, n.d.); that match is not reported in the 2016 paper. Silver et al. (2017) then reported AlphaGo Zero, trained by reinforcement learning from self-play alone with no human game data. Silver et al. (2018) generalised the method as AlphaZero, which, given only the rules and starting from random play, "convincingly defeated a world champion program" in chess, shogi and Go. Schrittwieser et al. (2020) described MuZero, which plans with a learned model and reached the performance of AlphaZero on Go, chess and shogi without being given the game rules, and achieved state-of-the-art results on 57 Atari games.

#3.7 Imperfect information

Poker presents hidden information, which breaks perfect-information search. Brown and Sandholm (2018) reported Libratus, which defeated four top professionals over 120,000 hands of heads-up no-limit Texas hold'em; Brown and Sandholm (2019) reported Pluribus, shown to be stronger than top human professionals in six-player no-limit hold'em.

#4.1 Minimax and evaluation

For a two-player zero-sum game, the value of a position $s$ to the player to move, searched to depth $d$ with evaluation function $e$, is

Shannon (1950) set out this scheme. Practical programs add alpha-beta pruning, which skips branches that cannot affect the result, to search deeper with the same effort. [VERIFICATION NEEDED: alpha-beta pruning is standard textbook material but is not specifically sourced in this article's reference list.]

#4.2 Temporal-difference learning

In TD learning, a value estimate $V_\theta(s_t)$ is updated towards the next estimate: $\theta \leftarrow \theta + \alpha\,[r_{t+1} + V_\theta(s_{t+1}) - V_\theta(s_t)]\,\nabla_\theta V_\theta(s_t)$. Self-play generates training data without human labels (Tesauro, 1995).

#4.3 Monte Carlo tree search with learned networks

MCTS repeats four steps: select a path through the tree, expand a node, evaluate it (by a rollout or a value network), and back up the result. In AlphaGo the policy network biases which moves are explored and the value network evaluates leaves (Silver et al., 2016; Browne et al., 2012). In AlphaGo Zero and AlphaZero a single network supplies both and is trained to match the search's improved move choices and the game outcomes (Silver et al., 2017; Silver et al., 2018).

Game-playing methods were developed for games, and this article does not assert that the specific systems were deployed elsewhere. The techniques (search, reinforcement learning, self-play, learned value functions) have influenced other research areas; for example, reinforcement learning from human feedback and agent research draw on this lineage, as discussed in TECH-RL-001 Reinforcement Learning.

Strengths of games as benchmarks. Clear objectives, measurable results and the possibility of unlimited self-play data.

Limitations.

• Narrowness: each system is specialised to one game or family (AlphaZero and MuZero extend the approach to several games but not to open-ended tasks; Silver et al., 2018; Schrittwieser et al., 2020).
• Sample cost: the self-play systems cited used very large amounts of computation; the papers report this and exact figures are in the primary sources and not repeated here.
• Brute-force chess programs offered little insight into human cognition; whether game success tells us about general intelligence is disputed.
• The interpretation of milestones: "defeated a world champion" should be read with the match conditions in mind. Silver et al. (2016) defeated a professional European champion, not the world champion; the Lee Sedol match is the one against a top world-class player and is sourced only to DeepMind's own page here, a developer claim.

Open questions. Games with many players, open-ended rules, natural language negotiation or very long horizons remain harder; the sources here do not cover them in depth.

By the mid-2020s, learned game-playing systems are widely regarded as superhuman at chess, shogi and Go, and strong poker programs have beaten professionals (Silver et al., 2018; Brown & Sandholm, 2018, 2019). This article does not report current rating figures or engine rankings, which change and are better held in the Living Reference (FACT-001 Factbook). See HIST-010 The Transformer and Foundation Model Era, 2017-2022 for the shift in research attention from games to language models.

No forecasts are registered in this article. Observed trend: the self-play and search ideas developed for games have reappeared in work on reasoning models that use additional compute at inference time; see TECH-REASON-001 Reasoning Models and Test-Time Compute. For registered forecasts consult FCST-REG-001 Forecast Register.

• Foundational Concepts:
  • HIST-003 Early Symbolic AI & The Perceptron: search and early game programs.
  • FOUND-012 Game Theory and Decision Theory: minimax and equilibrium concepts.
• Related Methods & Technologies:
  • TECH-RL-001 Reinforcement Learning: TD learning and policy optimisation.
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: later search-and-learning ideas.
• Advanced Implications & Future Trajectories:
  • HIST-005 The Deep Learning Breakthrough (2012): the deep learning context of AlphaGo.
  • AGI-001 Artificial General Intelligence: the contrast between narrow and general ability.

Natural language processing (NLP) is the field concerned with computational methods for analysing, understanding and generating human language. Its history is conventionally divided into rule-based, statistical, neural and pretrained-model phases, each defined by how linguistic knowledge is obtained: written by hand, estimated from corpora, learned as dense representations, or acquired by large-scale pretraining.

Language was among the first domains proposed for machine intelligence, since fluent use of language is a prominent feature of human cognition and a practical need (translation, search, dialogue). NLP has also been the area in which the central methodological shifts of AI have been most visible: from hand-coded grammars to statistics, from statistics to neural networks, and from task-specific networks to general pretrained models that now underlie large language models (TECH-LLM-001 Large Language Models). The phases overlap, and techniques from earlier phases persist in current systems (tokenisation, evaluation metrics, parsing resources).

#3.1 Machine translation and early optimism (1950s-1960s)

Machine translation (MT) was the first major NLP application. Hutchins (2004) describes the Georgetown-IBM experiment of January 1954, an early public demonstration of Russian-to-English translation by computer, which attracted attention and funding. [VERIFICATION NEEDED: the vocabulary and rule counts of the Georgetown-IBM system are widely quoted but were not checked against Hutchins (2004) text and are omitted.] Expectations of rapid progress were not met. The 1966 report of the Automatic Language Processing Advisory Committee (ALPAC), titled Language and Machines, evaluated the state of MT and the need for computational linguistics (ALPAC, 1966), and is widely cited as having contributed to a sharp reduction in US funding for MT. [VERIFICATION NEEDED: the report's specific conclusions and the extent of its funding effect were not checked in this draft and are described only in general terms.] This episode is one of the cases discussed in HIST-004 The AI Winters: Causes and Lessons.

#3.2 Dialogue and language understanding programs (1960s-1970s)

Weizenbaum (1966) described ELIZA, a program that conducted text conversations by matching patterns in the user's input and applying scripted transformation rules. ELIZA is commonly cited as an example of how readily people attribute understanding to conversational software. [VERIFICATION NEEDED: this interpretation is common in secondary literature and was not checked against the text of Weizenbaum (1966).] Winograd (1972) described SHRDLU, a system that understood and acted on English commands in a small simulated world of blocks by integrating syntax, semantics and reasoning about the scene. SHRDLU showed deep integration in a restricted domain but did not scale to open-ended language.

#3.3 The statistical turn (1980s-1990s)

Statistical methods entered NLP through speech recognition and MT. Brown et al. (1990) described a statistical approach to MT in which probabilistic models are estimated from parallel text in two languages. The shift was supported by annotated corpora: Marcus, Santorini, and Marcinkiewicz (1993) described the Penn Treebank, a large corpus annotated with part-of-speech tags and syntactic structure that became a standard resource for training and evaluating parsers. Papineni et al. (2002) introduced BLEU, an automatic metric of MT quality based on n-gram overlap with reference translations, which allowed fast comparison of systems, though it only approximates human judgement. The general machine-learning context is covered in HIST-007 Statistical Machine Learning, 1980s-2000s.

#3.4 Neural language models and word embeddings (2000s-2014)

Bengio et al. (2003) proposed a neural probabilistic language model that learns a distributed representation for each word jointly with the probability function for word sequences, which lets the model generalise across similar words. Collobert et al. (2011) trained a unified neural architecture to perform several tagging tasks with minimal task-specific feature engineering. Mikolov et al. (2013) introduced efficient architectures for learning word vectors from very large corpora and reported state-of-the-art performance on syntactic and semantic word-similarity tests with training taking less than a day on a 1.6-billion-word dataset. Pennington, Socher, and Manning (2014) proposed GloVe, which learns word vectors from global word co-occurrence statistics. The embedding idea and its descendants are described in TECH-TOK-001 Tokenization and Embeddings.

#3.5 Sequence-to-sequence models and attention (2014-2016)

Sutskever, Vinyals, and Le (2014) showed that an encoder-decoder network of recurrent units could map an input sequence to an output sequence, applying it to MT. Bahdanau, Cho, and Bengio (2014) introduced an attention mechanism that lets the decoder consult different source positions while generating each output word, removing the bottleneck of a single fixed-length vector. Wu et al. (2016) described Google's production neural MT system, a deep LSTM encoder-decoder with attention and sub-word units, which moved neural MT from research into widespread service. See TECH-ATTN-001 Attention Mechanisms.

#3.6 Transformers and pretrained models (2017 onward)

Vaswani et al. (2017) proposed the Transformer, built on attention without recurrence. Devlin et al. (2018) introduced BERT, a pretrained bidirectional Transformer fine-tuned for each task, reporting state-of-the-art results on a range of benchmarks. Brown et al. (2020) described GPT-3, a 175-billion-parameter autoregressive language model that performed many tasks from a few prompt examples. This phase is covered in HIST-010 The Transformer and Foundation Model Era, 2017-2022 and subsequent events in HIST-011 The Contemporary AI Era, 2022 onward (living record).

#4.1 The language modelling problem

A language model assigns a probability to a sequence of tokens $w_1,\dots,w_T$. By the chain rule,

N-gram models approximate each conditional by the previous $n-1$ words and estimate it from counts; neural models parameterise the conditional with a network (Bengio et al., 2003), and Transformers condition on the whole context with attention (Vaswani et al., 2017).

#4.2 Statistical translation

The noisy-channel formulation selects the target sentence $e$ that maximises $P(e \mid f) \propto P(f \mid e)\,P(e)$ for source sentence $f$, combining a translation model and a language model (Brown et al., 1990).

#4.3 Attention in sequence models

At output step $t$, an attention-based decoder forms a context vector $c_t=\sum_{i}\alpha_{ti}h_i$ with weights $\alpha_{ti}=\operatorname{softmax}_i\big(\text{score}(s_{t-1},h_i)\big)$ over encoder states $h_i$ (Bahdanau et al., 2014).

#4.4 Evaluation

BLEU compares n-gram overlap between system output and references (Papineni et al., 2002). Reference corpora such as the Penn Treebank (Marcus et al., 1993) supply gold-standard annotations for parsing and tagging.

• Machine translation: from the 1954 demonstration (Hutchins, 2004) to neural MT in production (Wu et al., 2016).
• Conversation: ELIZA (Weizenbaum, 1966) as an early demonstration; current assistants are discussed in HIST-011 The Contemporary AI Era, 2022 onward (living record).
• Parsing and tagging: trained and evaluated on the Penn Treebank (Marcus et al., 1993).
• General-purpose language tasks: question answering, translation and cloze tasks with few-shot prompting (Brown et al., 2020).

What each phase contributed. Rule-based systems showed what could be done in restricted domains. Statistical methods brought data-driven development and standard evaluation. Neural methods removed most feature engineering and supported transfer through shared representations. Pretrained models reduced the need for task-specific labelled data.

Limitations. Rule-based systems did not scale to open language (Winograd, 1972 achieved depth only in a small world). BLEU only approximates human judgement of translation quality (Papineni et al., 2002 introduce it as an approximation). Neural and pretrained models depend on large training corpora, and the sources cited in this article do not address data provenance or bias in depth; see TECH-DATA-001 Training Data, Contamination and Provenance.

Unresolved question. Whether large language models "understand" language or exhibit only statistical pattern use is debated; this article cites no source that settles it, and the question is treated in FOUND-005 Philosophy of Mind.

The dominant approach to most NLP tasks is the pretrained Transformer language model, adapted by fine-tuning or prompting (Devlin et al., 2018; Brown et al., 2020). Classical resources and metrics remain in use alongside it, though benchmark scores and system comparisons change rapidly and are tracked elsewhere (MODEL-001 Model Registry). The division of history into phases is a convenience of historians; contemporary research combines elements of several.

No forecasts are registered in this article. For registered forecasts see FCST-REG-001 Forecast Register. The persistent themes of the history (data availability, evaluation validity, and the relation between statistical competence and understanding) are expected to remain relevant, but this is an observation about the past pattern and not a registered prediction.

• Foundational Concepts:
  • HIST-003 Early Symbolic AI & The Perceptron: symbolic AI from which rule-based NLP derives.
  • HIST-004 The AI Winters: Causes and Lessons: funding cycles including MT.
• Related Methods & Technologies:
  • HIST-007 Statistical Machine Learning, 1980s-2000s: the statistical turn.
  • TECH-TOK-001 Tokenization and Embeddings: word and sub-word representations.
  • TECH-ATTN-001 Attention Mechanisms: attention.
  • TECH-LLM-001 Large Language Models: current language models.
• Advanced Implications & Future Trajectories:
  • HIST-010 The Transformer and Foundation Model Era, 2017-2022: the pretraining era.
  • HIST-013 Computer Vision: A History: the parallel history in vision.