AI Security & GovernanceApril 2026 · 1 min read
LLM Learning & Retrieval Methods — Architecture Reference
LLM Learning & Retrieval Methods — Architecture Reference — practical leadership guidance for security, privacy, and compliance program owners.
By Jim Venuto
Foundational
Self-supervised pretraining
The base stage. The model predicts masked or next tokens across a large, mostly unlabeled corpus, learning general linguistic and world patterns before any task-specific work.
- DataWeb, books, code — trillions of tokens
- SignalNext-token prediction
- CostHighest — weeks on GPU/TPU clusters
- OutputBase / foundation model
flowchart LR A\[("Raw corpus
web · books · code")\] --> B\[Tokenizer\] B --> C\[\["Transformer
randomly initialized"\]\] C --> D{{"Next-token
prediction loss"}} D -- "backprop" --> C C ==> E\[\["Base foundation model"\]\] style A fill:#ebe4d4,stroke:#1a1814,stroke-width:1px,color:#1a1814 style C fill:#1a1814,stroke:#1a1814,color:#f4efe6 style E fill:#8b2c1e,stroke:#1a1814,color:#f4efe6 style D fill:#c89d3a,stroke:#1a1814,color:#1a1814 style B fill:#f4efe6,stroke:#1a1814,color:#1a1814
Tags#smb