July 31, 2026
Chapter 2 of The Hundred-Page Language Models Book is when things started to get really exciting for me.
Chapter 2 – Language Modeling Basics
Chapter 2 introduces the fundamental building blocks of machine learning with natural language input and outputs: tokenization, embeddings, and model evaluation frameworks. Tokenization is the process of converting input words into “tokens” that are recognizable by language models, and embeddings convert those tokens into dense numerical representations. Together, they tackle the problem of efficiently representing arbitrarily large vocabularies in machine-readable formats.
I had heard the word “embeddings”, of course, and I understood, at a high level, that embedding models are something important for LLMs, and are used to capture semantic similarity between words. More than that, I could not have told you. HPLM explains the concepts beautifully:
Imagine that you have a language with a vocabulary of 10000 words. (Note that this is a very small language. The Oxford English Dictionary contains over 500,000 words, and modern LLMs model multiple languages simultaneously.) The most straightforward way to represent a word in this vocabulary is with a one-hot vector, an array of 10000 bits, with each index in the array corresponding to a word in the vocabulary.
One-hot vectors are easy to understand and perfectly machine-readable, but they have two major drawbacks: they are incredibly memory inefficient, wasting most of their space storing useless 0s, and they fail to encode any useful relationships between words. Indices are assigned to words arbitrarily, and words that are very similar (e.g. “happy” and “glad”) may end up far away from each other, whereas words with adjacent indices may have no meaningful semantic relationship.
Word2vec
The primary insight of embeddings is that it is possible to generate an alternative representation of tokens which is both more dense than one-hot encoding and represents semantic relationships between words. HPLM goes into detail on Word2vec, a family of algorithms used to generate word embeddings. Word2vec comes in several flavors, but they all share the same core conceptual idea. Imagine a snippet of text containing a target word (in this case “cat”) and several surrounding context words:
If you were to train a neural network such that, when given the target word as input, it produces the context words with high likelihood, then if you were to pass in a similar word, one that is likely to appear in similar contexts (e.g. “kitten”), you should expect the network to produce similar output. The structure of the network looks like this: two layers, an input layer that scales from the vocabulary size down to the embedding dimension size and an output layer that does the reverse. The output of the first layer is a vector of floats, also called the embedding vector, which can be used as a dense representation of the input word when training or using a language model.
The part that blew my mind
So far so good. It makes intuitive sense that the vectors for “cat” and “kitten” should be similar i.e. close to each other in vector-space. But then, the author casually drops this bomb – these embedding vectors support simple mathematical operations. For example, if you take the embedding vector for “king”, subtract the vector for “man”, and add the vector for “woman”, you get a vector very similar to the one for “queen”.
What?
How?
That sounds completely made up.
I had a little chat with Claude, trying to understand why this might be true. Claude made the argument to me that, within the vectors for “king” and “queen”, there is some subcomponent representing the concept of “royalty”. Honestly, that seems like nonsense to me.
It gets a little clearer when I think of the operations as if they were happening in the space of training examples, or at least I can squint at it and pretend that it makes sense. Imagine all of the context examples for the word “king”. Some of them will include context about gender, many will not. When you subtract the vector for “man”, you’re not removing only the examples that include both “king” and “man”. You are removing all of the context examples for the word “man”, as if the concept of “man” is being removed from your language entirely. That doesn’t land you on some genderless concept of “monarch”; it moves you somewhere really weird.
Now, add in all of the example texts for “woman”. Not only are we adding back in some examples for “queen”, we’re also moving our whole language back from the strange universe in which “man” doesn’t exist.
Ok, no, maybe this doesn’t actually help.
Visualizing embedding vectors
Obsessed with this idea, I decided to vibe-code an app that would let me play around with more of this word vector math. You can try it out here (Word2Vec Arithmetic, GitHub repo). Over a smallish vocabulary of about 1600 words, you can add and subtract them together, and get a list of the closest resulting word vectors from a larger vocabulary of 30K words, along with a visualization of the operation in vector space. Distance is measured by cosine similarity between vectors.
The quality of an embedding model should grow as the model is trained on a larger data set, and I wanted to see if I could visualize this improvement. I trained two embedding models, one on text8, which is the first 100MB of English Wikipedia, and one on enwik9, the first 1GB of English Wikipedia. I also downloaded one pretrained embedding model, Stanford NLP’s 6-billion token GloVe vectors.
So does it work? Well, sort of.
The “king - man + woman” equation returns queen for all three vector models. Interestingly, the result vector computed by the enwik9 model is just a little bit closer to its actual vector for “queen” than the result that comes out of the GloVe model.
The impact of model size starts to show up in “walking - walk + swim”. The two larger models return “swimming”, which is pretty cute. The small text8 model returns “crawling”. Initially, I was worried that this might be a vocabulary mismatch issue, that maybe “swimming” was simply missing from the top 30K most confidently-trained tokens in the text8 model. Once I restricted all three models to a common shared vocabulary, I was able to confirm that this is a genuine example of model quality improving as model training data size increases.
Another interesting result, where you really see the impact of model quality most clearly:
But a lot of the equations are simply nonsense, and Claude’s explanation that “king - man” should expose some kind of “royalty” vector didn’t show up at all.
And with that, I’ve sufficiently scratched this itch, and I’m ready to move on to Chapter 3 – Recurrent Neural Networks.