Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Day 7 · Seeing the Pattern

You can’t see fifty dimensions at once. Today you compress them to two so the structure becomes visible, then check whether the compression has misled you.

Describing one thing with many numbers

By now you have spent several days turning things into numbers. On Day 3 a text became a list of word counts. On Day 4 a sign became a set of shape measurements. On Day 5 a vowel became a pair of formant frequencies. In each case one object stopped being a thing you look at and became a row of numbers.

A row of numbers describing one object is a feature vector. The vowel from Day 5 had two features (a first formant and a second formant), so its vector was two numbers long: something like (650, 1100). A sign from Day 4 might have ten or twenty shape measurements (height, width, curvature at five points, the area of the enclosed loop), so its vector is ten or twenty numbers long. A text from Day 3, counted across the thousand most common words, has a vector a thousand numbers long.

Here is the move that makes the rest of the day possible. A vector of two numbers is a point on a flat chart: the first number is how far right, the second is how far up. The Day 5 vowels were exactly this: you plotted them on a chart of F1 against F2 and saw the vowels separate. A vector of three numbers is a point in a room: right, up, and toward you. You can still picture it.

A vector of twenty numbers is a point in a twenty-dimensional space. The arithmetic is identical (twenty coordinates instead of two), but you cannot picture it, because your eyes and your imagination stop at three. The space is real and the points sit somewhere definite in it. You just can’t look.

This is the whole problem of the day. Your data live in a space with more dimensions than you can see, and the structure you care about (which signs group together, which texts resemble which) is a fact about how the points sit in that space. You need a way to look at something you can’t picture.

Discussion questions

Why high dimensions are hard, not just unviewable

The trouble is worse than not being able to draw the picture. High-dimensional spaces behave in ways that work against you, and the umbrella name for this is the curse of dimensionality.

The piece of it that matters today concerns distance. The whole point of measuring your signs or vowels was so you could say two of them are similar (their vectors are close together) and two others are different. In two or three dimensions this works the way your intuition expects: near things are clearly near, far things are clearly far.

As dimensions pile up, that contrast fades. With enough features, almost every pair of points ends up at roughly the same distance from every other. The nearest neighbor is barely nearer than the farthest one. Distance stops sorting things cleanly, which is a problem, because “these signs are close, those are far” is exactly the judgment you want to make.

A plain way to feel it: with one feature, you sort your signs along a line and crowding is obvious. Add features and you give every sign more room to be unlike every other sign in some way. Pile on enough features and everything is a little unlike everything, so nothing stands out as especially similar. The signal you want, real groups, gets diluted by the sheer number of ways things can differ.

This is one reason reducing dimensions is not only about drawing a picture. Squeezing the data down to a few well-chosen dimensions can sharpen distances that high dimensions had smeared out.

Discussion questions

Why reduce dimensions

Each item you study (a sign, a vowel, a text) can be described by many numbers at once, which places it in a high-dimensional space no one can picture. Dimensionality reduction finds a two-dimensional view that keeps as much structure as possible, so clusters become visible. The compression distorts, and different methods distort differently.

The goal is to see structure that is genuinely there but invisible in a table of numbers. Stare at a spreadsheet of 150 signs by 20 measurements and you will see nothing: no person reads structure out of three thousand numbers in a grid. Project those 150 points down to a flat chart and a glance can show three clumps, an outlier off to one side, a thin bridge of points between two groups. The structure was always in the data. Reduction makes it something an eye can take in.

Two kinds of structure are worth looking for. Clusters (groups of points that sit together) are candidate categories: hands, periods, dialects, scribes. Outliers (lone points far from everything) are candidate oddities: a misread sign, a mistyped number, or a genuine exception worth a closer look.

Discussion questions

PCA: the linear view

Principal Component Analysis is the first tool to reach for, and the easiest to understand once you see what it is doing.

Picture your points as a cloud in high-dimensional space. The cloud is usually stretched: longer in some directions, flatter in others. PCA looks for the single direction in which the cloud is most stretched out: the direction of greatest variance, where the points are most spread apart. That direction becomes the first new axis, the first principal component. Then it looks for the next most stretched-out direction at right angles to the first, and that becomes the second component. And so on.

These components are new axes, built from the data rather than handed to you. When you plot the first two, you are looking at the cloud from the angle that shows its widest spread: the most informative flat photograph of a three-dimensional object you could take, generalized to many dimensions.

Two ideas make PCA honest to use.

The first is the explained variance ratio. Each component captures some fraction of the total spread in the data. The first might capture 40 percent, the second 25 percent, and so on down. When you plot only the first two components, you are seeing only their share (say 65 percent), and the rest of the structure is flattened out of view. A scree plot stacks these fractions as bars from largest to smallest. If the first two bars are tall and the rest are stubs, your two-dimensional picture is faithful. If the bars trail off slowly, much of the structure is hiding in the dimensions you cannot see, and you should not trust the flat picture too far.

The second is interpretability. Because each PCA axis is a fixed combination of your original features, you can sometimes read it. Suppose your Day 4 signs were measured for height, width, and loop area. If the first component turns out to be mostly height-minus-width, then “left versus right on the chart” means “tall-narrow signs versus short-wide ones,” and that is a statement you can take back to the manuscripts and check. Not every component reads this cleanly, but when one does, PCA hands you an interpretation, not just a picture.

What PCA blurs: it only ever rotates and flattens. It cannot bend the cloud. If your groups are separated in a curved or tangled way rather than along straight directions, PCA may smear them together, and you will need a nonlinear method to pull them apart.

Try it on real data. The end of the notebook runs exactly this on the G17 owls from the Isut data: PCA to two dimensions, colored by text. Because the features here are the raw pixels of each sign rather than named measurements, you read the axes by eye instead of by loading. Look at the owls sitting at the extreme ends of PC1 and PC2 (the notebook draws the actual glyph at each extreme) and try to name what changes as you move along each axis: overall size and ink, the openness of the loop, the slant of the body.

Discussion questions

UMAP and t-SNE: the nonlinear view

UMAP and t-SNE work on a different principle, and it is worth understanding because their strength and their danger come from the same place.

Instead of finding straight directions of greatest spread, these methods look at each point’s local neighborhood (who its nearest neighbors are) and try to build a flat picture in which those same points stay neighbors. They preserve who is near whom, and they are willing to bend and stretch the space however they must to do it. That bending is why they can separate groups that PCA smears: a curved arrangement in high dimensions can be unfolded onto the page.

Each has one main knob.

Now the danger, and it is the most important caveat of the day. Because these methods only promise to keep neighbors together, the parts of the picture they do not promise to preserve are exactly the parts an untrained eye reads as meaningful:

So what can you trust from a UMAP or t-SNE plot? Which points sit together. That is the one thing these methods are built to get right. Treat the membership of clumps as a real signal and the rest of the layout (distances, gaps, orientation) as decoration the method was free to invent.

Discussion questions

Reading a projection honestly

Pull the caveats together into a habit. When you look at any of today’s plots, sort what you see into two piles.

Trust: which points sit together. If the same points clump in PCA, in UMAP, and in t-SNE, and the clump survives different settings and different seeds, you are looking at real structure.

Don’t trust: exact distances, gap sizes, and orientation. How far apart two clusters are drawn, how wide the blank between them is, which way the whole picture is turned. These are products of the method, not facts about your manuscripts.

A short checklist for any projection you make:

A cluster that passes these is a hypothesis worth testing. A cluster that appears in one method, at one setting, at one seed, and matches nothing you know, is a picture and not yet a finding.

Discussion questions

Clustering to back up the eye

Eyeballing groups is a start, not a result. Two people can circle different clumps in the same plot. To turn a visual impression into something you can defend, you let an algorithm draw the groups and then score them.

Two clustering methods are worth knowing in plain terms.

Naming the groups by machine is only half the job. The other half is checking them against something you already know. If you have true labels (you know these signs are from the early hand and those from the late one), you can ask how well the machine’s grouping matches the truth. The adjusted Rand index does this with a single number. It runs from 0 to 1: a score near 1 means the clustering recovered your known groups almost exactly. A score near 0 means it grouped the points no better than chance. “Adjusted” means it already accounts for matches that would happen by luck, so the number is honest.

This closes the loop. The projection suggested three groups. K-means drew them. The adjusted Rand index against your period labels reports whether those drawn groups are the real ones. A high score turns “I think I see three hands” into “the measurements separate three hands, and here is the score.”

Outliers deserve the same discipline. A point sitting alone is a candidate, not a verdict. Find it by distance (it is far from every cluster center), then go back to the source. A vowel marooned far from its vowel category might be a genuine rare pronunciation, or it might be a formant mismeasured because the recording was noisy. A sign off on its own might be a real variant form, or two strokes that the segmentation accidentally merged. The plot tells you where to look. Only the manuscript tells you which it is.

Try it on real data. The notebook closes by doing this on the G17 owls: k-means draws three forms on the PCA coordinates, and then you check whether those forms line up with the text each owl came from. One form turns up almost only in the Shipwrecked Sailor, which is the result to look for. There are no hand labels here, so the text of origin plays the role the known hands would. Then pick an owl sitting apart from its group, open its crop, and decide whether it is a genuine variant form or a segmentation slip where two strokes were merged.

Discussion questions

Warm-up

Suppose you had to place every classmate on a two-dimensional chart so that similar people sit close together. Which two axes would you choose, and what would the chart leave out? That’s the problem in miniature.

Project the data

Open the notebook:

The notebook also runs UMAP twice with different random seeds, so you can watch the gaps between clusters shift while the clusters themselves hold.

Working with real data

The toy data had clean, known clusters so you could check the methods. Real material rarely looks so tidy. For a worked example on the genuine article, the notebook ends by reproducing a result from a study of hieratic, the cursive handwriting of ancient Egypt: Tabin et al. (2023), Collaborative Annotation and Computational Analysis of Hieratic, Figure 7.

A single hieratic sign was drawn many different ways. Take one sign, G17 (the owl), as written in two Middle Egyptian texts, turn every drawn instance into a row of numbers, and run PCA to two dimensions, colored by text. The owls separate into a few recurring forms. Two of them appear in both texts, and one is found almost only in the Shipwrecked Sailor. The clustering and adjusted-Rand machinery from earlier then puts numbers on those forms.

Discussion questions

Making results interactive

Academic papers are limited in their ability to convey data with complex, multidimensional relationships. But websites can do things differently. For a recent paper, I decided to accompany the handful of printed figures with an interactive version, which allows readers to explore the full dataset in detail, something that would never be possible in print.

We’ll take a look together at Hieratic Sign Complexity over Time and discuss how this does something academic papers cannot.

Discussion questions

Measuring Manuscripts
Day 6 · Putting a Number on It
Measuring Manuscripts
Day 8 · From Curiosity to Project