Over the past few months I’ve been working on mapsnap, a program to automatically georeference old insurance maps. My previous post introduced the project and explained why it’s so interesting and worth doing. This post gets into some more of the technical details.
You can browse all the georeferenced maps at mapsnap.org.
Most volumes of the Sanborn Fire Insurance Maps come with some special sheets that we can use to guide georeferencing. The most important one is the keymap. This is a graphic depiction of which blocks are covered by which pages. Here’s part of one in Brooklyn:
The large black numbers here refer to pages: page 30 covers the Manhattan Bridge approach, page 34 is just to its east, etc. When you manually georeference a Sanborn map on OldInsuranceMaps.net, placing the keymap is typically one of the first steps. With the keymap in place, you can quickly get a sense for where each of the other pages is, how it’s oriented, and what its scale is.
At first glance, the keymap might seem like more than just a guide to fitting each of the individual pages. Isn’t this the answer key? It tells us exactly where to put each page!
On some level, the keymap does have all the answers. But it also turns out to be tricky to work with in practice. mapsnap does make heavy use of the keymap and it greatly improves the quality of the fits. But getting there involved more twists and turns than I would have expected.
Georeferencing a keymap
Just like with the manual workflow on OldInsuranceMaps, the first step of using a keymap in mapsnap is georeferencing it.
At a high level, this works the same as georeferencing a normal page: OCR the street labels, extrapolate them to get candidate intersections, use RANSAC to separate the signal from the noise. In the details, though, it differs in almost every way.
For normal pages, mapsnap operates on 25% scale images. These are much faster to process (16x fewer pixels) and the street labels are still large enough to read easily. The text that’s illegible at this scale is information we don’t need: addresses, business names, building heights.
For keymaps, the streets are much smaller and there aren’t any smaller labels to ignore. Typical street labels on a quarter-scale keymap are only 7-8px tall, too small for reliable OCR. EasyOCR works much, much better on the 100% scale images, where the labels are closer to 30px tall.
One snag I ran into here: EasyOCR doesn’t want to work with images more than 2560px across, since larger images blow up memory usage. If you give it an image larger than this (a typical LoC Sanborn scan is 5600x8300), it quietly downscales it for you. This is helpful in that it doesn’t crash your process, but it’s misleading. It led me to believe, incorrectly, that using full-scale imagery didn’t make a difference. Eventually that conclusion became so surprising that I had Claude dig into it, and we discovered the downscaling. At this scale, EasyOCR has to run on tiled images.
For most Sanborn pages, we’re happy if we can identify at least four or five streets. For large city keymaps, there are typically hundreds of streets, which produce tens of thousands of candidate intersections. This is a different regime for RANSAC. We only need two intersections to produce a fit. For regular pages, I try all pairs and pick the best fit. But that blew up for keymaps and I had to introduce an alternate heuristic.
The good news is that, with so many streets and intersections to choose from, mapsnap can almost always produce a good fit for the keymap.
Reading a keymap
With the keymap georeferenced, the next step is to read off the page numbers from it. Their pixel location gives us a good estimate for the center of each page.
Unfortunately, EasyOCR is really bad at this. It’s better at finding bigger chunks of text like a street name than at finding the short numbers that we need to identify a page.
EasyOCR has two parts: a text detector (CRAFT) that locates bits of text, and a recognizer that reads the text in each box. To reliably detect the page numbers on keymaps, I wound up having to replace both of them.
Training my own models meant I needed training data. One morning in June, I hand-labeled about 1,000 page numbers on 13 keymaps. That turned out to be enough to train both halves:
- The detector is a custom CNN based on MobileNetV3. It slides across the keymap and flags each patch that contains a page number.
- The recognizer is a custom CRNN (a CNN feeding an LSTM) that reads the number in each of those patches.
On those 13 keymaps, EasyOCR read 37% of the page numbers correctly. The CNN + CRNN read 93%. Given the monumental effort it took to train an LSTM back in 2015, I’m amazed how easy this process was with Claude in 2026.
There’s a lot more truth data available, in principle. If a volume on OldInsuranceMaps.net has a georeferenced keymap and georeferenced pages, you can put the two together to label the keymap: the center of page 7 should have a “7” near it. I haven’t built that yet, though.
Detected page numbers in Columbus, OH. Pretty good! A “268” is misread as “263” on the bottom row.
The models went through a few iterations, notably picking up the ability to read page numbers with letters like “1499J” in Los Angeles or “3A” in Asheville.
I’ve kept hand-labeling keymaps since then, and there are now about 2,000 labeled page numbers on 29 keymap sheets. The current pipeline reads 89% of them correctly. That number is flattering, though, since most of those sheets were in the training data. On three keymaps I labeled after training (Columbus, Richmond and Miami), it reads 74% correctly, and the accuracy varies wildly from volume to volume: 96% in Columbus, but only 43% in Miami. Labeling a few keymaps in a new typeface is still the most reliable way to improve it.
Using the keymap location
If all goes well, then for each page we have an approximate location for its center. On a typical volume, the median error is 30–60m (100–200ft), though some volumes do much worse. mapsnap uses this primarily to restrict the vocabulary for street recognition.
My previous post mentioned that OCR was restricted to matching real street names, but I didn’t say much about where these came from. The Library of Congress lists a county for each map, so I started by using all the streets in the county. A Sanborn volume is typically much smaller than a county, but in practice this worked decently well. While there were some ambiguous intersections (“5th and Main”), there were usually enough other streets to disambiguate where in the county the page was.
A Chicago map. The teal dot in the center is the keymap location, and the circle is the restricted street search radius. There are very few streets in the radius.
With an approximate location, though, we can do much better! For each page, mapsnap will only match street names within a few hundred meters of its location: roughly twice the spacing between neighboring page numbers on the keymap, which works out to 270m in Chicago and 600m in D.C. Often there are very few streets in that circle, just a few dozen.
This solves a number of seemingly unrelated problems.
Street names that aren’t streets
Chicago has a LAKE SHORE DRIVE and a MICHIGAN AVE, but when you see LAKE MICHIGAN written on the map, it probably refers to Lake Michigan, the lake, not these two streets. Something similar happens for BRIDGE, CANAL, and city and state names. If there are lots of real streets in an image, RANSAC can sort this out, but if they’re sparse then it can be a problem.
Don’t match SOUTH, BRANCH, CHICAGO, or RIVER. (Chicago p106w)
My initial thought was to disallow ambiguous words like these. But the keymap locations solve this problem much more elegantly. If CANAL STREET happens to run near a page, then we’ll consider CANAL as a street label. Otherwise we won’t. The existence of a BRIDGE STREET elsewhere in the county doesn’t mean we’ll detect all the bridges as Bridge Streets.
Ambiguous intersections
In Washington, D.C., where is 3RD and C? It’s ambiguous! There’s 3RD ST NW & C ST NW, there’s 3RD ST NE & C ST NE, and there are SE and SW versions, too. The four intersections form a rectangle around the Capitol, half a mile to nearly a mile apart. The Sanborn maps do include quadrants like NE/NW in the street labels, but they’re often far away from the street name that they modify, and EasyOCR doesn’t do the best job of detecting them. I found resolving this ambiguity directly to be quite difficult.
Once again, restricting OCR using keymap locations solves the problem neatly. Typically only one intersection will be close to a page’s keymap location, so there’s no ambiguity. Lovely!
Marginal reads
A heavily restricted vocabulary also helped to boost correct interpretations of difficult reads. For example, CLARK on Brooklyn 1939 vol. 2 p13:
CLARK has 6" W.P. (six-inch water pipe) written right next to it, and there’s a double-dashed line depicting the pipe. These are fire insurance maps, after all! But that mark really throws off EasyOCR. With a short list of candidate streets, rather than tens of thousands, it was able to see through the noise and get the correct read. (I eventually wound up fine-tuning EasyOCR’s recognizer and teaching it to ignore these ubiquitous water pipe markers.)
Regions
The keymaps have a colored region around each page number showing the area depicted by that page. This is extremely helpful information! If the page number gives us a location for the center of the page, the region gives us its size, its orientation, and its extent. It basically gives us the fit!
mapsnap uses these regions in one important way: they help decide the scale of a page. Most Sanborn pages for dense downtown areas are drawn at 50ft/in, but 100ft/in and 200ft/in are also common. A page drawn at 100ft/in covers four times the area of one drawn at 50ft/in. So if a page’s keymap region is several times larger than most other pages’ regions, that’s a good sign it’s drawn at a different scale.
mapsnap does try to detect the colored regions around each number, but I’m much less happy with where this has wound up. I use a k-means model to try to detect each distinct color, the background, and the black fill. It often works, but sometimes it really doesn’t.
Claude suggested using SAM (Segment Anything Model) from Meta, but I didn’t get anywhere with this. This problem is difficult for a few reasons:
- You have to ignore really obvious segment boundaries like thick, dark lines. In this image, street lines go through the middle of pages 89, 90, 99, 100, and many others.
- The colors are often not very distinct from the background, particularly if the paper has yellowed over the years. Yellow or brown on light brown paper is not particularly clear, e.g. in page 96 above.
- Conversely, sometimes you need to ignore clear color changes when these correspond to pieces of tape, areas where the page has aged unevenly, or areas where some of the reverse side of the page comes through. You can see this clearly on pages 89, 95, 101, 104, and 105.
If you have ideas on how to do this, I’m interested! This has proven to be one of those maddening computer vision problems that feels simple until you try to get a computer to do it.
Complications
Keymaps are a big win for mapsnap, but they’re not an easy, unambiguous win. They come with some complications.
First of all, how do you tell which page is the keymap? It’s not the first image: that’s usually the cover. It’s almost always on p0 or p1, but which one varies. In a random sample of the Library of Congress’s volumes, most keymaps were on p1. Many older volumes have a p0 that’s a title page with a street index, and the real keymap on p1. And sometimes there are multiple keymaps, like Brooklyn’s p0 and p0b. These cover different subsets of pages and need to be combined.
Sometimes the keymaps have an inset that needs to be ignored or georeferenced independently (Los Angeles) or have a “graphic map of volumes” showing where the other volumes for a city are in relation to this one (Miami). Unfortunately, the graphic map of volumes looks an awful lot like a keymap! If mapsnap reads the volume numbers as page numbers, it can lead to disasters.
Some keymap numbers are legitimately hard to read. Here’s a doozy from Asheville, NC:
That’s a 3, 3A, 3B, 3C, 3D, 3E, 3F, 3G, and 3H. But good luck figuring that out from the pixels alone. The 3B in particular reads more like “3 smudge.” You have to figure out that it’s “3B” by process of elimination. I have an issue proposing to do something like this, but I haven’t found it to be a clear win.
Restricted OCR makes this all very high stakes. If you place a page in the wrong spot, it effectively “poisons” the reads by removing all the correct street names from EasyOCR’s vocabulary. mapsnap does allow “fallback” matches from every street within the keymap’s full extent (792 streets in Chicago, rather than 26,000 in the county) if the restricted vocabulary doesn’t produce a good read. But it’s a delicate balance to keep good reads without letting bad ones back in.
Some recent mistakes illustrate the impact a bad keymap can have:
- In the 1898 Bronx volume, the keymap georeferenced to a tiny 142m patch. Using it left OCR with only 19 distinct street names and mapsnap placed none of the 69 pages. Ignoring it gave OCR 237 street names and placed 12 pages.
- In Wichita 1897, the detected “keymap” was actually a title page with a street index, and it georeferenced to a 237m sliver of the city. mapsnap now throws out any keymap that spans less than 2.5km. That check took Wichita from 0 of 69 pages placed to 42.
Bonus: georeferenced keymaps
If a volume has a keymap and mapsnap is able to georeference it, mapsnap will generate a IIIF georeference annotation for the keymap in addition to the main map content. This is an interesting artifact in its own right. If you’re just interested in streets (as opposed to addresses or buildings or businesses), then the keymap might be all you need. Here are a few georeferenced keymaps on Allmaps:
- Brooklyn 1939 v2 keymap
- Asheville, NC 1957 keymap
- Miami 1950 Vol. 1 keymap
- Washington, D.C. 1916 keymap
Location priors, if you can read them
So is the keymap the answer key? Not quite. Even when everything goes right, it places a page to within 30–60m. That’s a great prior, but mapsnap only counts a page as well georeferenced if it’s within 25 feet (8m). (I’ll have more to say on my scoring metric in a future post.) And when something goes wrong (a misread number, the wrong page taken for the keymap, an inset), the keymap location can be off by kilometers. So mapsnap treats the keymap as a prior, not as a placement: it narrows the search, but the streets on the page still have to agree.
Used that way, keymaps are a great source of location information, and mapsnap makes heavy use of them. Like good engineering, they solve a whole class of seemingly unrelated problems. In Chicago, adding the keymap cut the mean error of placed pages from 114ft to 45ft. When a bug hid Oakland’s 1903 keymap, mapsnap placed 84 of its 148 pages. With the keymap, it placed 129.
Keymaps were one of the biggest wins over mapsnap’s core OCR + RANSAC algorithm. They have one more important use: the keymap location is where the eponymous “snap” operation starts looking for each page. Snap was the biggest win of all, and it’s the subject of the next post.