The Atlas work list
59 items open on the Atlas itself, 15 of them waiting on somebody outside it. This is the Atlas's own list: each family archive keeps its own, summarised just below.
59 open on the Atlas 594 open across 8 family archives
The family archives keep their own 8
Each archive's list is the live one and lives on its own site. These are counts taken on 3 November 2026, so a number here can be a little behind the page it links to. Open means running, next or blocked.
- Blažević 39 open 12 running 24 next 3 blocked 40 done
- Booyzen 63 open 36 next 27 blocked 148 done
- D'Arcy 29 open 4 next 25 blocked 122 done
- Defranceschi 138 open 14 running 104 next 20 blocked 134 done
- Falco 41 open 4 running 28 next 9 blocked 70 done
- Lerena 88 open 2 running 67 next 19 blocked 128 done
- Luwinski 107 open 3 running 62 next 42 blocked 153 done
- Mazza 89 open 14 running 62 next 13 blocked 94 done
The Atlas's own list
Walk every FamilySearch collection's waypoint tree
Running now. 3,504 collections; the tree descends to the individual book. 707,471 volumes so far against 4,903 when the day started. The catalogue is gated but the records API is not, which an earlier note in import-fs-catalogue.py had written off without trying — the same mistake as declaring Scotland's parishes closed when data.gov.uk had them. Publishes every two hours while it runs. US collections are half the total and come last, so the rest of the world lands first.\nSupervised now: scripts/supervise.sh restarts it if it dies, bounded to five attempts.
Place the harvested volumes on the places they belong to
AS OF 11 OCTOBER: 200,338 volumes under 14,268 names are unplaced, out of 3,891,215 considered. Everything below is the history of getting there and the figures in it are the figures of their day — this page carried four different totals at once and a reader had no way to tell which was live. The number above is the current one.
347,272 of 775,686 books land on a place. The rest name somewhere this atlas has never drawn, and they are written to data/fs-volumes-unplaced.json ranked by how many books are waiting — the map's next places, not a silent loss.
STALE SINCE SEPTEMBER. «347,272 of 775,686 books land on a place» is four harvests out of date. Measured 9 October: 3,272,076 volumes land on 43,524 places, and 257,524 across 16,541 names do not. See row 19, which now carries the full table.
Cross-border collections: a register moves when a jurisdiction moves
Done 20 September. Of 1,165 Croatian places, none was reached by a collection filed under another country — the stated thesis of this atlas was dead. Two causes: placeIds were resolved through the wrong endpoint, which answered an unrelated place with HTTP 200 (Italy, Pola and Trieste came back as Monomie, New South Wales); and a title disagreeing with its places was treated as an error and the places discarded. Now ten explicit filing groups — Habsburg, Yugoslav, Prussian, Russian, Ottoman, Nordic, Low Countries, Iberia, these islands, the Alps — because records cross for historical reasons, not geographic ones. Verteneglio finds Brtonigla, Isola d'Istria finds Izola. Kept honest by scripts/test-cross-border.py.
A surname timeline for every country, not only France
Done 20 September. Where a name is found today and its variations were already global; WHEN it was borne existed for France alone. 363,574 surnames now carry a birth timeline from Wikidata, reaching the 1400s. Aggregated over the variant family and folded, so a rare spelling inherits its family's history — Defranceski has no notable namesake, De Franceschi has eleven, Italy from the 1550s and Croatia from the 1800s. The bias is stated on the page: these are people notable enough for an encyclopaedia, not a census.
Stop a name inheriting every country its archive touches
Done 20 September. Testing on David's own name found De Franceschi attested in 29 countries including Albania, Cuba, Paraguay and Zimbabwe — all from one Friulian family's archive. The fallback gave any unplaced name the archive's whole geography. Archives that place names individually do it for most of them, so silence in that list is information. The fallback survives only where an archive has one country and cannot mislead.
Antenati: the Ministry's own registers, on the map
Done 20 September. 5,591 registers harvested from the SAN open-data feed in September and read by nothing since — the file sat in data/ while Italian places showed only FamilySearch. 4,159 are now on 72 comuni, 68 of which this atlas drew because a register pointed at them.
Scotland's civil parishes, searchable
Done 20 September. 871 parishes from data.gov.uk, harvested and then read by nothing. The civil parish is the unit every Scottish record is filed by — the OPRs, the statutory registers, every census — and typing one returned nothing at all.
Resume the Antenati harvest at volume 6,000 of 464,770
Blocked on THEIR side, not ours. www.san.beniculturali.it is unreachable — port 80 refuses and the parent site times out. Stopped cleanly with its resumption token intact. Test with curl "http://www.san.beniculturali.it/oaicat/OAIHandler?verb=Identify" before retrying.
9 OCTOBER: THE RECORDED BLOCKER IS GONE AND THE HARVEST IS STILL NOT RUNNABLE. THE REASON CHANGED COMPLETELY.
The test this row names now PASSES. www.san.beniculturali.it is up; port 80 answers a 301 to https, and verb=Identify returns a valid OAI-PMH response — «SAN OAI repository», earliestDatestamp 2011-12-01, granularity YYYY-MM-DD. The harvester's OAI constant is still http://, which is fine: it fetches with curl -sS -L and follows the redirect.
BUT EVERY LISTING VERB DIES AT A GATEWAY TIMEOUT. ListRecords and ListIdentifiers with no date range both return 504 Gateway Timeout at ~61 seconds, as does a resume with the saved token. Only the trivial verbs answer — Identify in 1.4s, ListSets in 1.5s.
SELECTIVE HARVESTING IS THE WAY THROUGH, AND IT MEASURES THE JOB EXACTLY. Adding from/until brings the response inside the limit, and probing year by year found that all 464,770 records are datestamped 2015 — 2013 and 2017 through 2026 return noRecordsMatch, and so do single months of 2015, so the stamps sit on a handful of days. With from=2015-01-01&until=2015-12-31:
- page 1: HTTP 200 in 44.8s, 50 identifiers,
completeListSize="464770" - page 2, by resumption token: HTTP 200 in 45.7s, 50 identifiers
SO THE PAGE SIZE IS 50 AND EVERY PAGE COSTS ABOUT 46 SECONDS, RESUMED OR FRESH. 464,770 / 50 = 9,296 pages. At 46s plus the 2s pause that is roughly 124 hours — five days of continuous polling — with every single page landing within about fourteen seconds of the 60-second gateway timeout. That is why the first run reached 6,000 records and stopped: not a refusal, a throughput wall.
AND THERE IS NO BULK FILE. The note above points at dati.gov.it as the open-data route; package_search?q=antenati returns exactly 2 datasets — «Oggetti digitali SAN in formato PICO» and «in formato METS-SAN» — and BOTH resources resolve to the same OAIHandler?verb=ListRecords URL that times out. The Ministry's open data IS the OAI endpoint. The mets-san prefix returns noRecordsMatch for this set and date range. ListSets shows a second set, m-ANTENATI beside pico_m-ANTENATI, which is untested.
WHY IT MIGHT STILL BE WORTH FIVE DAYS, AND WHY IT MIGHT NOT. Antenati is this atlas's only substantial source of FREE holdings: 74 of the 8,557 per-place collection entries, reaching 3 places. Row 76 established that the eight-colour legend is really two colours because almost nothing on the map is free. Finishing this harvest is the one job in the file that would change that, because it is free Italian civil registration across a great many comuni. Against it: row 80 measured Italy at 46.4% of all volumes already, and warned that adding Italian places «moves the headline and almost nobody's research». Both are true at once. It saves as it goes and resumes, so it can be run in days rather than one sitting — the question is whether five days of polling buys colour that matters.
11 OCTOBER: THE SERVER IS DOWN AGAIN, AND THIS TIME IT IS THE WHOLE HOST. The harvester was taught two things first — a --from/--until window so it enters by the 2015 date range the probing above found, and a fallback for the case the 9 October run actually hit: a dead resumption token answers with SILENCE, not with badResumptionToken, so the run now drops the token and re-enters by the window instead of stopping. The trial run exercised both paths correctly and then found nothing on the other end.
www.san.beniculturali.it resolves to 2.42.228.36 and both port 80 and port 443 are shut. verb=Identify, which answered in 1.4s two days ago, now returns nothing after 75 seconds; the parent site does the same. dati.gov.it still lists exactly the two datasets and both still point at this host, so there is still no second route. Not a bot gate and nothing to work around — the Ministry's server is off.
The 5,591 volumes already harvested are untouched and the resumption token is still in data/antenati-registers.json. The windowing and the token fallback are the part of this that was ours to fix, and they are fixed. Retest with curl -m 20 "http://www.san.beniculturali.it/oaicat/OAIHandler?verb=Identify"; when it answers, python3 scripts/harvest-antenati-oai.py picks up where it stopped.
index.json is over its 800 KB budget
Done 21 September. Resolved by measuring the right unit. The budget counted bytes on disk; GitHub Pages gzips JSON about four to one, so an index failing at 1,084 KB is 271 KB over the wire — the number nobody has ever downloaded was the one failing the audit. Budgeting disk bytes creates pressure to delete real places to satisfy a figure no reader experiences. The budget is 400 KB transferred. A field f that nothing read came out too: 59 KB on 10,046 rows.
Point recordatlas.org at GitHub Pages
Bought 17 September; the A records still point at the registrar's landing page. scripts/cutover-domain.sh does the whole switch and refuses to run until at least two of GitHub's four addresses appear, because a CNAME file and a base of / are not independently safe. Replace the A records with 185.199.108–111.153.
DONE, AND THIS ROW WAS STALE. Checked 9 October: recordatlas.org resolves to ALL FOUR of GitHub's addresses — 185.199.108.153, .109.153, .110.153, .111.153 — the site answers with server: GitHub.com and an x-github-request-id, and site/public/CNAME contains recordatlas.org. The cutover described above has happened; the row simply never got closed. This session's browser specs have been running against https://recordatlas.org all day, which is the strongest evidence there is.
AdSense answers unfilled
Everything this atlas controls is verified correct, live, on 21 September. The slot renders as <ins class="adsbygoogle" style="display:block"> with data-ad-client=ca-pub-4156851882993001 and data-ad-slot=1255898967, format horizontal, full-width-responsive; the adsbygoogle script loads; the push call is present; and ads.txt at daviddef.github.io answers 200 with the matching publisher line. The earlier faults are both gone — the slot was a flex child with zero width, which is what «No slot size» meant, and it is display:block now. So an unfilled slot from here is AdSense's own decision about approval or inventory, not a mistake in the page. Only David can see the AdSense dashboard.
Ask the Meertens Instituut about the Familienamenbank
DRAFTED, not sent — docs/letter-meertens.md. It asks for surname distributions at surname/region/count grain, offers credit and whatever licence they set, says plainly that the site carries advertising in case that makes the use commercial in their terms, and says a clear no is itself useful because the atlas records why each country is missing. Only David can send it.
Waiting on NARA and Trove
NARA's API key needs a human to request it. Trove replied with ticket RSref189042 and nothing since.
The site is 844 MB of GitHub Pages' 1,024 MB
Done 21 September. The surname dumps were a sixth of the site and are gzipped now, 165 MB to 17. places-full.json was 143 MB imported by nine pages, of which 135 MB duplicated the per-place detail files — slimmed to 6.9 MB, which also fixed a build that had started dying out of memory. 815 MB of 1,024 and tracked by scripts/audit-site.py.
Run the whole build before pushing, not one stage of it
Done 20 September. Three CI failures in one afternoon, all from running npm run data and calling it tested. One of them — row is not defined — was sitting in my own local log unread. The gates work; skipping them does not.
Put every harvested dataset onto a page
Done 21 September. Three datasets had been harvested and read by nothing. Antenati: 5,591 registers from the Ministry's open data, 4,159 now on 72 comuni. Scotland: 871 civil parishes, the unit every Scottish record is filed by, and typing one returned nothing at all. And this work list, which the estate keeps one of per archive and this atlas had none of.
David's own surname returned 404
Done 21 September. A surname got a page only with a CURATED variant, so 564 names a family archive had actually placed in a country had none — Defranceski among them, attested in nine countries with every variant phonetic. And the birth timeline was null on every page: the module resolved its data file relative to import.meta.url, Vite bundles it elsewhere, the read threw and a bare catch swallowed it. A silent catch around a path is how a whole feature disappears with nothing failing.
A hidden tab never wrote the URL
Done 21 September. Reported by the archive-kit session. URL writes coalesced on requestAnimationFrame alone, and a browser runs no frames for a hidden tab — so somebody who switched away, changed the year and came back had an address bar describing the map they left. I had seen this myself while testing the record-kind filter and written it off as an automation artefact.
Place the remaining unplaced books — 799,897 across 46,345 names
AS OF 11 OCTOBER: 200,338 volumes under 14,268 names are unplaced, out of 3,891,215 considered. Everything below is the history of getting there and the figures in it are the figures of their day — this page carried four different totals at once and a reader had no way to tell which was live. The number above is the current one.
Measured 21 September: 1,571,627 volumes are placed, across 7,778 places — L'Aquila 72,724, Avellino 62,558, Bergamo 55,242. What is still unplaced is 870,018 volumes under 31,530 names. The largest single cause found so far is that an unplaced name carries its COLLECTION's country and not its own: 168 Friulian villages sat under AT holding 32,579 books, because the collection is the Austrian Küstenland. The exonym harvest now searches the historical filing groups, and the unplaced list is no longer cut at 4,000 names, which had been hiding 200,641 books from the only thing that looks for them.\n21 September, measured: 2,187,438 volumes placed on 9,910 places; 799,897 volumes across 46,345 names still unplaced, and the list is now written out in full rather than cut at 4,000. Three systematic causes found in the top of that list and fixed in the matcher: «不明» is Japanese for unknown and held 2,516 volumes that no gazetteer will ever match; «Zona de reclutamiento de Madrid» is 5,718 volumes of Madrid behind an administrative wrapper; «Auckland Court» and «Christchurch Court» are a thousand volumes each standing in cities this atlas has drawn. Unwrapped readings are tried last, never first, because put first they made «Hampton Court» resolve to Hampton.
MEASURED AGAIN 9 OCTOBER, AND EVERY NUMBER IN THIS ROW WAS BADLY STALE. From data/fs-volumes-world.json.gz and data/fs-volumes-unplaced.json:
| 21 September | 9 October | |
|---|---|---|
| volumes placed | 2,187,438 | 3,272,076 |
| places they sit on | 9,910 | 43,524 |
| volumes unplaced | 799,897 | 257,524 |
| names unplaced | 46,345 | 16,541 |
The unplaced pile is down 68% and the placed count is up by a million. The rows kept advertising the September figures, which told a reader the atlas was far worse than it is.
The matcher's own diagnostics for the remainder: 262,498 unmatched, 356,974 with no path name, 286,712 placed on an administrative division rather than a settlement, 362,755 refused as too far from their parent, and 224,842 spans re-derived from the title. A further 23 names holding 4,974 volumes are marked unplaceable on sight — «unknown» in several languages, and record types like a binding or a bundle of wills.
AND THE TAIL HAS CHANGED CHARACTER, WHICH MATTERS MORE THAN THE COUNT. fs-volumes-unplaced.json describes itself as «Settlements FamilySearch has books for and this atlas has never heard of», and that is no longer what is in it. The largest entries are ADMINISTRATIVE AREAS: «Ile-de-Montréal, Lapraire, Chambly, Vaudreuil-Soulanges counties» (2,842), Тверь (2,730), Бессарабия (1,446), Тула (1,293), Szabolcs (1,141), Jász-Nagy-Kun-Szolnok (1,117), Toscana (1,001), Antwerpen (1,000). Those are not villages nobody has drawn; they are counties, oblasts, provinces and a historical region, filed at the rung the record was filed at. Chasing them as missing settlements will not find them. By country the remainder is IT 44,387 · US 29,202 · RU 26,326 · DE 25,085 · SE 20,900 · BY 11,679 · MX 10,179 · NL 9,822 · CZ 9,737 · CA 8,805.
Close the countries with no surname register — was eight, now three and a half
Italy, Denmark and Belgium closed; Croatia, Sweden and Finland rewritten — all on 22 September, and all by a reader rather than this project. David sent the Croatian statistics office's own name application, which answers. Checking the rest the same way found the identical mistake under Denmark, whose entry proudly recorded that all 2,308 tables in the statistics bank had been searched for «navn» — they had been; Denmark publishes on its website. Then Belgium, which had been rechecked and confirmed that same morning by reading the open-data category directly: Statbel keeps its surnames on the family-names theme page. FOUR ENTRIES, ONE MISTAKE: search a machine-readable index exhaustively, then make a claim about an institution. The pattern is the finding. Belgium is also the best surname data here — per municipality, 565 communes, 106,869 names, CC BY 4.0, which is surname to place to count and nothing else in the directory has that grain. Left: Czechia (possible and refused — the ministry's own record says both, so it needs a formal request, not a script), the Netherlands (Meertens, needs the letter), Finland (one manual download from a portal that 403s us), Sweden (a lookup and a top-100 table, no list published anywhere). Croatia is half closed: 141 surnames the atlas already attests, asked one at a time.
FINLAND CLOSED, 9 OCTOBER — AND IT WAS NEVER REFUSED. THE PORTAL HAD MOVED.
This row listed Finland as «one manual download from a portal that 403s us», and _refused.json had the receipts: avoindata.fi answered «403 to every request» and «403 to an identified crawler». Retested today it answers 200 — because avoindata.fi now redirects to avoindata.suomi.fi, and the September probe did not follow the redirect. A 301 nobody follows is indistinguishable from a refusal if you only read the status code. Nobody changed their mind about crawlers; the address changed.
What was behind it: «Väestötietojärjestelmän suomalaisten nimiaineistot» — the DVV's own name lists from the Population Information System, CC BY 4.0, dated 2026-08-04, two spreadsheets. scripts/harvest-finland-surnames.py reads the surname sheet with the standard library (an .xlsx is a zip of XML, so this adds no dependency) and writes data/frequencies/fi.json: 23,265 surnames, 4,917,684 bearers — Korhonen 21,144, Virtanen 20,295, Mäkinen 18,833.
THE LIMIT IS PUBLISHED AND BELONGS IN EVERY ANSWER DRAWN FROM THIS FILE. The DVV withholds any surname with fewer than twenty bearers — «Sellaisia sukunimiä joiden nimenhaltijoita on alle 20, ei esitetä tietosuojasyistä». So fi.json is complete above twenty and silent below it, and a Finnish name missing from it is far likelier to be rare than unknown. The count is living Finnish citizens holding the name, resident in Finland or abroad, which is why it reaches the emigration this atlas exists to follow.
AND THE LESSON IS THE SAME ONE THIS ROW ALREADY LEARNED ONCE. Its own headline finding was «search a machine-readable index exhaustively, then make a claim about an institution» — four countries, one mistake. This is the fifth instance with a new face: search an address exhaustively, then make a claim about a host. Recheck the refusals; «they said no» and «we asked the wrong URL» read identically in a log.
Left: three and a half. Czechia (needs a formal request), the Netherlands (Meertens — retested today, cbgfamilienamen.nl still 403 and nfb.meertens.knaw.nl still does not answer at all; the letter at docs/letter-meertens.md is the only route, which is row 12), Sweden (retested today, api.scb.se lists BE0001 and both name folders still answer 400 — unchanged), and Croatia still half closed.
The Kosina test: a Slovene parish, a German exonym, and an s/z spelling
David's Blazevic research has a Martin Kozina born about 1841 in «Kranjska župa sv. Martin kod Krainburg» — Carniola, the parish of St Martin near Kranj, whose German name is Krainburg. Driven against the live map on 21 September: Kranj, Krainburg, Kranjska and Carniola all frame Carniola at 46.348,14.067; Kozina lands at 45.607,13.943 and «Kosina» lands on it too, through the spelling tier; Šmartno and Smartno both reach 46.189,14.939. One name failed and is now fixed: «Krain» flew to Ukraine, because "ukraine" contains "krain" — a country is matched on whole words now. What the map cannot do is show his register: FamilySearch publishes 2 collections for Slovenia and the waypoint walk finds Kranj zero times. Those books are on Matricula and the Slovenian archive portals, which this atlas does not scrape, so Kranj's page names the archives that hold them instead of pretending to the books.
Make surname pages lean enough to afford 16,138 of them
Done. 16,135 surnames earn a page and all 16,135 are live and in the sitemap. The row was blocked for weeks on two numbers that were both wrong: 24.6 KB a page (measured off Defranceschi, which carries eleven country blocks when the median surname carries one) and «985 MB of 1,024» (the RAW build tree, when the limit applies to the artifact GitHub deploys). Admitting the class took the artifact from 158 MB to 164 MB. Twelve thousand pages cost six megabytes of what actually ships. Two cuts paid for part of it — nearest-archive listing and class-based style scoping — and CI now prints the weight on every build so no decision here is ever made against a guess again.
A showcase of the family archives at the foot of the map
Live at the foot of the home page as «Eight families, the same method», all eight archives including Luwinski. Two errors found afterwards by checking the cards against the data behind them and both fixed: Booyzen printed the same name twice and omitted Booyens, which is the form its archive actually records, and Luwinski listed Sauerbaum among the places. audit-site.py now checks the cards every run.
The Otočac question: can the map see a place indexed across two countries?
David asked whether the atlas would pick up Otočac being indexed by two collections across a border — Church Books 1839-46 and Austria Küstenland 1848-58. Traced end to end on 21 September. Half of it works and the other half cannot. Otočac's page does carry its FamilySearch collections and the cross-border matcher does cross borders where there is something to cross with — IT to HR 9 places, IT to SI 16, AT to DE 2, with no false positives. But the Austrian side of David's example is not reachable by this method at all: «Austria, Catholic Church Records, 1565-2007» (collection 5000002) has been walked and is recorded in the harvest's own indexOnly list. It publishes no browsable volumes through the open waypoint API, so its tree names no places, Croatian or Austrian. The Küstenland attribution David saw is in the FamilySearch CATALOGUE, which is gated, and this atlas does not climb gates. So the honest position is: the map shows Otočac's Croatian books and names the archives for the rest, and no amount of better matching will reach the Austrian volumes while they are only in the catalogue.
Stop measuring the site against the wrong limit
audit-site.py reported «the deployable site is 1418 MB and GitHub Pages publishes at most 1,024 — this will not deploy» about a build that deployed successfully minutes later in the same CI run. It was adding up raw bytes in dist/ and comparing them to a limit that applies to the COMPRESSED artifact, which GitHub stored at 189 MB. A 7.5:1 ratio, on a site that is almost all HTML and JSON. That one wrong comparison is why 16,138 surname pages were refused as unaffordable for weeks. The check now estimates the compressed size, sampled per file extension with a seeded sample so builds stay comparable, and prints the raw figure beside it so a local du does not look like a contradiction.
The map waited on a fetch a quarter of which said «nothing here»
Done 22 September. index.json is the one file the first paint waits on and it had reached 533 KB over the wire. There is no fat in the rows — 28 bytes each, and dropping every derivable id saves 3% because gzip already collapses that repetition. The honest saving was not in the rows but in which rows: 4,932 of the places carry no volume and no collection. They are real ground, correctly placed, nobody has walked them, and the difference between «not surveyed» and «nothing here» is one of this atlas's promises — so they stay on the map. They just stop being something the map waits for. 409 KB draws the atlas; 124 KB fills in the ground behind it a moment later. The split's real cost was everywhere else: ten static pages imported index.json and would have silently dropped those places — no /place/<id>/ page for any of them — so they now import the index whole from lib/atlas.js. The audit's integrity checks, the deployed-site comparison and the unread-field gate were all reading half a file and believing it was the whole one.
Draw the migration a surname made, not just where it ended up
David's, 22 September. The surname page says where a name is borne and where a record names it, as two lists. It does not say that the one became the other. Blažević on this map is a column of countries; what a reader wants to see is Croatia to Germany, Croatia to the United States, with the weight of each line. The data for it is partly here already — archive-surnames.json carries each archive's declared country against every country its research reaches, which is a claim about movement — and partly not: nothing in this atlas yet holds «N families of this name left HERE for THERE». Acta Croatica has exactly that for Croatia and cannot be ingested (no licence, unstated derivation — see data/_declined.json), so the honest version is built from emigration sources this atlas can cite, not from arrows we felt like drawing. An arrow is a strong claim. Drawn from a guess it is the most confident-looking lie a map can tell.
Make every surname a page worth citing: spellings, meaning, history
David's, 22 September, and the largest thing on this list. The aim is for Record Atlas to become a source people cite for names, by drawing pictures no other site can — which it can only earn by being right, one name at a time. Three parts, and they are not equally hard. SPELLINGS is nearly solvable now: the atlas already links s/z and c/k alternations between two independently attested names, and Acta Croatica's Horvat page (Horvath, Horwath, Thorvat, Horváth, Horwat, Horvát, Thorwat, Horvatt, Horvaht) shows the standard to reach. MEANING and HISTORY are research per name, not a harvest — and the moment a page asserts an etymology it cannot cite, it stops being citable itself. So: no generated etymologies, no plausible guesses. Each one carries its source or the page says nothing. 16,138 surname pages at that standard is years of work, which is the right answer rather than a reason not to start: begin with the names the family archives actually turn on.
PROGRESS, 23 Sep: the surname page now draws the one picture no other surname site can. Every such site can tell you a name is common in Croatia; this one also holds 40 record regions with an era table naming the polity and the archive its paper went to, and the two are halves of one question — my people were called this, they were there, so who has the paper, and for which century. A Blažević page now lists the seven pieces of Croatian ground this atlas has drawn, the Military Frontier among them, each linking to its own timeline. It is careful about what it is not: a register counting a name in a country says nothing about which village, and a region may cover only a corner of a large country — Acadia reaches into Maine and is not a statement about the United States. Spellings, meaning and history are still the rest of this row.
Denmark's 294,496 surnames are harvested and will not build
Done 22 September. 294,496 Danish surnames, CC BY 4.0, now in the corpus — verified by a full local build before pushing: 41,263 pages, 19,779 surname pages, peak RSS 4.6 GB against a 10 GB ceiling. It took three wrong diagnoses. First GitHub Pages' 1,024 MB, which the deploy's own report already answered — 385 MB stored. Then the heap ceiling, raised twice. The stack trace named the real one: «Zone Allocation failed» inside FastJsonStringifier, 55 seconds in, heap at 1.4 GB. Four build entrypoints imported surnames.json, so Vite inlined 152 MB into a module's source text, in a compiler arena --max-old-space-size does not govern. Reading it from disk fixed it in one change. Every wrong guess here was reasoning where a measurement was available: the size was in the build log, and the cause was in the stack trace.
France had four providers, and one of them was a link to Geneanet
Done 22 September. David asked whether the atlas had mapped Geneanet's collection catalogue. It had not, and the catalogue cannot be read: it sits behind a Cloudflare managed challenge whose own page carries «noindex, nofollow», and this project does not climb bot gates. But the question exposed the gap behind it. The atlas held four providers covering France — Bas-Rhin, Haut-Rhin, Moselle and a link to Geneanet itself — for a country with one of the deepest register series in Europe. Geneanet is largely an aggregator OF the archives départementales, so the answer was to go to them. 96 of them now, from Wikidata, each with a checked https address and coordinates a median 1.3 km from its own town. Every one is access «unsurveyed», because nobody here has opened 96 archive sites to see what is free and what is not, and marking them «free» because departmental archives usually are would be the map asserting 96 things nobody verified. What was refused is the useful part: two hosts that no longer resolve at all were dropped rather than shipped, which is the ARHiNET lesson from the same morning; two abolished archives — Seine-et-Oise and the Archives de la Seine, départements dissolved in 1968 — were caught by P576, the filter the US county harvest already needed; and six archives alive only on http are held back, because this project's data gate requires https and relaxing a rule to admit one's own data is the wrong way round. Those six need their current address found by hand. Next: 555 archives communales are on Wikidata too, and are a larger and much more variable question.
Belgium hands us surname-by-town, and now the map draws it
Done 22 September. All 565 Statbel communes matched to coordinates through Wikidata's P1567 — an exact code join, not a name join, which matters when every Belgian commune is spelled twice and there are a dozen Sint-Martens- to pick between. Wikidata holds 2,981 Belgian NIS codes and most are deelgemeenten with a letter suffix; the five-digit municipal ones match Statbel exactly, 565 of 565. A surname page for a Belgian name now draws the communes: Peeters is 29,888 people across 411 of them, 2,060 in Antwerpen, and the picture shows it clustering in Flanders. The caption says it is the only country that can be drawn this way, because a reader seeing Belgium detailed and France blank would otherwise read it as a fact about the name rather than about what states publish.
The sitemap 404s that were never there
Not a defect, and recorded because believing it was cost real time. The audit printed «45,829 urls listed, 45,782 pages built» for weeks. I read the gap as 47 urls pointing at nothing, wrote it into this list as «handed to search engines that answer 404», and reported it to David as a defect. listed is every url in every sitemap; built counted only the five kinds that check knows how to walk, so /about/, /surnames/, /coverage/ and the paginated /places/N/ were on one side and not the other. Two definitions, not two numbers. Verified independently: 45,409 listed, 0 without a page. The note now compares like with like.
Poland's scanned metrical books are behind a Vaadin app
David's, 22 September. Genealogia w Archiwach — genealogiawarchiwach.pl — was not here, and it should be: scanned civil registration from the Prussian partition, parish duplicates before 1874, population books from 1837, free to download. It has a provider row now. Its holdings are not harvestable as things stand. Every url returns the same Vaadin shell, robots.txt and sitemap.xml included, and the data moves over a stateful UIDL protocol — there is no crawlable surface to read, which is a different thing from the bot gates this project declines to climb. The route, if there is one, is the Polish archives' own catalogue rather than this front end: szukajwarchiwach.gov.pl is already a provider here and its bulk harvest was declined as a bot gate, so both doors are shut for now. Worth asking NDAP directly whether the fonds list is published anywhere machine-readable — that is an email, like Meertens, not a harvester.
23 September: the letter is drafted — docs/letter-ndap.md, in Polish and in English, for David to send. Two things were checked first so that it does not ask for something already public. dane.gov.pl, the national open data portal, has no Naczelna Dyrekcja Archiwów Państwowych institution and no state-archives dataset at all. And szukajwarchiwach.gov.pl serves an Imperva JavaScript challenge for robots.txt ITSELF, marked noindex,nofollow — so we cannot read the archives' own crawling policy, never mind crawl. That is worth stating in the letter rather than hiding: we stopped at the door, and said so.
What it asks for is the inventory and not the scans — which fonds exist, which archive holds each, which place and which years. That is the layer that turns «Poland» into «write to the State Archive in Poznań for this parish, these years», and it is what this atlas is made of.
Five thousand archives on the map that nobody could click
Done 22 September. data/archives-wikidata.json held 4,929 archives and not one website — every row an empty sites list, because the file came from the generic bounding-box harvester written for cemeteries, which selects a label and a coordinate and nothing else. For an atlas whose premise is «here is the paper and here is who holds it», five thousand dots a reader can see and cannot open is the least useful thing it draws. The client had rendered sites[0] as a link all along; the field was simply never filled. Re-harvested worldwide with P856 and a transitive class net: 10,786 archives across 156 countries, 5,276 of them with a website. Germany goes from 541 to 1,069. Dissolved institutions excluded on P576, the filter the French archives and the US counties both needed. Still unverified and still drawn apart from the curated providers, because nobody here has opened them.
Argentina publishes surnames by province and will not serve them to us
The best find of the portal sweep. RENAPER publishes «Distribución de apellidos en Argentina» under CC BY 4.0 — three CSVs, one of them surnames BY PROVINCE, which is the grain that makes Belgium the richest thing in data/frequencies/. Argentina took a great deal of European emigration and this atlas already holds places there. datosabiertos.renaper.gob.ar resolves and accepts a connection on 443, then never completes the TLS handshake — at 60 seconds and at 180. http 301s to the https that does not answer. Nothing said no; it simply does not reply to us, which may be geography or may be a bad afternoon. Two ways through: retry it in a week, or one person with a working connection fetches three files — apellidos_mas_frecuentes_pais.csv, apellidos_mas_frecuentes_provincia.csv and apellidos_cantidad_personas_provincia.csv. The harvester for Belgium is the template.
Sweep open-data portals rather than guess a country an afternoon
Eight countries were checked one at a time and four of those verdicts were wrong in a single day. scripts/sweep-surname-portals.py asks nineteen national catalogues at once, each in its own language — the word for «surname», not the English one, because missing that is how a sweep quietly finds nothing. First run: Ireland's VSA110, which closed Ireland, and Argentina's RENAPER distribution, which is blocked on its host. Nine portals answered nothing and five errored — 403 from Mexico, 401 from Brazil, a dead TLS version from Chile, a 404 from Australia's api path. Those five are the next work: an error is not an answer, and «the catalogue found nothing» is the exact sentence that was wrong four times, so the output file says as much in those words. It is a FINDER, not a verdict — every hit still needs somebody to open it.
Test the thing end to end, because nothing tests the client
Partly done 22 September — unit layer in, browser layer not. 22 tests on Node's own runner, no dependency to install, wired into npm run check so they gate the build. They pin where the real bugs were: the box order read as x,y pairs, France's frame dragged to Réunion, the clamped height that squashed Belgium, the padding that put a reader inside a Spanish town, and «three countries with no count must not earn a page». Writing them found two things immediately: atlas.js could not load outside a bundler, and node --test test/ dies on a directory in Node 26 — which would have failed on CI's first run. STILL MISSING: the browser layer. Playwright over the built site — search a place, search a surname, click a dot, set a year, a phone viewport — is the half that would have caught every bug David found by looking. It needs a browser download in CI and its own job, not a slot in a build that already takes forty minutes.
DONE 9 OCTOBER, AND THE MISSING HALF HAS BEEN THERE A WHILE. This row's «STILL MISSING: the browser layer … it needs a browser download in CI and its own job, not a slot in a build» is satisfied: 27 Playwright specs across three files — first-thirty-seconds.spec.js, map.spec.js and accessibility.spec.js — in their own workflow, «The first thirty seconds», which installs --with-deps chromium, waits for the site to answer and then drives it. It runs against the LIVE site, not a local build.
AND IT EARNED ITSELF TODAY, TWICE OVER. The row's premise was that a browser layer «would have caught every bug David found by looking». On 9 October it caught one he had not: searching a surname and a place together — «Blazevic Gracisce» — answered «No places of that name here yet» about a place this atlas holds and had just flown the map to, three cold loads in four. Chasing that failure found TWO bugs: the panel was keying the gazetteer shard on the first three letters of the WHOLE query, so it looked for a Croatian village in /g/bla.json; and elsewhere() redeclared var byName, which hoists and shadowed the module's place-name-to-place-id map for the entire function, so the district and commune sections either threw and rendered a blank panel or built their parent link from the value 1 and pointed at /place/1/.
A THIRD THING THE LAYER TAUGHT: THE SPEC CAN BE THE BUG. «second twelve» was failing with «Cannot serialize result: object reference chain is too long» — because page.evaluate(() => map.setView(...)) returns the Leaflet map, and a bare arrow hands it to Playwright to serialise. The map's object graph grew with the cell layers until it crossed the depth limit, so a passing spec turned into a failing one with no change to the site. Braces fixed it. When this workflow mails a failure, read what failed before trusting that the site is broken.
Germany has 1,069 archives on the map and not one provider
Done 22 September. 1 provider to 35: the Landesarchive, Staatsarchive and Landeskirchliche Archive. German civil registration begins in 1876 and the parish registers before it are largely with the church archives, so which of the two holds a record depends on its date and both are now in. Twelve did not answer and are not in. Done with scripts/promote-archives.py, which is general: a country, a name pattern, every URL opened.
Five open-data portals errored rather than answered
Done 22 September. Three of the five were this project's fault, not the portal's: Australia had moved its API under /data/, Colombia is Socrata rather than CKAN and wanted a different path and result shape, and Chile answers curl but not urllib, whose default context refuses the TLS version it offers. All three answer now and none of them publishes surnames — which is a verdict, where before there were three errors being read as one. Chile could have been forced with a lowered SECLEVEL and was not: weakening a TLS client to read a public catalogue is a bad trade, so it was read by hand instead. Mexico still returns 403 to every client and Brazil 401 without an API key. Those two remain UNANSWERED, which is not the same as answered no, and the file says so.
Survey the 204 French archives nobody has opened
France went from 4 providers to 204 in a day and every new row says access «unsurveyed», which is honest and is not the end of the job. Most départementales put their registers online free; some charge, some are index-only, a few have nothing. Until somebody looks, the map cannot colour them, and colour is the whole legend. Sample twenty by hand first to find out whether the answer is usually the same, because if it is, the remaining 184 are a much smaller job.
DONE 9 OCTOBER, AND IT TOOK TWO ROWS NOT ONE. The sampling advice above was right about the départementales and wrong about the communales, because they are not one population. Row 81 surveyed the 95 ad- departmental archives — 89 free, six withheld with a dated reason each. Row 87 surveyed the 108 ac- communales and found no single answer at all: 33 free, 27 onsite, 11 catalogue, 1 mixed, 1 account, and 35 with the reason recorded. Twenty of 203 France rows turned out to have the wrong URL, and five communes had an entire archive portal this atlas did not know about. Sampling twenty would have predicted the first half perfectly and the second half not at all.
Place the books still without a place — 257,524 of them, and most are not villages
AS OF 11 OCTOBER: 194,187 volumes under 13,966 names are unplaced, out of 3,891,358 considered. Everything below is the history of getting there and the figures in it are the figures of their day — this page carried four different totals at once and a reader had no way to tell which was live. The number above is the current one.
Long-running, and the number that most changes what this atlas is worth. Every one is a volume that exists, is catalogued, and has no dot — so a reader whose parish is in that pile is told there is nothing. Continues under the matcher; see row 19.
THE NUMBER IN THIS ROW'S TITLE WAS WRONG BY A FACTOR OF THREE AND HAS BEEN CORRECTED. 799,897 was the 21 September figure; it is 257,524 as of 9 October, across 16,541 names. The title said 799,897 for eighteen days on a page that exists to tell readers what is missing.
The other half of the correction is that the pile is no longer what this row assumes. It opens «Every one is a volume that exists, is catalogued, and has no dot — so a reader whose parish is in that pile is told there is nothing», and that was true when the tail was villages. The top of the list now reads Тверь, Бессарабия, Szabolcs, Toscana, Antwerpen — counties, oblasts, provinces and historical regions, not parishes. A reader whose PARISH is unplaced is a rarer case than this row claims; a reader whose COUNTY is the filing rung is the common one. See row 19 for the table.
THE NAME ON THE SPINE IS NOT THE NAME ON THE MAP, AND THAT IS 57,186 VOLUMES. The largest country in the pile was ITALY — 44,387 volumes — which is also the country with the best gazetteer coverage here, 63,163 places. That asymmetry was the clue: these are not villages nobody mapped, they are names that no longer exist. Traetto was renamed Minturno in 1879, Caccavone became Poggio Sannita in 1921, Pietrafesa became Satriano di Lucania in 1887. FamilySearch files a register under the name on its spine.
AND NOTHING THIS ATLAS ALREADY HELD KNEW THEM. Measured rather than assumed: of 1,411 unplaced Italian names the gazetteer — own names and GeoNames alternates together — knew zero. Minturno's Wikidata item carries no Italian aliases at all. Istat publishes administrative variations and the downloadable file begins in 1991, a century after the renames that matter; the older history is in SITUAS, behind a JavaScript application.
WIKIPEDIA KNOWS THEM, AS REDIRECTS, and a redirect is the encyclopaedia's own statement that two names are one place rather than a resemblance — which is the standard this atlas refuses to drop everywhere else. scripts/match-via-wikipedia.py asks each country in ITS OWN language edition, fifty titles a call, and every coordinate it keeps is put through geo.country_of before it is believed.
| names | volumes | |
|---|---|---|
| placed | 2,263 | 57,133 |
| the ground is in another country now | 493 | 6,031 |
| an admin unit, not a settlement | 81 | 5,432 |
| no article or no coordinate | 11,080 | 147,999 |
| no Wikipedia language listed for that country | 1,274 | 19,167 |
Fed into match-fs-volumes.py as decided placements, that moved the pile from 257,524 volumes under 16,541 names to 200,338 under 14,268 — 2,221 new dots, each carrying the sentence that justifies it: «da.wikipedia redirects «Åbenrå» to «Aabenraa»».
THREE THINGS IT REFUSES TO DO. A region is not a dot: «Toscana» carries 1,001 volumes and resolves perfectly to the Italian region, and putting those on a point would plant them in a field and call it an archive — so admin units are set aside under their own heading. An ambiguous old name lands on a disambiguation page, which has no coordinate, so it is skipped rather than guessed. And 147,999 volumes have no article anywhere, which no matcher can fix and none of this pretends to.
WHAT IS LEFT UNDECIDED AND SHOULD BE. 493 names resolve correctly to ground that is now in another country — Italian names for the Julian March and the Italian Dodecanese, Krnov filed Czech and now Polish, Derryvullan filed Irish and now in the UK. The promotion path takes a place's country from the VOLUME's filing, so feeding these in would create an Italian dot at Slovene coordinates, which is precisely what geo.py exists to prevent. They are kept, with both the country they are filed under and the country they are in, and nothing is placed until that is decided.
AND THE BORDER CASES ARE DECIDED: 223 OF THEM ARE THE SAME PLACE. 541 names resolved to ground that is now in another country, and the first instinct — throw them away — was wrong, because the country check was doing TWO jobs at once. Some of those are a border that moved. Others are «London» filed in New Zealand resolving to London, England, which is the «Lewis must not land on the wrong Lewis» failure this atlas refuses everywhere.
THE CUT WAS MEASURED, NOT PICKED. Forty-five were judged by hand first: Northern Irish parishes filed under Ireland, Luxembourg communes filed under France, Posen for Poznań. The genuine ones run 2 to 174 km outside the country they are filed under. The wrong ones start at 392 — Annapolis at 469, Queens at 400, Salisbury at 7,810, London at 15,079. No overlap, so 180 km is the line, and three named exceptions carry their own reason: St Croix (US territory), Calino (the Italian name for Kalymnos, which Italy held 1912-47), and Islas del Maíz (Corn Island IS Nicaraguan — a Caribbean island the country outline does not reach).
THE DOT GOES WHERE THE GROUND IS. match-fs-volumes.py took a promoted place's country from the VOLUME's filing, which would have made an Italian dot at Slovene coordinates — precisely what geo.py exists to prevent. A placement may now name its own country, so Aghagallon is a GB dot reached by an Irish filing and Altwies is a LU dot reached by a French one. All 222 that landed agree with geo.country_of, and the country pairs are a list of administrations that moved rather than a list of mistakes:
| FR→LU | 88 | Luxembourg under the Département des Forêts |
| IE→GB | 40 | Northern Irish parishes filed under Ireland |
| CZ→PL | 21 | the Silesian border |
| FR→DE | 17 | the Rhineland under Napoleonic départements |
| IT→SI | 13 | the Julian March |
318 names and 2,903 volumes are left unplaced as a different place that shares a name, each carrying the distance that says so.
The provider `checked` dates will start ageing out
Done 22 September. scripts/recheck-providers.py takes the N oldest, opens them and writes the date back; 120 done in the first run and one moved (Archion). A link that stops resolving is reported and NOT dropped — removing an institution is somebody's decision, not a script's, and ARHiNET is why the discipline exists.
Nothing tells a reader when the atlas last changed
Every place page now says when it last changed. The footer's build date could not do this: it moves whenever anything anywhere in the atlas moves, so it tells a reader nothing about THEIR parish.
Fingerprinted on what a reader would notice at that place — its collections, volumes, names, parishes and burials — and deliberately not on the build date or on counts derived elsewhere, or every place would «change» on every build and the line would be noise. Checked in isolation before shipping: a rebuild with identical content does not move the date, adding a field the reader cannot see does not move it, and a volume arriving does.
IT REFUSES TO OVERCLAIM. Until a place has been seen to change, first and last are both the day tracking began, and «not seen to change since 23 September» is a different sentence from «changed on 23 September». The second would be a small lie renewing itself on every build, so the page says the first.
data/_place-history.json is committed on purpose — the whole value is that it remembers across builds, and CI regenerating it from nothing would stamp today onto all 21,473 places every run.
Collect more content: depth where the atlas is already deep
David's, 22 September. Breadth is the obvious gap and depth is the quieter one. Croatia, Istria and Carnia are the ground this project knows best and they are still thin below the parish: no notarial records, no land registers, no school or conscription lists, and those are where a family appears when the church books stop. The Antenati harvest (row 8) is 6,000 of 464,770 volumes and blocked on the Ministry's own server. Matricula is NOT an option and the earlier note here was wrong to imply it: it covers Austrian, German and Polish parishes and its images and descriptive metadata are CC BY-NC-ND. This site carries advertising, so republishing its inventory is exactly what that licence refuses. It is linked, correctly, and will stay linked — see data/_declined.json. Depth here has to come from sources that permit it.
Collect more content: breadth across the 199 countries with nothing
David's, 22 September. 199 of 237 countries have no curated provider. Mexico, Brazil and the Philippines got their first twelve today, which leaves the rest. The order that matters is places-already-on-the-map: Colombia 143, Guatemala 122, Chile 118, the Dominican Republic 109, Honduras 88, Turkey 81, India 73 — every one of them a country where this atlas holds ground and cannot name an archive to write to. Partly done 22 September, and a line drawn. 42 countries gained their first provider from the national-archive layer, taking the directory to 506 across 84. The obvious next step — bulk-promoting every archive the Wikidata layer holds for a country with none — was considered and REFUSED. Indonesia has 158, Israel 128, Japan 80, Georgia 47, and they are archives in Wikidata's sense: corporate record offices, film archives, museum collections. Promoting them would move them from a layer that says «unchecked, and not all genealogical» to the diamonds that mean somebody looked, and the count of curated providers would stop meaning anything. They are already on the map, already clickable since the websites arrived, and already honestly labelled. What the next country needs is the French and German treatment: a name pattern that picks the institutions that hold civil or parish registers, and every URL opened. scripts/promote-archives.py does that and takes about twenty minutes a country.
The Others layer has 5,276 links nobody knows are there
Done 22 September. The label now counts them: «10,786 mapped by others, 5,276 with a website · unchecked, and not all genealogical». A layer worth turning on should say what turning it on gets you.
A surname page for a name with nothing to say
Done, and NOT by relaxing the page rule, which is what this row suggested and would have been the wrong answer. 946,575 names against 23,930 pages means the fix as written is 922,645 new pages built out of one register row each — the textbook definition of a doorway page, on a build that already takes twenty-three minutes for 46,000 and ships into a one-gigabyte ceiling.
Instead: ONE page, /surname/, that answers for every name in the corpus. It reads the same sharded register the map's panel reads — s/<first three folded characters>.json — so there is no new data, no build cost, and a name added to the corpus is answerable the moment it lands without a page being generated for it. Exact match first, then names beginning with what was typed, then names containing it; never the whole shard, which for «bla» is 1,853 names and answers nothing.
The map's panel now links there for the thirty-nine names in forty that have no page, where it used to stop dead. And a name no register has counted is told so plainly, with the point that matters: that is a statement about this atlas's registers, not about the name. Several states publish no surname table at all and a few suppress small counts, so absence here is not absence in the world.
Measure the first paint on a real connection
Done 22 September, and the answer was not the file size. Measured live: first contentful paint 4,500 ms, and index.json was not REQUESTED until 4,821 ms because the map is built inside window.load, which waits for every image, font and CDN script. The 533→419 KB split saved a hundred kilobytes on a request that had not been made. Fetch now starts at parse time with a preload; locally index.json completes at 1,472 ms, before DOMContentLoaded. A budget is not a measurement.
The atlas cannot be corrected by the people who would know
The page and the plumbing are done: set contact.email in site/src/data/site.json and /add/ grows a plain «or just write» route with no account and no template. Left empty deliberately — publishing a personal address on a public site is the owner's decision, and a contact link that goes nowhere is worse than none. A forwarding address would do; it does not have to be a personal one.
Cemeteries and churches are harvested and barely used
Done, and half of it turned out to be done already: the 108,054 churches were wired to place pages as «the nearest parishes», with the distance shown because eight kilometres to Pazin is the useful fact about Gračišće. The 48,482 cemeteries were not, and now are — the nearest five within twelve kilometres, deduplicated between Wikidata and OpenStreetMap on «same name within 300 m», each linking back to the source that knows it. The page says plainly what they are NOT: points on a map, nobody here has visited one, and whether the burial register survives is a question for the parish or the municipality above. Unnamed cemeteries stay out — a shape with no name is not a place anyone can ask about.
One accessibility pass, properly
Partly done 22 September. The panel rewrites itself on every answer and said nothing while doing it — aria-live=polite now, and the result count is its own status region. The map is labelled for what it is: 21,302 dots on one canvas cannot be tabbed through, and making them focusable would be 21,302 tab stops, so it says so and points at the search box and /places/, which reach the same answers in text. That is a fallback rather than an ARIA costume over an unusable control. NOT DONE: a real screen-reader run, contrast checks, and the year slider, which is two overlapping range inputs and almost certainly unusable by keyboard.
Wire the Croatian and Belgian lookups into a re-runnable job
Half of this was already done and the note had not caught up: harvest-croatia-surnames.py stopped reading a fixed list of 466 some time ago and now asks about every name the corpus believes is Croatian, skipping anything already in its scan cache. What was missing was the cadence — nothing ran it.
.github/workflows/surnames.yml runs it weekly, and two details are the whole design. The scan cache is gitignored, correctly, so a CI run would start blind and re-ask every name it has ever asked, every week, at a government form — actions/cache restores it. And HR_MAX caps one run at forty questions, so even with the cache lost the backlog drains over a few weeks instead of DZS being asked eleven hundred questions in a row. Taking by volume what a service offers by question is the thing this project does not do.
IT DOES NOT COMMIT, which is deliberate and follows the link-rot job next door. These are answers about people's names from national services and they should land in the repository because somebody looked at them, not because a cron job had a quiet Wednesday. Results come back as an artifact and an issue with the commands to fold them in.
The atlas speaks English at people researching Croatia
Done for the chrome, which is the first thirty seconds and the part that decides whether somebody stays. Croatian and Italian, 47 strings: the navigation, the search box and its label, the layer switches, the eight access classes, the legend summaries, the opening panel, the loading notice, the phone sheet's way out. The language comes from ?lang=, then a remembered choice, then the browser's own preference, so it works for somebody who never finds the switcher. lang= on the html element follows it.
WHAT IS DELIBERATELY NOT DONE, and the row itself asked for this: the prose. The paragraphs on this site are careful about evidence — what «attested» means, what a grey dot does not mean — and a translated «attested» that drifts is worse than an English one that does not. Those stay in English until somebody who speaks the language reads them, and the footer says so rather than implying a full localisation.
NO HREFLANG, on purpose. There is one URL per page; pointing hreflang at a page that is still mostly English would be a claim this site cannot back. Translated routes are the next step and would triple a 46,000-page build.
21,085 place pages share one social card
site/public/og has six images, one per page TYPE, so every place on earth shares a single grey rectangle when it is shared. A card naming the place, its country, its volume count and its shape on a small map would make a shared link legible before it is clicked — and the machinery exists: lib/surname-map.js already composes SVG at build time, and scripts/build-og-images.py already writes PNGs.
DONE 11 OCTOBER, FOR THE 1,907 PLACES WORTH DRAWING. scripts/build-og-images.py now writes a card per place as well as per type: the place's name sized to fit on one line, its country, how many volumes are on it and the span those volumes themselves declare. It reads places-full.json — the same file place/[id].astro builds its routes from, so a card can never be drawn for a page that does not exist — and the page reads the card DIRECTORY rather than re-deriving the condition, so a build without Pillow falls back to the generic card instead of pointing og:image at a 404.
NOT ALL 54,043, AND THE ARITHMETIC IS THE REASON. A card is 26 KB; every place page would be 1.48 GB against the 2.23 GiB of headroom left after 9 October. 1,907 places have a collection reaching them and the rest are pages nobody shares. Those 1,907 cost 49 MB and 47 seconds, generated in CI and gitignored rather than committed — this repository has already paid once for carrying generated binaries.
NOT DONE: «its shape on a small map». The outlines in site/public/c/ are country-level, so the shape would be the country — a cartouche of Italy behind the word Giustimana, which is decoration rather than information. Left out on purpose rather than forgotten.
The data is CC0 and there is no way to use it
Done. /data/ documents every published file — path, size, shape, what it is and, read out of the file itself, its licence. That last part corrected the first draft: three licences are served out of this one directory, and the OpenStreetMap-derived sets are ODbL share-alike, so telling somebody they were CC0 would have put them in breach. The repository still carries no licence file for this atlas's own compilation; the page says so rather than inventing one.
An archive cannot show its own holdings here
516 institutions are listed and not one of them can put «our records, mapped» on their own site. A small embeddable widget — one archive, its places, its volumes — turns the directory into something the directory's own subjects want to link to.
Nobody has told the 516 archives they are listed
Most carry access «unsurveyed», which means this atlas has published a claim about an institution without asking it anything. Writing to them is both the correction path and the reach: an archivist who is told «you are on this map, here is what it says, tell us what is wrong» usually answers, and sometimes links back. Needs the contact address of row 49.
There is no way to hear that the atlas changed
Done. /feed.xml is an Atom feed of the changelog, one entry per day rather than per commit — a day here is routinely fifteen commits and fifteen notifications is how a feed gets dropped. Advertised in every page's head and linked in the footer.
Surname pages are the front door and are treated as a side room
Done, as far as navigation goes. Surnames are second in Explore and named as the question — «Look up a surname» — instead of fourth under a word that reads like a table of statistics. The opening panel of the map now asks for a surname first as well. The surname pages themselves are still the next thing to deepen.
«Records in Bolivia» is a question nobody here answers well
Done. Every country page opens with what a reader came for: how many sources can be opened tonight, who to write to for the rest, what FamilySearch holds and over what years, whether the jurisdiction layer reaches this ground, and how many places have actually been walked. Writing it caught a real trap — 317 of 516 providers are unsurveyed, so the first draft would have read «2 of 19 sources can be opened» for countries whose other seventeen nobody has ever looked at. Unsurveyed is now its own number and never the bottom of a fraction.
The map opens by asking the reader to already know the answer
Done. The panel opens on «Start with what you have» — a surname, a village by any name it ever had, or only a country — with three live examples (Peeters, Gallignana, Breslau) that run the search on click, and one sentence saying the colour is what it costs to look.
Nothing points at the atlas from where genealogists already are
Cyndi's List, the FamilySearch Wiki's country pages, national genealogical society link lists — these are where people looking for exactly this already go. Being absent from them is a bigger gap than any ranking problem.
Play the years, because that is the whole argument
Done. A play button runs the year window forward in ten-year steps from 1500, fifty years at a time, redrawing the jurisdiction shading and refiltering the shelf at every step. p starts it; Escape, or touching either slider, stops it.
21,302 dots at world zoom is a grey smear
Done. Below zoom 6 the shelf is drawn as one bubble per grid cell — 12° cells at zoom 2 down to 2° at zoom 5 — area by count, tooltip naming places AND volumes so the bubble cannot be mistaken for a book count. The individual markers come off the map entirely while bubbles are up, because a cluster drawn over the dots it summarises is just more ink.
A surname and a place cannot be asked about together
Done. «Blazevic Istria» is read as a name and a place: the query is split only when the whole of it matches no place, so nothing that works today changes, and the panel answers both halves at once. It refuses to conflate them — a register counting 11,163 people in Croatia is a fact about Croatia and not about the four villages on screen, and the panel says which places sit in an attested country and leaves the rest unmarked.
The panel is a wall of text on a phone
Done. Under 700px a result becomes a sheet over the bottom of the map with its own way out, instead of filling a box below the fold that the reader never sees. Driven off :has() on the body's hidden attribute, so there is no state to keep in sync and a browser without :has() keeps today's stacked panel.
A place's sources cannot be taken away from the screen
Done. A print stylesheet in the shared stylesheet — chrome, ad slot and map gone, every link followed by the URL it goes to, tables that do not break across a row — and a «Print this plan» button on every place page. The links mattered most: a printed «FamilySearch» that does not say where it goes is no use to somebody typing it back in.
The map is blank while it loads and says nothing
Done. The map says «Loading the atlas…», then «Drawing 21,302 places…» the moment the index arrives, and the notice is removed when the markers are on. A number rather than a spinner: it tells the reader the wait is proportionate to something.
The legend lists eight colours and teaches nothing
Done. The legend now states the two things that make it readable: that the colours run from what you can open tonight down to what needs a letter, and that a place takes the colour of the MOST OPEN thing it has — so red means red is the best on offer there, and green says nothing about what is underneath. It also says the swatches are filters, which was undiscoverable.
No keyboard shortcuts on a tool people will use for hours
Done. / focuses the search, y toggles the year, p plays the years, Escape clears the box or closes the phone sheet. None of them steal a keystroke from a field.
The jurisdiction layer covers 33 regions and is the reason to be here
Seven more record regions, 33 → 40. The Military Frontier is the one that earns its place: from 1746 a Lika or Slavonian village belonged to its REGIMENT, not to a county, so its registers were kept by the regimental chaplain and filed under Ogulin, Otočac, Slunj, Gradiška, Brod or Petrovaradin — and a search of the county archive for anything before 1881 finds nothing, which is not the same as there being nothing. Also the Netherlands (civil registration from 1811, and 1796 in the parts annexed early), Denmark (two parish registers kept separately from 1814, so a gap in one is usually survivable in the other — and Schleswig filed under Prussia from 1864 to 1920), Norway, Sweden (the 1686 church law and the husförhörslängder), Finland (the Karelian parishes' registers were evacuated WITH the people, so they are in Finland and not where the places are) and Portugal (civil registration only from 1911).
AND BELGIUM, added after checking the feature on a real name rather than the one it was built for. «Peeters» is about as Belgian as a surname gets, and its page said «no record region is drawn yet for Belgium» and then filled the screen with Bohemia, Pomerania, Acadia and Alsace — every region in every country where some register had counted a Peeters. Honest, and useless. So: nine eras for Belgium and the Southern Netherlands, whose hinge is 1796 — France annexes the ground in 1795 and civil registration begins the year after, so before that date there is no state record of a birth anywhere in the country, only a baptism. Its trap is that there is no central archive to write to: the State Archives keep the registers in thirteen provincial repositories, and a request to Brussels about a Limburg parish goes to the wrong building.
The list is also capped at five countries now, ordered by how many people a register counted — so the countries a name actually belongs to come first and the tail is a count rather than a wall. 41 regions.
The first thirty seconds are never rehearsed
Done, as far as a machine can do it. test/e2e/first-thirty-seconds.spec.js walks the path a stranger takes, in order, and asserts each step gives them something to act on: second one the map says what it is doing, second five the panel asks for something they actually have, second ten the world is bubbles rather than a smear, second fifteen a surname answers with ONE heading, second twenty a name and a place are asked together, second twenty-five the panel is not still answering the previous question, second thirty there is a way out. Plus the phone, where the answer must be a sheet and not a box below the fold. It is not a substitute for watching a real stranger — it is a rehearsal, so the same ground is not lost twice.
The archive-kit pin is 156 commits behind, and holding it is a decision
Raised by the archive-kit session on 23 September: site/package.json pins TheArchiveKit#8174676 (16 Sep), kit HEAD is 2790d85, and the eight family archives are all on HEAD. It flagged five living-gate commits missing from this pin, c1f9958 above all — «the living check read only HTML, and an atlas serves its people as JSON» — on the grounds that a gate which cannot read JSON will report clean either way.
CHECKED HERE BEFORE DOING ANYTHING, and the answer is that neither half applies to this repository.
No gate runs in this build at all. npm run check is check-data.py and the node tests; CI runs the build and audit-site.py. checkliving is not invoked anywhere, so there is no gate to be out of date. The kit is used for exactly one thing here: WorkList.astro, which renders this page.
And there are no people to gate. This atlas has no GEDCOM, no people file and no living flag — the only string «living» in the whole corpus is the surname Living, 280 of them in the American register. archive-surnames.json states the boundary in its own header: surname strings only, no given name, no date, no place, no relationship, no identifier. The one place this could approach an individual is the Belgian commune grain, 510,811 rows of surname-by-municipality — and Statbel suppresses under five, so the smallest number published anywhere on this site is 5.
So the pin is not a privacy risk. It is still drift, and it reads as drift from outside, which is why this row exists rather than a shrug. Repinning is 156 commits of change to a shared dependency, and the kit session's own warning is that eight gate tools did not exist at this pin and should be expected to fail on first run with real work behind them. That is David's call on his own estate, not one to make blind from here, and doubly not with two deploys in flight. If it is repinned: 2790d85 or later, never 337a353, which raised NameError in checkliving for twelve minutes this morning.
The one thing worth lifting regardless of the pin is c1f9958's LESSON rather than its code: if this atlas ever publishes a person as JSON, the gate has to read JSON. Today it publishes none.
Attribution was met by memory, and memory failed twice
/data/ shipped in the morning saying «all CC0» over OpenStreetMap data. Corrected, and by the afternoon it said «there are three licences» over seven. Both were caught by a person looking, which is not a system, and the second only because the archive-kit session asked about something else entirely.
So check-data.py now refuses the build on three rules. A data file that names a source must declare a licence FIELD — two of them stated theirs in an English sentence inside source, where no machine could read it, which is why the page printed «none declared» over CC BY data and was not wrong to. Every licence in data/ must be named on /data/, which is the page that tells a reuser what they are taking on. And every licence asking for attribution must name its holder in the site footer, because CC BY asks for credit wherever the work is used and not only where it is catalogued.
It found twelve on its first run, and one of them mattered more than the licence question that prompted it: every volume title in every p/<id>.json is FamilySearch's catalogue metadata — «Defunciones 1980-1989» is their cataloguing, not our writing — and the page was filing those files under «this atlas». The fact that a collection covers a place is not theirs to own and is the whole point of this project; the strings are theirs. The page says so now.
The rule it cannot enforce is whether the credit is adequate. It checks that a name appears, and does not pretend to judge more.
Its own first version ran AFTER the loop that prints failures, so twelve findings were counted into the exit code and never shown. A gate that stops a build without saying why is worse than no gate.
Three filters, three different shapes, and two of them hidden in full screen
David: «could things like the filters rather be a multi select dropdown? its got really messy. similarly the «what it costs» and «what kind of record» dropdowns? maybe same consistency?»
He was right on both counts. The layers were eight loose pills in a row while the other two were tidy disclosures reporting how many were on — so the control a reader touches most looked like scattered chrome beside two that looked designed. All three are the same control now, each stating its own state in its summary, so a closed one still answers «what is on the map».
AND IT SURFACED THE SAME BUG HE HAD JUST REPORTED, two controls further along. «What it costs» and «what kind» lived under the map, outside the element that goes full screen, so full screen hid them exactly as it hid the search box. All three are in the bar now, which full screen carries.
They are popovers rather than fold-outs: an inline disclosure pushed the map down every time somebody looked at a filter, and in full screen it would have changed the height of a bar whose height is measured.
TWO THINGS ONLY MEASURING CAUGHT. The popover rule appeared to work while the dev server was serving a stylesheet frozen at startup — the toggling all behaved, and only reading the computed position showed it was still static and the map was still moving. And the first attempt at keeping the last dropdown on screen flipped it to right-aligned under 900px, which pushed it 49 pixels off the LEFT edge: the bar WRAPS, so that control is at the right-hand end at one width and the left-hand end at another, and no breakpoint knows which. It is measured and nudged now, and verified on screen at 1000, 820, 640, 480 and 380 pixels.
And the count moved up with them, at David's asking. It reported on those filters and sat nowhere near them — under the map, which is outside the element that goes full screen, so the one line saying whether a filter did anything was missing in the mode where the map is biggest. It sits at the right-hand end of the bar when there is room and takes its own line when there is not, which matters because it is not always three words: set a year nothing can answer for and it becomes a sentence explaining why the map is empty, with a way out. Checked in that state — the bar grows one row, the buttons stay put, the map keeps its 600 pixels.
That emptied .ra-foot entirely, so it is gone.
The colour legend promises a spectrum the data does not have
David asked «i dont even see much for eg. costs, almost nothing for most categories? can you check as well», and the count is worse than «not much». Of 21,473 places: unsurveyed 19,416 (90.4%), account 2,029 (9.4%), onsite 19, catalogue 6, free 3 — and index, mixed and paid are ZERO. Eight categories, three of which cannot remove anything and three more that remove almost nothing.
The counts are on the filter now, and the three empty ones are drawn dead and disabled, so the control no longer offers choices that do nothing. That is honesty, not a fix.
THE FIX IS SURVEYING. unsurveyed is not a property of the world, it is this atlas not having looked: 317 of 519 providers carry no access class, and a place takes the best class among the collections reaching it. Row 40 (the 204 French archives) is the same debt in one country. Until that is paid the map is effectively two colours — grey for «nobody looked» and green for «FamilySearch, account needed» — and the eight-colour legend describes an atlas that does not exist yet.
Same for record kind: only 1,727 of 15,780 loud places carry any kind bit, so any kind filter hides seven places in eight. The counts say so now.
9 OCTOBER: THE THREE EMPTY CATEGORIES ARE NO LONGER EMPTY, AND SURVEYING IS WHAT DID IT. This row said index, mixed and paid were ZERO and had to be drawn dead on the filter. Measured now, across 555 providers: free 310 · unsurveyed 117 · catalogue 46 · onsite 43 · mixed 11 · index 11 · paid 9 · account 8. Every one of the eight classes can now remove something.
The ones that filled them are worth naming, because they are what the legend was always for:
mixedarrived with Cannes, which says its documents are «consultables sur place et EN PARTIE sur Internet» — the honest middle the atlas previously could not express.paidis now mostly German church books: Kiel states «Das Landeskirchliche Archiv ist Mitglied im Kirchenbuchportal Archion», and Bayern announces its Nuremberg card indexes going onto Archion in July 2026. Archion is a subscription. That is a real paywall on the single most wanted German source, and the map can say so.accountgained Vichy, which is not a sign-up at all: you e-mail the service and it posts you credentials.onsitewent from 19 to 43, almost entirely French communes and German church archives that state their own terms rather than merely lacking links.
AND unsurveyed IS NO LONGER 90% OF ANYTHING. Providers: 60% unsurveyed when row 80 measured it, 21.1% now. Better than the headline: of the 117 that remain, 62 carry a dated reason — refused, timed out, no page published — so the population that has genuinely never been looked at is 55 providers, not 117. The place-level counts this row quotes have not been remeasured and are still the old ones.
9 OCTOBER, MEASURED AGAIN AND THE DIAGNOSIS IN THIS ROW IS WRONG. «THE FIX IS SURVEYING» HAS BEEN DONE AND THE MAP HAS NOT CHANGED COLOUR BY ONE PIXEL.
Provider access classes, after France, Bulgaria and Germany: free 310 · unsurveyed 117 · catalogue 46 · onsite 43 · mixed 11 · index 11 · paid 9 · account 8. A real spectrum.
Place access classes — which is what actually colours the map — measured across every shard in site/public: unsurveyed 41,916 (95.5%) · account 1,969 (4.5%) · onsite 19 · catalogue 6 · free 3. Compare those last three with the numbers at the top of this row: onsite 19, catalogue 6, free 3. Identical. Nothing moved.
WHY: THE MAP NEVER CONSULTS THE PROVIDERS. A place takes its colour from its COLLECTIONS (scripts/build.py, line 748):
acc = min((c["access"] for c in cs if c.get("access") in RANK),
key=RANK.index, default="unsurveyed")
and not one of the 3,504 collections carries an access field — every one is None, so every place with walked volumes falls through to default="unsurveyed". The 1,969 account places are not surveyed either; they are the division fallback at line 785 defaulting to "account" because c.get("access") is None.
AND THE JOIN ALREADY EXISTS, UNUSED. Line 405 builds paccess = {p["id"]: p["access"] …} — provider id to access class — and grep -c paccess scripts/build.py returns 1. It is computed and dropped. Every collection in the file joins to a provider by URL host: 3,504 of 3,504, 100%.
BUT WIRING IT WOULD NOT PRODUCE A SPECTRUM EITHER, AND THIS IS THE REAL FINDING. All 3,504 collections are one host: www.familysearch.org. The 310 providers now marked free hold ZERO collections in this file. Every French départementale, every Bulgarian state archive, every German Landesarchiv is a dot with coordinates and an access class and NO HOLDINGS. So inheriting provider access would recolour roughly 41,916 places from grey to account — which is honest, because FamilySearch does require one, and it would stop the map saying «nobody has looked» about places this atlas has walked volume by volume — but the map would then be two colours instead of two colours.
SO THERE ARE TWO JOBS HERE AND THIS ROW HAD THEM CONFLATED:
1. Wire paccess through to collection access. Small, self-contained, and it removes the map's largest remaining untruth — 41,916 dots that say «unsurveyed» about collections the atlas has read. Honest, not pretty.
2. Harvest holdings from the 310 free providers. This is the only thing that can put free on the map in quantity, and it is rows 44, 45 and 56, not this one. Surveying told us WHERE the free paper is; it did not fetch any of it.
The lesson for the surveying rows: an access class on a provider is a fact about an institution. The map colours GROUND. Those are joined only by holdings, and holdings are the part nobody has harvested outside FamilySearch.
CORRECTION, SAME DAY. THE PARAGRAPHS ABOVE ARE WRONG AND THE MAP IS NOT LYING. I measured data/collections.json, found no access on any of its 3,504 rows, and concluded that 41,916 grey dots were a dangling join. They are not. Two different things are called a collection in this build:
- A place's OWN collections, written by the harvesters into
data/places.json, already carryproviderandaccessand always did. Measured: familysearch 7,319 entries (account) · dapa 1,151 (onsite) · antenati 74 (free) · beweb 13 (catalogue) — FOUR providers with real holdings, not one. These are what colour a dot. - The coverage claims in
data/collections.json— «Italy, Napoli, Civil Registration, 1809-1936» reaching a town it does not stand in — carried no access, and were only ever used for the division fallback, which guessed «account».
So the 95.5% grey is honest: those places have no collections at all. 1,907 of 7,391 places in places.json have any, and the map's 43,913 classed dots are mostly gazetteer places the atlas has never read. «Unsurveyed» is the correct word for them.
WHAT WAS ACTUALLY WIRED, AND IT CHANGES NO COLOUR TODAY. paccess was still dead code and the division fallback was still guessing. Coverage claims now inherit their holder's class by URL host — 3,504 of 3,504 join — and the or "account" guess is gone, replaced by the real value with an honest grey default. Every colour is identical afterwards, because every coverage claim in the file is FamilySearch's and «account» is what the guess already said. The gain is that it stops being a coincidence.
AND THE REAL FINDING IS THE ONE I ALMOST MISSED: onsite HAS 1,164 COLLECTION ENTRIES AND NINETEEN PLACES. A dot takes the BEST access among its collections, and account outranks onsite, so wherever FamilySearch also reaches, the 1,151 dapa collections — material you have to travel for — are invisible. The map says «account» and never mentions the rest. Same for free: 74 entries, 3 places. That is defensible for a reader asking «can I get at this» and it is NOT a spectrum problem that surveying can fix. If this row wants eight colours doing work, the question is whether a dot should show its best access or its RANGE of access — and that is a design decision, not a harvest.
11 OCTOBER: THE FIX THIS ROW NAMES IS THE WRONG FIX, AND SAYING SO IS THE POINT OF A WORK LIST. The row says «The fix is surveying». Surveying has largely happened — 314 of 559 providers are now classed free — and the map has not moved by a single dot. Measured across every dot the map can draw, all 43,913 of them in index.json and the 174 viewport cells:
| colour | dots | share |
|---|---|---|
| unsurveyed | 41,916 | 95.5% |
| account | 1,969 | 4.5% |
| onsite | 19 | 0.0% |
| catalogue | 6 | 0.0% |
| free | 3 | 0.0% |
| index · mixed · paid | 0 | — |
WHY, TRACED RATHER THAN GUESSED. A place takes its colour in build.py from one of two things and nothing else: the best access among the COLLECTIONS on it, or — when it has none — the best access among the COVERAGE CLAIMS that reach it on its own chain. So:
- every collection on the map comes from four providers:
familysearch7,319 (account),dapa1,151 (onsite),antenati74 (free),beweb13 (catalogue); - and every one of the 3,504 coverage claims is FamilySearch's, which this file's own comment already records: «every coverage claim in the file is FamilySearch's … the next provider to publish coverage claims gets its own class instead of FamilySearch's».
Those five numbers are the whole legend. index, mixed and paid are not rare on this map, they are STRUCTURALLY UNREACHABLE: no provider carrying those classes has a single collection or a single coverage claim, so no dot can ever take them. The three empty swatches drawn dead and disabled are not a data gap waiting on a survey; they are a join that does not exist.
SURVEYING CLASSIFIES A PROVIDER. THE MAP COLOURS A PLACE. Nothing connects the two unless that provider also says which ground it covers. 314 free providers are surveyed, listed, linked — and invisible, because not one of them has told this atlas where its holdings are. That is also why row 8's stalled Antenati harvest looked so load-bearing: Antenati is the only free source with holdings attached, and it is the reason the free count is 3 rather than 0.
SO THE WORK IS COVERAGE CLAIMS, NOT MORE SURVEYING — a claim is «this provider covers this ground», which is metadata rather than a harvest, and it is what lights a dot. Row 40's French archives and the Bulgarian and German sweeps were not wasted: they are the prerequisite. But another afternoon of classing providers will not change one pixel until the claims exist.
AND THE SECOND QUESTION, ASKED PROPERLY RATHER THAN GUESSED FROM FIVE. If the work is coverage claims, the next thing worth knowing is which free providers would let this atlas read them at all. The first five tried — ranked by how many drawn places their countries hold — came back four shut: the NLI behind a Cloudflare challenge, IrishGenealogy 403, Hungaricana 403, Matricula naming ClaudeBot in robots.txt under Disallow: /. That looked systemic. It is not, and a sample of five chosen by size is a sample of big national institutions, which are exactly the ones behind a WAF.
scripts/sweep-crawl-policy.py read all 314, robots.txt first and the front page only where robots permitted it, under a user agent that names this atlas and carries a contact. Nothing was crawled. -> data/_crawl-policy.json
| open | 133 |
| refused (401/403) | 127 |
| robots forbids everyone | 31 |
| robots forbids AI crawlers by name | 11 |
| bot gate | 6 |
| nothing answered | 6 |
133 of 314 would serve us, and the biggest single win is open: the National Archives of Ireland, whose ground holds 2,130 of this atlas's drawn places. FIBIS (1,284), The Gazette (1,247), Hrčak and NSK digitalna (1,165 each) follow.
ELEVEN NAME AI CRAWLERS EXPLICITLY and are therefore closed to this session whatever user agent it wore: Matricula, Digitalarkivet, BAnQ numérique, the National Archives of Australia, Gazettes.Africa, eGGSA, the Zentral- und Landesbibliothek Berlin, Delpher, the SS. Cyril and Methodius National Library, and the departmental archives of Gironde and Mayenne. Several name claudebot or anthropic-ai outright. Their access class and their link stay — a reader can still open them, which is the point of listing them — but this atlas will not read them, and now does not have to rediscover that one at a time.
THE FIRST JOIN IS BUILT, AND IT MOVED 2,322 DOTS. scripts/harvest-nai-census.py turns the National Archives of Ireland's census into coverage claims: one row per county per census year, carrying the record count the archive's own API reports for it. 67 claims, 8,830,076 records. The browse page is a JavaScript app but it is driven by api-census.nationalarchives.ie, whose robots.txt asks a one-second crawl delay and disallows only /admin — so this is one request per year, not a scrape.
| colour | before | after |
|---|---|---|
| free | 3 | 2,325 |
| account | 2,032 | 2,032 |
| onsite | 19 | 19 |
| catalogue | 6 | 6 |
| index · mixed · paid | 0 | 0 |
| unsurveyed | 51,983 | 49,661 |
AND A CORRECTION TO THE TABLE ABOVE THIS ONE. It said 43,913 dots and 95.5% unsurveyed. That was the LOUD half only — index.json and the 174 viewport cells — and left out index-quiet.json, the 10,130 places with no volume and no collection that the map fetches afterwards. The whole map is 54,043 dots and was 91.9% unsurveyed before this. The shape of the finding was right and the denominator was wrong, which is the kind of thing this file exists to catch.
1821-1851 ARE DELIBERATELY ABSENT. The same browse tree offers them; the API returns no county breakdown, because those censuses survive as fragments. No evidence of which ground they cover, so no claim — the fragments are still findable on the archive's own site.
Three namings are written into the file rather than applied quietly: King's Co. and Queen's Co. are the 1901 and 1911 names for Offaly and Laois; North Mayo and West Cork are NAI's own 1911 splits of counties it does not split in 1901; and Tryone is a misspelling of Tyrone in NAI's data, carried as a separate facet with 11,470 records behind it.
WHAT THIS PROVES AND WHAT IT DOES NOT. It proves the diagnosis: one holder's claims, built in an afternoon, took free from three dots to 2,325. It does not finish the row. index, mixed and paid are still zero, and 49,661 dots are still grey. The next 132 open providers are the rest of the job.
A SECOND HOLDER, AND THE FIRST index DOT THIS MAP HAS EVER DRAWN. Open Archieven — the open API behind more than two hundred Dutch and Belgian archives — as 193 claims carrying 339,854,071 records.
| colour | before Ireland | after Ireland | after the Dutch |
|---|---|---|---|
| free | 3 | 2,325 | 2,325 |
| account | 2,032 | 2,032 | 2,032 |
| index | 0 | 0 | 2,481 |
| onsite | 19 | 19 | 19 |
| catalogue | 6 | 6 | 6 |
| mixed · paid | 0 | 0 | 0 |
| unsurveyed | 51,983 | 49,661 | 47,180 |
The Netherlands is 1,520 of 1,712 dots; Belgium 956 of 2,676.
WHY IT IS index AND NOT free, WHICH IS THE WHOLE POINT. A record there is a name, a date, an event and a shelf reference — «Nationaal Archief, Archive 2.12.14, Registry 488» — and no image. Checked by fetching a record in full, not assumed from the homepage. That is exactly what this atlas means by index: «An index only — no images». The swatch was drawn dead because nothing in that class had ever made a claim, not because nothing is indexed.
THEIR RULES, OBSERVED. «Open Archieven is gebouwd op open data.» The Spelregels ask for no more than four requests a second, a recognisable program name, and that the HOLDING ARCHIVE be named as the source. So every claim is titled with its archive and that name travels to the place page, which is where a reader meets it. Two requests per archive, 193 archives, and no record fetched.
WHAT IS TRUNCATED, SAID PLAINLY. The API stops reporting distinct places at 200, which 107 of 193 archives reach — the Brabants Historisch Informatie Centrum covers all of North Brabant and is given 200. The other 86 are complete. A truncated archive is represented by the places it holds the most records for, so a dot that misses out is one it barely touches. (The first draft of this harvest reported all 193 as capped: the check was testing «did we get every group», which is true for a small archive too.)
THE FRENCH TAIL IS A TRAP, AND WORTH WRITING DOWN. 111 of the 133 open providers are French departmental archives — and France has 190 dots on this map, 143 of them grey. Ranked by open providers, France looks like the whole job; ranked by grey ground, it is 0.3% of it. The order to work in is grey dots, not provider count.
mixed and paid are still zero and 47,180 dots are still grey.
AND WHY account HAS NOT MOVED, WHICH IS NOT A BUG. With two holders added it was worth asking why the ORIGINAL 3,504 claims colour only 2,032 dots when they name 111 countries. They name 223 distinct places between them, and every one is coarse:
| place type | claims |
|---|---|
| COUNTRY | 1,780 |
| STATE | 1,626 |
| PROVINCE | 100 |
| DEPENDENCY | 49 |
| DISTRICT · DEPARTMENT · other | 37 |
Not one municipality, not one town. And reach() refuses to colour a dot from a country-level claim on purpose — rank 4 is «the country has a collection» and says nothing about the ground under the dot, which is the Aargau lesson written at the top of that function. So 1,780 of the 3,504 can never colour anything, and the rest land only where a state or province name matches a dot's own adm1.
2,032 IS THAT CATALOGUE'S HONEST CEILING, not an unfinished job. FamilySearch files by province — «Italy, Napoli, Civil Registration» — and a province is not a town. There is no shortcut hiding in the data already here: every further colour has to come from a holder that publishes at the grain of a place. Ireland gave counties, Open Archieven gave 25,494 place rows, and that is why those two moved 4,803 dots between them while 3,504 claims sat still.
WHAT THE REMAINING GREY ACTUALLY NEEDS. 47,180 dots, holder by holder, each with its own route and its own terms. By grey ground the next are Ukraine (1,674), Canada (1,042), Denmark (1,001), Portugal (787), Germany (656), Poland (649) and Belarus (502) — and Ukraine's only open provider is Szukaj w Archiwach, which is already declined as bot-gated, so that 1,674 has no route at all today.
AND THEN THE SAME SOURCE AGAIN, ASKED BETTER: 2,481 TO 2,925. The first Dutch pass took one answer per archive, and 107 of 193 came back at the API's 200-place ceiling — so the LARGEST archives, the ones covering the most ground, were the worst served. The fix is to ask narrower questions rather than more of them: any answer that comes back full is split in half by year and asked again, recursively, until a slice answers under the ceiling. On the Brabants Historisch Informatie Centrum that turns 199 places into 765.
Belgium was where the headroom was, and that is where it went: 960 dots to 1,334. The Netherlands moved 1,521 to 1,591, because it was already nearly done.
TWO THINGS THAT PASS FOUND IN THE WORK ALREADY PUSHED.
A mention is not coverage. Splitting turned Archief Delft from 199 places into 1,398, and the bottom of that list was Cairo, Cardiff, Javazee and Hydaspes with one record each — a sailor who died in the Java Sea, somebody born in Wales. All true, none of it coverage. A claim here colours a dot and tells a reader this is a place to go and look. The unsliced top-200 had hidden this by accident, because it was already a cut by count; splitting removed the accident, so the rule had to be stated: a place needs at least ten records to count as covered. That is a judgement and it errs toward saying less — a hamlet whose parish genuinely left eight entries is dropped with the strays.
And every one of the 193 claims already pushed had no dates at all. Not because the API lacks them: because this harvest asked with sort=name_asc, which makes that endpoint answer 503 «Service temporarily unavailable» EVERY time, while the default sort answers 200. It was asked for in ascending order so the first and last could be taken, which min() and max() never needed. The script got nothing back 193 times and wrote from: null, to: null into every row without complaining. 188 of 193 now carry a span; Archief Delft is 1620-1945.
DENMARK: 811 DOTS, AND IT WAS NINE REQUESTS RATHER THAN THE SEVERAL HOURS THIS ROW PREDICTED. The earlier note here deferred Rigsarkivet as expensive — «the open half has no API: parish-level coverage would mean crawling down through themes to something like 2,100 parishes». That was wrong, and wrong in a way worth recording: each geographic collection embeds its ENTIRE inventory in the page as var data = [...] to feed two combo boxes. The nationwide church books are 2,999 rows in a single 241 KB page. Nine pages hold church books, civil marriages, probate, fire insurance and property registers — 139 claims over 11,280 place rows.
The split that mattered still held: soeg.rigsarkivet.dk is the search and answers Disallow: /, so it was not read; arkivalieronline.rigsarkivet.dk is the viewer and answers Disallow:, so it was.
NO RECORD COUNTS AND NO YEAR SPANS, because the pages carry neither. Ireland's claims have both. records: 0 here means NOT PUBLISHED, and the file says so rather than letting a zero be read as «none».
| colour | this morning | now |
|---|---|---|
| free | 3 | 3,142 |
| index | 0 | 3,083 |
| account | 2,032 | 2,039 |
| onsite · catalogue | 19 · 6 | 19 · 6 |
| mixed · paid | 0 | 0 |
| unsurveyed | 51,983 | 48,293 |
AND A SUBDOMAIN INHERITED NOTHING, WHICH WOULD HAVE COST ALL 811. A claim takes its access class by matching its URL's host against a provider's. Rigsarkivet is listed at www.rigsarkivet.dk and browses at arkivalieronline.rigsarkivet.dk, so all 139 claims matched no provider, carried no class, and would have coloured not one dot. The claim was right, the provider was right, and the join between them was a string comparison on a hostname. It now falls back up the domain one label at a time, stopping at two so co.uk can never match. Measured before it was kept: exactly 139 claims newly resolve and zero existing claims change class.
AND THE BUILD REPORTED SKIPPING A FILE IT WAS USING. The claims loader read counts.records directly, after already adding the claims — so a holder that publishes no counts raised a KeyError, the log said «not usable — skipped», and the 139 claims were in the build anyway. A report that contradicts the build is worse than no report.
Remember the searches somebody already made
David: «is there a way to remember the last searches someone made? and show them as quick pills?» A researcher works down a list of parishes and comes back to the same three or four names all week, and every visit started from an empty box.
The last eight, most recent first, under the search box. In localStorage and nowhere else — no account, no server, nothing that leaves the machine, and a «Forget» that empties it. Every read and write is wrapped, because private windows and blocked site data throw on access rather than returning empty, and a map that would not draw because it could not remember a search would be a poor trade.
REMEMBERED ONLY WHEN THE SEARCH WORKED, which took two goes. The first guard was «is a panel showing», and a search that finds nothing still shows a panel — saying «nothing here, and here is why» is the honest answer — so «zzzzqq» went straight into the list of things that worked. The second version used a real flag, and then dropped «Blazevic»: a place match is decided 140ms after the last keystroke, but a surname match waits on a shard fetch that lands long after any reasonable timer. So the timer catches the fast answers and the slow ones report themselves when they arrive, as long as the reader has stopped typing.
23 September, later: moved off the bar entirely. It was a standing row of real estate for something wanted at the start of a visit and never again, so it is a dropdown under the search box now — on focus, gone the moment you type, because these are the searches you made and not suggestions for the one you are making.
Derive our own surname origins, because Wiktionary answers 2.8% of them
Done 30 September: 2.8% to 98.0%. scripts/derive-surname-origins.py reads 940,562 surnames and hands back an origin for 921,288 of them (98.0%) — 86,833 strong, 583,554 fair, 250,901 weak. Wiktionary's 25,972 rows are still Wiktionary's, kept separate and still labelled as its own; these are derived and say so.
IT IS NOT AN ETYMOLOGY AND THE FILE SAYS SO IN ITS FIRST SENTENCE. It does not know what a name means. It knows where the name is now, where it was first recorded, and whether a place carries it too, and every row carries its own basis — the working, not just the answer. That was the condition this row set for itself and it is the reason the panel can print «first attested in Croatia in the 1710s, and also a village in Istria» rather than «Croatian».
Three bugs in the derivation were found by David's own examples rather than by me: a three-letter-language short-circuit that answered before the evidence was read, ties broken by dictionary insertion order, and register counts outweighing everything else — which is why Defranceschi came back as anything other than Italy. The origin now also sets where the map opens when a surname is chosen.
Cost: the first cut was a 197 MB JSON file. Structured and gzipped it is 8.3 MB, the same trick and the same reason as the gazetteer.
WHAT THIS ROW DID NOT CLOSE is carried on as row 79: the 52,515 names with a Wiktionary entry and no etymology in it, where the panel still says nothing at all.
Say so when Wiktionary has the name and no etymology for it
Closed 8 Nov 2026: the surname page already says «No origin given» for an entry that exists and names nothing, with the count (52,640 of 78,628), and states it rather than linking. This row was left open after that shipped.
Two different silences, and only one of them is written. 52,515 surnames have a Wiktionary entry that contains no etymology. «The dictionary holds this name and does not say where it is from» is a finding; «the dictionary has never heard of it» is a different one; the panel currently prints nothing for either, so a reader cannot tell which they are looking at — and for the first group there is a page worth opening.
Cheap and bounded: the entry is already known to exist, so this is a label and a link, not a harvest. It is deliberately NOT an invitation to quote Wiktionary prose — David on the reference itself: «Wiktionary shouldn't be a reference or link, it's far too small; just harvest the origin of the name.» What goes on the page is the absence, stated.
Grow the corpus beyond Italy and the USA — and France is the whole answer
David, on the headline «21,754 places · 2,893,805 record volumes · 528 archives in 83 countries · 41 record regions»: «how do we break that and push it larger?» Measured first, because three of those four numbers are less than they look.
THE VOLUME COUNT IS ONE COUNTRY. Italy is 46.4% of all 2,893,805 volumes. Italy and the United States together are 67%. Nine countries hold 1% or more; twenty countries that have a place on the map hold fewer than a hundred volumes. Adding Italian places moves the headline and almost nobody's research, because the shelf there is already deep.
THE PLACE COUNT IS MOSTLY CONTEXT-FREE. 21,754 places is 16,058 drawn plus 5,696 quiet — and only 8,474 of them (39%) sit inside one of the 41 record regions. Parishes, burial grounds, the era table and «who governed this ground» are all region-scoped, so 13,280 places (61%) can never show any of it, whatever is harvested. A place outside a region is a dot with volumes and no story.
THE ARCHIVE COUNT IS MOSTLY UNOPENED. 329 of 547 providers are unsurveyed — 60%. We know where they are and nobody has looked. And 66 countries have a provider but under a thousand volumes: Bulgaria has 30 providers and zero volumes, Serbia 6 and two, Bosnia 4 and none, Greece 2 and none, Norway 1 and none.
SO THE ORDER OF WORK, BY RETURN PER UNIT OF EFFORT:
1. More regions. A region is a polygon and a list of eras — days of work, no crawl — and it switches on parishes, burial grounds, the era table and the jurisdiction trap notes for every place inside it at once. This is the only lever that improves places already on the map. 41 regions reach 39% of them.
2. Survey the 329. Each one converts a known address into volumes and links. This is the lever that grows the volume count outside Italy.
3. The 66 countries with archives and no records, Bulgaria first, because 30 providers and zero volumes is either a harvest that was never run or a gate nobody has written down.
4. Not more Italian places. Breadth in the 46% is the one thing that flatters the headline and helps nobody.
Row 79 and the cemetery work of 30 September are the same lesson in miniature: the map looked empty at Senj because the data was filtered, not because the ground was bare.
FOLLOWED UP THE SAME DAY, AND THE ANSWER IS ONE COUNTRY. «Survey the 329» is not a diffuse backlog. It is concentrated: France 200, Germany 32, Bulgaria 30, and everything else in single figures. France alone is 61% of every unsurveyed provider this atlas holds.
Set beside Italy it is embarrassing, and it is the clearest instruction this worklist has ever contained:
| providers | unsurveyed | places | volumes | |
|---|---|---|---|---|
| Italy | 129 | 2 | 2,897 | 1,343,253 |
| France | 205 | 200 | 81 | 1,513 |
| Germany | 37 | 32 | 191 | 4,218 |
| Bulgaria | 30 | 30 | 0 | 0 |
FRANCE HAS MORE ARCHIVES CATALOGUED THAN ITALY AND A THOUSANDTH OF THE VOLUMES. 203 of its 205 are regional — the archives départementales and communales, which are the free, online, état-civil and parish-register backbone of French genealogy and among the most generous public record holdings in Europe. Nothing about France is harder than Italy. The only difference in this atlas is that somebody surveyed Italy's 129 and nobody has opened France's 200.
Bulgaria's 30 are not even triaged — triage: null on every one, all state archives, so nobody has so much as looked at whether they answer.
So the order narrows: France first, then Germany's 32, then Bulgaria's 30 — before any further breadth anywhere. And note what this does NOT need: no new region, no gazetteer work, no licence question. The addresses are already in providers.json and have been all along.
9 OCTOBER: THE ARCHIVE COUNT IS NO LONGER MOSTLY UNOPENED. This row measured 329 of 547 providers unsurveyed, 60%. It is now 117 of 555, 21.1% — and 62 of those 117 carry a recorded reason, so only 55 have never been looked at.
France was the answer, and then two more countries were: Bulgaria's 30 and Germany's 31 both collapse to a handful of institutions. Bulgaria's 28 state archives are branches of one agency on one site — one survey, 30 free rows, and the national library turned out to hold freely digitised defteri and sidjili, the Ottoman population and court registers, which is a better door than the archives for anything before 1893. Germany's 31 are really about twelve portals: Niedersachsen's five share Arcinsys with Bremen and with the Kreis, Stadt, church and university archives; Baden-Württemberg's six share Online-Findmittel.
THE LESSON FOR THE REMAINING 55 IS COUNT THE INSTITUTIONS, NOT THE ROWS. France cost two long rows because its 108 communes were 108 separate websites. Bulgaria cost an afternoon because its 30 archives were one. Before surveying a country, group its rows by host: the ratio of hosts to rows predicts the work almost exactly.
What is left is a long tail — FR 42 (all with reasons), BR 6, HR 5, CZ 4, JP 3, PH 3, then ones and twos. No country is worth a row of its own any more.
Survey France's 95 departmental archives — the half of the backlog that holds the records
Row 80 said France was 200 unsurveyed providers and 61% of the whole backlog. Looked at properly, that is two different populations filed together:
- 108
ac-rows — archives COMMUNALES. A commune's own municipal papers. Thin online, and mostly not where registers live. - 95
ad-rows — archives DÉPARTEMENTALES. One per department, and these are the ones that matter: French departmental archives put état civil and registres paroissiaux online, free, for most of the country. Two were surveyed — Bas-Rhin and Haut-Rhin.
Surveying France alphabetically would therefore spend most of the effort on the wrong half. Do the 95 first.
A LIVENESS SWEEP OF ALL 95, 5 October: 85 answered a scripted request, 10 refused. The refusals are the interesting part, and they are NOT link rot:
| code | count | what it means |
|---|---|---|
| 403 | 3 | Forbidden to a script |
| 418 | 2 | the teapot code sites use to turn scripts away |
| 404 | 2 | the only ones likely to be genuinely wrong addresses |
| 000 | 3 | no connection completed |
ad-tarn answered 418 and is perfectly alive: in a browser it resolves to the real Archives Départementales du Tarn, with an anti-DDoS flood page standing in front of scripts. It has since been surveyed and promoted to free — «consultation en ligne de documents numérisés», no account. So a refusal from a sweep is a gate, not a grave, and every one of those nine rows now says so in its triageNote rather than being written off.
WHAT IS LEFT is the slow part and there is no shortcut: each departmental portal is bespoke software, so there is no pattern to match and no bulk promotion to be had. 93 of the 95 still need a browser, one at a time. Alsace CEA and Mayotte need a corrected URL first.
9 October: ten surveyed in one sitting, 80 left. All ten free, none asking for an account to reach the registers. What each one says about itself is now in its what, because the access class is the least interesting thing a survey finds:
| Calvados | its 4E description records that the registers were «microfilmage par les Mormons» in the 1960s and that THAT microfilm is what was digitised, «complété pour la période postérieure à 1860» — the 1860 line, stated by the archive that drew it |
| Charente-Maritime | «Plus de 5,5 millions d'images consultables depuis chez vous», and separately the PROTESTANT registers 1561–1793 and 1806–1831 |
| Bouches-du-Rhône | parish and civil registers XVIe s.–1925, and notarial répertoires running from 1361 |
| Côtes-d'Armor | ~5 million views, and the rare admission that «des pages ont malheureusement été omises lors des opérations de numérisation» — written into what, because a blank there is not evidence the register is blank |
| Cantal | «état civil des origines jusqu'en 1932 (années 1950 pour les décès)» |
| Creuse | XVIe siècle to 1922; births and marriages stop December 1920, which is the privacy window and not a gap |
| Corrèze · Côte-d'Or · Rhône · Charente | all free, all with the register search open |
CHARENTE IS THE ONE WITH A SHAPE WORTH KNOWING. Two tiers, divided by the hundred-year rule: «les tables décennales ne comportant pas de données à caractère personnel, elles sont visibles par tous, sans restriction ! Les actes d'état civil ne sont diffusables sur Internet qu'une fois un délai de 100 ans passé» — for those you sign in with FranceConnect, the French state identity service. Classed free, not account, because the login is only for the most recent century, which is closed on paper too; and not mixed, because nothing here is paid.
AND A SECOND BOT GATE, CONFIRMING ad-tarn WAS NOT A ONE-OFF. lasource.archives.lacharente.fr answers a browser only after a «Making sure you're not a bot!» interstitial. A refusal from a scripted sweep is a gate, not a grave — twice over now.
WHAT IS NOT CLAIMED: no individual register image was opened on any of the ten. The bar is the one ad-tarn was promoted on — catalogue and register search reachable with no account asked anywhere — and ad-calvados says so in its own triageNote.
THE RATE. Calvados took about fifteen browser calls, its catalogue being JavaScript and its commune list behind a select widget. The other nine took one to three each, once the shape was clear: home page, find the «archives numérisées» or «archives en ligne» link, read that one page — and where the page is server-rendered, fetch it from the page context without navigating at all.
Same day, seven more: 21 of 95 free, 74 left, and one genuinely shut.
ALPES-MARITIMES IS THE FIRST REAL REFUSAL. www.archives06.fr answers a real browser — not a script — with «The requested URL was rejected. Please consult with your administrator.» and a support ID. That is a web application firewall declining the connection outright, not an anti-DDoS page a browser walks through. Left unsurveyed and written down rather than worked around; it wants a human on an ordinary connection, or a letter to the archive.
CHER is the one with the research caveat. The greffe collection of parish registers before 1792 is digitised in full, but «la collection communale, souvent plus ancienne, n'a été numérisée que pour les communes ayant procédé à un dépôt» — so the older series is largely NOT online, and its absence from this site says nothing about whether the register survives. Diffusion is 100 years for births and 75 for marriages; the most recent acts online are 1947.
Also free: Alpes-de-Haute-Provence, Dordogne, Drôme, Eure-et-Loir (which also holds the Voves internment camp files), and Finistère, whose état civil search faceted by commune and act type runs with no account at all.
A THIRD BOT GATE. recherche.archives.finistere.fr also answers only after «Making sure you're not a bot!». With ad-tarn and ad-charente that is three, and the pattern is now established well enough to say it plainly: a scripted sweep's refusal predicts nothing about whether an archive is open. The 5 October liveness sweep's ten refusals should all be re-tried in a browser before any of them is believed.
Still 9 October: 25 of 95 free, 70 to go, two shut.
GUADELOUPE is not a mainland département and its holdings say so. 46,371 archive records open with no account, and its thematic collections are esclavage and BUMIDOM — the state scheme that moved Antilleans to mainland France from 1963 — alongside oral testimonies and the Algerian war. Anyone tracing a Guadeloupean family through the twentieth century is tracing it through BUMIDOM, and this is where that paper is.
GERS is the second genuine refusal: www.archives32.fr answers a real browser with «Attack detected» and serves nothing. With Alpes-Maritimes that is two firewalls declining outright — a different thing from the five «Making sure you're not a bot!» hosts, which a browser walks straight through.
Also free: Gard (and it holds the records of refugees from the Spanish Civil War), Gironde — «Registres paroissiaux et d'état civil (1538-1935)», plus passeports pour l'étranger, which is where a Bordelais emigration appears once the registers stop — and Haute-Garonne, whose date picker takes the calendrier républicain, Vendémiaire onward, because that is what the registers of 1793–1805 are actually dated in.
THE TALLY OF GATES SO FAR: five hosts answer a browser only after an anti-bot interstitial (tarn, charente, finistère, gironde, haute-garonne) and all five are open archives. Two refuse a browser outright (alpes-maritimes, gers). The first group would have been written off entirely by a scripted sweep, which is seven in twenty-nine surveyed — a quarter of the backlog misjudged if the sweep is believed.
28 of 95, 67 to go. Three more, and two of them change what the row is worth:
HAUTE-SAVOIE IS ANOTHER STATE'S PAPER. Savoy was Sardinian until 1860, and the digitised list says so: beside the état civil sit the Mappes Sardes, the Tabellion d'Ancien Régime (1660–1798) and the Tabellion de la Restauration Sarde (1814–1861), with the comptes de châtellenie behind them. A researcher expecting only French series here will miss most of it. That is the jurisdiction trap this atlas documents for Istria and Alsace, in a third place.
HAUTE-LOIRE STATES ITS REUSE TERMS, WHICH ALMOST NONE OF THEM DO. «La réutilisation des informations publiques conservées par les Archives de la Haute-Loire est libre et gratuite, quelle que soit la finalité poursuivie… aucune autorisation du service d'archives n'est nécessaire.» Free, for any purpose, no permission. Worth recording as a licence and not just an access class, since this atlas has to tell a reuser what they are taking.
HAUTE-MARNE has both collections online and names them: collection départementale (1 E, 1566–1942, by commune — the duplicates kept since the Ancien Régime) and collection communale (E dépôt). Where Cher has only the greffe half digitised, Haute-Marne has both, and the difference is exactly what decides whether a missing register is missing or merely elsewhere.
AN AUDIT PROMPTED FROM OUTSIDE, AND IT FOUND SOMETHING. The session working the D'Arcy family took the note above about scripted refusals, re-tested its own four «blocked» verdicts, and one was simply wrong — BillionGraves answers a browser at once; the 403 and 400 that condemned it were facts about the request. Its conclusion is worth more than the correction:
Four of my five coverage verdicts describe the SOURCE: covers it, partly, holds nothing, cannot answer. «Blocked» describes the ATTEMPT, and nothing in the row recorded which attempt or by what means. It is also the only one that feels like diligence — it says the archive tried and was refused, which closes the question and invites nobody to reopen it.
THE SAME FAULT WAS HERE, IN 22 ROWS. Ten ac- and twelve ad- providers carried the note «the homepage renders client-side; see scripts/render-pages.mjs» — undated, with no record of what was tried. What happened was a scripted fetch returning no usable content, and «renders client-side» was the explanation assumed for it. Three different things produce that empty body and nothing distinguished them: a site that genuinely needs JavaScript; an anti-bot interstitial, of which this survey has met eight, every one fronting a free and open archive; and a firewall refusing outright, of which it has met two, both genuinely shut.
PROVED MIXED RATHER THAN ASSUMED. ad-l-allier carried that exact note. A browser gets its home page at once — 62 links, «Archives numérisées» in the navigation — and it is now surveyed and free, holding the Cahiers citoyens and the Grand Débat national of 2019 beside the registers. The other 21 notes now say what was actually tried, when, and that a browser has not been pointed at them yet.
32 of 95 free, 63 to go.
FIVE OF THE TWELVE «renders client-side» ROWS OPENED, AND ALL FIVE WERE FREE. 37 of 95 now. None of the twelve needed a renderer; they needed somebody to look.
| Loiret | registers to 1902, and over them 435,911 volunteer annotations with 172,051 bookmarks and their own search — a name index on top of the images, which is the difference between browsing by commune and date and searching for the person you want |
| Hautes-Pyrénées | indexed by what a life does — «Naître, vivre et mourir» — including registres de passeports and the cartes d'identité de Français (1943-1944) |
| Lot | 17th century to 1922, and like Haute-Loire it states reuse: «la réutilisation de ces images est également libre et gratuite» |
| Aube | recensements now to 1975, and it says what is missing from them: «les listes de nombreuses communes sont absentes» for 1946 through 1975 |
| Indre | état civil, recensements, matricules, contrôle des actes, enregistrement |
TWO ARCHIVES IN THIRTY-SEVEN STATE THEIR REUSE TERMS — Haute-Loire and Lot, both saying free for any purpose with no permission needed. That is worth tracking separately from the access class: free says a reader may look, and it does not say an atlas may republish.
And Aube joins Cher and Côtes-d'Armor in the small group that documents its own gaps. Three in thirty-seven volunteer the thing that protects a researcher from reading absence as evidence.
40 of 95. MANCHE ANSWERS THE ABSENCE QUESTION BETTER THAN ANY ARCHIVE HERE.
Everything surviving up to 1892 is microfilmed and online. And if the act is not there, it tells you why:
«Si vous ne trouvez pas l'acte que vous recherchez, c'est sans doute que le registre a été perdu avant son arrivée aux archives ou détruit lors des bombardements en 1944. On trouvera dans l'inventaire de la sous-série 2 E, publié en 1943, le détail des registres paroissiaux disparus.»
An inventory published the year before the bombing, listing what was already lost. So in the Manche a researcher can tell a 1944 loss from an older one — absence with a documented cause and a finding aid for it, which is the strongest form of the thing Cher, Côtes-d'Armor and Aube each gave a weaker version of.
VENDÉE is the biggest index met so far: over six million images, and the «Noms de Vendée» base of 3.5 million nominative records over the top — larger than Loiret's 435,911 annotations. Sarthe keeps pre-1902 and post-1902 as separate collections with complete matricules.
AND ONE OF THE TWELVE NOTES WAS TECHNICALLY RIGHT. Vendée really does render client-side; a fetch of its pages returns nothing. That was never a reason to leave it unsurveyed — it is free and enormous — and it is the cleanest illustration of the fault: the note was not wrong about the mechanism, it was wrong to be an ending. Eight of the twelve are now surveyed and every one is free.
ALL TWELVE «renders client-side» DÉPARTEMENTALES ARE NOW SURVEYED, AND ALL TWELVE ARE FREE. Not one needed a renderer to be classified; one — Vendée — genuinely renders client-side and is six million images of free archive behind it. 44 of 95. The ten remaining rows with that note are all ac- communales.
YVELINES IS A JURISDICTION TRAP AND NOBODY WOULD GUESS IT. It holds the registers of «une partie de l'ancienne Seine-et-Oise» — the département abolished in 1968 and split into six. A commune now in the Essonne, the Val-d'Oise or the Hauts-de-Seine may have its pre-1968 registers at Montigny-le-Bretonneux. 1.6 million views: parish registers 1533–1792, births 1792–1920, marriages 1792–1943, deaths into the 1950s for some communes. With Savoy's Sardinian tabellion and Alsace's German decades, that is the third French jurisdiction trap this survey has turned up, and the only one caused by a boundary change inside living memory.
VAL-DE-MARNE states the communicability tiers more exactly than anyone else: everything 1549–1922 online; some marriages and deaths to 1937 by commune; births and marriages 1923–1943 and deaths to 1952 in the reading room only; the greffe collection ends at 1952 and after that it is the mairies. Anybody writing a general rule about French privacy windows should read that row first — the rule is not one number.
SAÔNE-ET-LOIRE: «des millions de données accessibles gratuitement et à toute heure». VAR: digitised holdings and inventories searched together, with a form for reporting an anomaly in a scan — the quiet sign of an archive that expects its own images to be wrong sometimes.
DONE 9 OCTOBER. 89 OF 95 FREE. SIX WITHHELD, AND THE SIX ARE THE INTERESTING PART.
All 95 have now been looked at by hand. Eighty-nine are free — no account, no fee — and the six that are not carry no access value at all, because nothing positive was seen. They fail in four distinct ways, and the distinction matters because three of them look identical from a script:
- Two WAF refusals. Alpes-Maritimes answers every request with «Request Rejected … Your support ID is: 16102489503195326698»; the Gers with «Attack detected … Request UID: ashV112x9hlqhv57PjFA5gAAAG8». Both serve those pages with HTTP 200, so a status-code sweep would have recorded them as healthy sites. The Pyrénées-Atlantiques does the same thing at 403.
- One unresolved challenge. The Oise sits behind a Cloudflare interstitial that was still spinning after sixteen seconds in a real browser.
- One network blackout. archives.vosges.fr resolves to 92.103.52.35 and then drops every connection on 443 and 80 alike — no refusal, no bytes. www.vosges.fr (92.103.52.33) does the same, so the whole département is dark from here, not the archive.
- One dead service page. Mayotte's only published URL 404s — and FranceArchives, the national portal, publishes the same dead URL. The service is real (1 impasse des Terrasses, Mamoudzou; archives.mayotte@cg976.fr) and has no working web page at all. Nothing is claimed about what is online, because nothing was seen.
AND ONE THAT IS FREE AND STILL UNREACHABLE: THE PYRÉNÉES-ORIENTALES. Eleven free series — parish and civil registers, matricules, censuses, hypothèques, the Napoleonic cadastre, the Retirada and internment-camp iconography — and then this, on the département's own page: «Pour intensifier la sécurité, le Département des Pyrénées-Orientales a décidé de bloquer l'accès hors de France au site Internet : http://archives.cd66.fr. En conséquence, les internautes résidant à l'étranger ne pourront plus se connecter sur les "archives en ligne" … pour une période indéterminée.» A declared geographic block, not a bot gate, and the only one found in France. Our navigation to archives.cd66.fr failed, exactly as the notice says it would. Recorded free, because it is free — with the fence stated in the row, since a reader in Sydney or Chicago will hit a wall the colour on the map would otherwise deny.
CORSE CLEARED ON THE SECOND ASK. It had been written off as an unresolved Cloudflare challenge after 28 seconds. Retried, it cleared in under eight — and behind it is a 1990s frameset (Internet_THOT/FrmSommaireFrame.asp) whose body is empty until you read the frames. Eleven series, including Corsica christiana and the plan terrier, the Genoese-era land survey. A retry was worth more than a renderer.
ROUGHLY A FIFTH OF FRENCH DEPARTMENTAL ARCHIVES SIT BEHIND AN ANTI-BOT INTERSTITIAL, AND EVERY SINGLE ONE THAT CLEARED WAS A FREE, OPEN ARCHIVE. Somewhere near twenty of the 89 showed «Making sure you're not a bot!» or a Cloudflare check on first contact. Not one of them, once through, asked for an account or a fee. The interstitial correlates with nothing about access. Any future sweep that treats a gate as a finding will mis-record a fifth of France as closed.
THINGS NO SUMMARY OF «FRENCH ARCHIVES» WOULD TELL YOU, FROM THE LAST THIRTY:
- Saint-Pierre-et-Miquelon publishes its censuses as PDFs and says why the series has holes: «Suite à l'incendie du bâtiment des archives en 1992 certains documents n'existent plus et d'autres ont été endommagées». Documented absence again, the fifth archive in this survey to offer it.
- Polynésie française publishes a récolement of the état civil 1852–1934 island by island, with separate columns for Tahitian and French, and two explicit headings dividing what is online from what is on the shelf at Tipaeru'i. The digitised colonial registers are at ANOM, not Pape'ete.
- Vienne and Deux-Sèvres run one portal between them. Two départements, one search. No other pair in France does this.
- Seine-Saint-Denis was carved out in 1968, so its état civil search is a list of communes — Saint-Denis, Montreuil, Aubervilliers, Pantin — and not a department-wide series. The fourth jurisdiction trap, after Savoy, Alsace and Yvelines.
- Territoire de Belfort has a digitised collection on the history of the Anabaptist Mennonites; the Nièvre has déclarations de grossesse 1758–1790; the Somme has the mills. These are not état civil and no aggregator lists them.
- The Somme and the Val-d'Oise both volunteer what is NOT online — media held back for rights or closure periods, and a published list of the partly-digitised finding aids. Meurthe-et-Moselle warns readers in public that AI-generated answers to research questions are not to be trusted.
- Savoie's sign-in is going away: online document reservation ends 1 February 2026 and moves to the reading-room intranet. Searching was never gated.
WHAT THIS DID TO THE MAP. France began this row with two surveyed departmental archives and ends with eighty-nine. The unsurveyed share of all providers falls accordingly, and free is now the largest access class in the file. The remaining French backlog is the 108 ac- communales — a different and much thinner population, and ten of them still carry the «renders client-side» note that this row proved worthless for the ad- rows.
The corpus was waiting on 30 country files, and the map now loads by viewport
David: «do the corpus work». The brief said a transliteration pass over Cyrillic «could be worth ~103,000 volumes». Measured, it was worth 6,896 — and it was the wrong fix for the right problem.
WHAT THE BRIEF GOT WRONG. A transliterator turning Тверь into Tver, checked against the real gazetteer with the country constraint applied, matched 170 names / 6,896 volumes. The other 96,000 had nothing to romanise to: those villages were not in the gazetteer under any spelling. And the romanisation was never needed anyway — GeoNames carries Тверь and Калинин, the name Tver bore from 1931 to 1990, in the row for Tver all along. build-gazetteer.py threw both away with a Latin-script filter whose stated reason was that non-Latin names are «transliterations of the CURRENT name» and «what a genealogist needs is the name on the document in their hand». Sound for Europe west of the Bug; wrong east of it, where the document says Тверь and FamilySearch files it under Тверь. Keeping GeoNames' own Cyrillic is an exact match restored, not a resemblance — which is the standard this atlas holds everywhere else.
THE REAL CAUSE WAS COVERAGE, AND IT WAS NOT A CYRILLIC PROBLEM. The gazetteer was nineteen per-country files on a worldwide cities5000 floor, and the floor starts at five thousand people. So every country outside those nineteen had its villages missing:
| places before | unplaced names | volumes waiting | |
|---|---|---|---|
| Netherlands | 582 | 1,748 | 43,246 |
| Belarus | 164 | 3,674 | 39,588 |
| Denmark | 259 | 971 | 19,058 |
| Russia | 6,026 | 6,434 | 68,048 |
| (Croatia, for scale) | 12,007 | — | — |
354,058 volumes — 56% of the entire unplaced corpus — were waiting on 33 missing country files, not on the matcher. Three of the 33 were bad trades and were left out, with the numbers said: China 30.7 MB of download for 623 volumes, India 15.0 MB for 385, Indonesia 9.9 MB for 415. The other 30 were taken, which is 99.6% of the gain.
DONE 7 October. 50 country files now, up from 19; full in the gazetteer's own metadata says which 50, taken from the filenames that were read rather than inferred from the rows.
| before | after | |
|---|---|---|
| gazetteer places | 720,252 | 1,727,886 |
| volumes on a place | 2,933,990 | 3,051,301 |
| places holding books | 32,841 | 42,421 |
| unplaced names | 33,996 | 19,423 |
| unplaced volumes | 627,718 | 403,830 |
Netherlands 43,246 unplaced → 9,822. Belarus 39,588 → 11,683. Sweden 64,039 → 20,900.
AND THE GUARD HELD, WHICH IS THE POINT. Refusals rose 330,611 → 437,188: now that more names reach a real place, more of them are caught landing over 400 km from the province their own path names, and refused. A wrong dot is still worse than a missing one. Тверь itself is still unplaced, 2,730 books, because it arrives as a guberniya and the matcher will not promote a dot from an area name — the Polla trap. That refusal is deliberate and was not touched; it is 388,788 volumes, and whether provincial capitals are worth drawing is its own decision, already flagged in the code and still open.
THE NOTE WAS LYING IN BOTH DIRECTIONS. It read «every populated place GeoNames holds for the countries listed in countries, NOT worldwide» — true of nineteen countries and false of two hundred. The Netherlands had 582 places in a file claiming to hold every one of them. Fixed, and the strong claim is now scoped to full.
TWO THINGS THIS BROKE, BOTH FOUND BY BUILDING IT. Shard keys are isalnum(), which is true of Cyrillic, so the build began writing тве.json files that SmartFind.astro — whose key regex is /[a-z0-9]/ — can never ask for: the same build-disagrees-with-client fault its own docstring was written about, wasting bytes instead of losing names. Keys are ASCII now, and Cyrillic search in the box is a separate thing to want, needing the client regex widened in the same change. And the placed corpus passed 512 MB decompressed, which is V8's hard cap on a single string: surname-volumes.js died with «Cannot create a string longer than 0x1fffffe8 characters» nineteen minutes into the build. match-fs-volumes.py now writes byPlace one place per line — still one valid JSON document, so all six Python readers were unaffected — and the JS reader walks the lines, parsing one place at a time. 1.1s for 564 MB.
WHAT IT COST, AND WHAT WAS DONE ABOUT IT. A place with books on it is drawn eagerly; a place with nothing on it yet is deferred. Placing 117,311 more books moved 27,275 places across that line, so the map's first paint went from 418 KB to 1,021 KB. The audit's verdict was «past 700 KB the answer is loading dots by viewport, not trimming fields», and David's was «do the viewport loading first then push» — so it is done, in the same change rather than left as a debt.
index.json now carries the twelve biggest places in each five-degree cell — 2,550 of 42,841, 79 KB over the wire instead of 1,021. The other 40,291 ride in pi/<lat>_<lon>.json, and three things pull them in, in order: the cells the viewport covers, the ring around them because panning is the commonest gesture, then everything else in the background.
| map first paint | before the corpus | after it | after the viewport split |
|---|---|---|---|
index.json | 418 KB | 1,021 KB | 79 KB |
| places in it | 15,566 | 42,841 | 2,550 |
TOP TWELVE PER CELL, NOT TOP N BY VOLUME, because ranking the world by volume and taking the head gives a first paint that is Italy and the United States with Scandinavia and the Balkans blank — a map that looks like the atlas has nothing there, which is the one impression this project works hardest not to give.
AND THE BACKGROUND SWEEP IS NOT OPTIONAL. The map's search resolves against every place it holds, so a viewport scheme that left the rest unfetched would make a village findable or not according to where the reader had panned. It is a loading ORDER, not a budget: every place still arrives. Measured on the dev server — map drawable at 22 ms, every cell in by 2.9 s — and a deep link to ?at=gracisce, a village that exists only in cell 45_10, still opens its panel.
FOUR THINGS IT BROKE, THREE CAUGHT BY THE PROJECT'S OWN GUARDS. The home page read «12,673 places · 963,618 record volumes» on an atlas of 52,964 and 3,055,526, because lib/atlas.js — the module written precisely so static pages do not read half an index and believe it whole — knew about the quiet half and not the cells. data-shape.test.js failed on 52964 !== 12673 and named it. /data/ was counting the same way and documenting index.json as «every place the map can draw», which it is no longer; it now says so and documents pi/ as its own dataset. check-built.py and audit-site.py both gate the cell declarations now, the same way they already gated the quiet half — a cell promised and not built is places silently gone, and atlas.js throws rather than quietly building a smaller atlas.
The build grew 7,451 MB → 9,625 MB and 65,423 pages → 78,142; audit-site.py says «this will not deploy», but it said that about the last three deploys too, which all deployed — the estimate is deliberately pessimistic (4.2:1 against a measured 7.5:1). Problems reported fell 13 → 12: the index-weight one is gone.
AND IT SHIPPED HALF-APPLIED, WHICH NO GATE CAUGHT. The deploy went green — 9,872 MB, 165,078 files, deploy-pages success — and put «52,964 places · 2,780,922 record volumes» on the live site. Every newly placed village was there and its books were not, and the volume count was LOWER than the 2,893,805 it replaced.
data/fs-volumes-world.json.gz is gitignored and lives as a release asset, because the 740 MB of waypoints behind it are not in a checkout and no runner can rebuild it. Its two siblings out of the same matcher run — fs-volumes-promote.json and fs-volumes-unplaced.json — are committed. So pushing shipped new committed data against an artefact nine days older, and every gate passed because every gate checked one side of the pair. The fix was the one line the repo's own .gitignore has always documented: gh release upload harvest-data data/fs-volumes-world.json.gz --clobber.
SO THE PAIR IS NOW CHECKED. match-fs-volumes.py stamps one run id into all three of its outputs, and check-data.py stops the build when the fetched artefact's stamp differs from the committed data's, naming that command. Verified by tampering with the stamp and watching it fail, then restoring it and watching it pass — because a gate nobody has seen fire is a gate nobody should trust.
WHAT IS LEFT IS A DIFFERENT PROBLEM. The unplaced list is now headed by the United States (158,634) and Italy (44,387), both already fully covered — so what remains there are county and area names, not missing villages. Row 19's frontier, not this one.
Three letters of «San Giovanni» fetched 5.9 MB — and Cyrillic fetched nothing
Row 82 put 103,255 volumes' worth of Russian, Belarusian and Moldovan place names into the gazetteer under the names the documents actually use — Тверь, and Калинин for the sixty years it was called that. They are in the data and they are not reachable from the search box, which is a gap worth naming rather than leaving for somebody to discover by typing.
WHY. build.py's shard_key takes the first three alphanumerics, and isalnum() is true of Cyrillic, so it began writing тве.json — while SmartFind.astro builds its key with /[a-z0-9]/ and can never ask for one. Thousands of files nobody would ever fetch: the same build-disagrees-with-the-client fault that file's own docstring was written about. Shard keys are ASCII now, which is correct and is not the feature.
WHAT THE FEATURE WOULD COST, MEASURED RATHER THAN GUESSED — the rule row 82 set for itself, applied to its own leftover:
| shards | bytes | |
|---|---|---|
| now, ASCII keys | 8,900 | 244.2 MB |
| with Cyrillic keys | 15,984 | 279.1 MB |
| difference | +7,174 files | +34.8 MB |
35 MB on a 9,625 MB build is not the objection. The objection is the shape of it: the biggest new shard is нов at 1,178 KB — Novo- being to Russian place names what San- is to Italian ones — so a reader three letters into a word downloads a megabyte. That is the same complaint the gazetteer sharding already carries against san at 6,485 KB and has carried, unfixed, since the shards were built.
SO IT IS ONE CHANGE OR IT IS NOT WORTH MAKING: widen the client's regex, drop the ASCII restriction in shard_key, and split the handful of shards that common prefixes make too big — four letters where three is not enough, which would help san and par as much as нов. Doing only the first two ships a working search with a one-megabyte worst case.
NOT DONE, AND NOT BECAUSE IT IS HARD. Nobody has asked to search in Cyrillic; the family work this atlas is built around is Istrian and Dalmatian. It is written down here so the decision is made on the measurement rather than by forgetting.
DONE 11 OCTOBER, AND THE MEASUREMENT TURNED THE ROW AROUND. This was filed as a Cyrillic feature with a shard-size precondition. The sizes are the disease and Cyrillic is the cheap rider: 21 gazetteer shards and 37 surname shards were over a megabyte, g/san.json at 5.9 MB and s/cha.json at 3.4, and Italy is 46.4% of this corpus so that is the hot path. The row blamed нов at 1.2 MB; ASCII already had the same problem five times worse.
WHAT A READER NOW DOWNLOADS, measured over seventeen real queries — 44.6 MB before, 2.7 MB after:
| query | was | now |
|---|---|---|
| san giovanni | 6,270 KB | 47 KB (gio) |
| sanremo | 6,270 KB | 3 KB (sanr) |
| colonia nueva | 2,594 KB | 197 KB (nueva) |
| los angeles | 2,054 KB | 214 KB (ang) |
| martin (surname) | 5,348 KB | 476 KB (mart) |
| chapman (surname) | 4,276 KB | 190 KB (chap) |
| Тверь | no key at all | 4 KB (тве) |
| lovinac, Đakovo | 49 / 35 KB | unchanged |
Surname shards: worst 512 KB, down from 3.4 MB, none over budget. Gazetteer: worst is san at 3.55 MB and is reachable only by typing exactly those three letters — every longer query goes to a child or to another word. The «was» column is reconstructed by summing each old shard's descendants, so it is a slight over-estimate; the tree had been rebuilt before anybody thought to measure it.
ONE RULE, PUBLISHED, NOT TWO THAT HAVE TO AGREE. The Belgian shards had already refused a four-character key for the right reason — «it puts a second sharding rule in the codebase that the client must match exactly, which is a bug this project has already had once between build and client». So build.py decides per shard by measuring and writes the decision to public/shards.json, and the client reads it. Eleven copied key loops across RecordMap and SmartFind are now one file, js/shard-key.js, which also holds the one fold.
THREE BUGS, ALL CAUGHT BY TESTS RATHER THAN BY READING THE CODE.
1. The first splitter dropped records whose spelling IS the prefix when the loss was under two per cent — meaning the surname «Char» stops being findable by typing C-H-A-R. That is the d'arcy bug of row 2's shard_key docstring, where 10,962 names were unfindable by their own spelling. Splits now write the prefix's own file too.
2. The agreement check was rewritten to RUN shard-key.js under node instead of re-typing the client's loop in Python, and failed on its first run: keying on the first word left «Le Roy», «De Witt» and «St John» asking for nothing, because build.py keys them ler, dew, stj straight across the space. The old check could not have caught it — it compared Python to Python, and its imitation of the client had itself drifted to isalnum() while the shipped JavaScript used /[a-z0-9]/.
3. SmartFind's fold never had the đ ł ø ß æ œ ı table that build.py and RecordMap both have, so «Đakovo» folded to đakovo on that box and went looking for a key the build writes under dak. RecordMap's own copy had separately lost the dotless ı. On an atlas built out of Istrian and Dalmatian parish books, Croatian đ is not an edge case.
THE TWO CORPORA ARE NOT THE SAME SHAPE, AND THE MANIFEST SAYS SO. The gazetteer indexes a place under every WORD of every spelling, so the client keys on one word and «San Giovanni» can ask for gio. A surname's form is the whole name including spaces, and ler spans one. byword is published per directory rather than guessed, which is bug 2 above, fixed properly.
AND san IS SPLIT AFTER ALL. The first design left it whole, on the grounds that splitting leaves 3.5 MB of «san»-the-word at the three-character address. That was wrong, and «Sanremo» is why: one word, no other word to key on, reachable only through the whole 6 MB. Splitting costs nothing — the leftover is fetched by a query of exactly «san» and by nothing else, while sanr became 3 KB.
FamilySearch files «Vas», GeoNames holds «Vas County» — 71,210 volumes between them
David asked for the measurement before the work, and the measurement killed the work and found something better — which is the point of doing it in that order.
THE IDEA WAS WRONG, AND SO WAS THE NUMBER I USED TO JUSTIFY IT. I proposed rebuilding admin-divisions.json from the 50 full country files, on the grounds that Germany holds 35 divisions and the harvester drops any division with no town of 5,000 — the same cities5000 floor that cost row 82 its villages. The number I attached was unplacedButAreaIsInTheGazetteer, 388,788.
That number is not a loss. It is a counter incremented inside resolve_path, and decided placements run afterwards as a third chance. Its biggest entry is Kings County, 103,480 books — which are on Brooklyn, and have been since September. Quoting it as recoverable was the same mistake as the 103,000 in row 82: a figure that measures one thing offered as evidence for another.
THE REAL LOSS WAS 403,830 VOLUMES, AND 142,713 OF THEM NAMED A DIVISION THIS ATLAS ALREADY HELD. So no download would have helped. The blocker was a naming convention:
FamilySearch files the Hungarian civil registers under path["Vas", "Acsád"], labels["County", "Town or Registration District"].data/admin-divisions.jsonholds it asVas County.
The kind of the division is in GeoNames' name and in FamilySearch's label, so the two never met. County is the last word of 3,145 of the 20,841 division names.
DONE 8 October. Divisions are indexed a second time under the name without its kind, Vas County → Vas:
| baseline | now | ||
|---|---|---|---|
| volumes on a place | 3,051,301 | 3,089,346 | +38,045 |
| unplaced volumes | 403,830 | 343,101 | −60,729 |
| unplaced names | 19,423 | 17,573 | −1,850 |
| placed on a named division | 148,346 | 201,167 | +52,821 |
| refused as far from its parent | 437,188 | 459,905 | +22,717 |
The refusals rising is the guard working: more names now reach a real division, so more are caught landing somewhere their own path contradicts.
A SECOND INDEX, NOT MORE KEYS IN THE FIRST. Merging them could only make matching worse — a real division named Vas and another named Vas County would collide into one ambiguous entry, and division_for would start refusing a name it answers correctly today. The stripped index is consulted only when the exact name is not in the file at all; a name that is present but ambiguous stays refused rather than falling through to a looser answer. Lewis is seven divisions and Franklin County twenty-four, and both are still unplaced, correctly.
AND THE KINDS WERE CHOSEN FROM THE DATA, NOT GUESSED. The commonest last words in the file include Sul, Norte, Grande, Minas and Piauí — Brazilian place names, not kinds. Stripping those would have turned Rio Grande do Sul into Rio Grande do. Only words naming a KIND of division are in the list, and City is deliberately out of it: thirty rows, and Kansas City is the shape of mistake it invites.
IT ALSO TOOK 101 DECISIONS OFF A HUMAN AND GAVE THEM TO AN AVERAGE, UNTIL IT WAS CAUGHT. The matcher's rule is that «a placement never outranks the shelf or the gazetteer», and that is right — those are evidence, a placement is a judgement. But this branch is neither: it is the population-weighted centre of a division's towns, the fallback for when no town in the path survived. A decided placement IS a town that survived; somebody worked out that King County's registers are at Seattle, Etowah's at Gadsden, Choctaw's at Hugo. Measured before the guard went in: 101 of 283 decided placements intercepted, replacing a named real town with a point in a field. Placements now outrank this one fallback, and The Bronx (16,307) and Gadsden (1,013) kept their books.
AND THEN THE EVIDENCE TURNED UP, IN THE COLLECTION'S OWN TITLE. What was left was 82,894 volumes under 472 names matching several divisions — Lewis is seven, Franklin County twenty-four — with no parent in the path to tell them apart. Measured before writing anything: 471 of those 472 names get exactly one parent from the title of the collection filing them.
| filed as | collection | resolves to |
|---|---|---|
Baldwin US | Georgia, County Marriages, 1785-1950 | Baldwin County, Georgia |
Kings CA | Canada, Nova Scotia, Marriages, 1864-1918 | Kings County, Nova Scotia |
Queens CA | Canada, New Brunswick County Deed Registry | Queens County, New Brunswick |
That is not a looser standard. It is the one this file already sets twenty lines further down, where a collection filed under both GE and US is read as American because its title begins «Georgia»: «the title is the corroboration, which is the rule everywhere else in this project».
| before | after | ||
|---|---|---|---|
| volumes on a place | 3,089,346 | 3,122,511 | +33,165 |
| unplaced volumes | 343,101 | 257,559 | −85,542 |
| placed on a named division | 201,167 | 286,712 | +85,545 |
| refused as far from its parent | 459,905 | 512,285 | +52,380 |
THE PATH'S OWN PARENTS ARE TRIED FIRST AND WIN, because a level in the same book is better evidence than a word in the title of a collection holding thousands. Whole words only, and never an adm1 of three letters or fewer, which would match inside some other word in every title it met.
52,380 MORE REFUSALS IS THE GUARD, NOT A FAULT. The distance check reads path levels while disambiguation reads area levels, so where the title says Georgia and a deeper level of the same path points 400 km away, the book is refused rather than placed. Those volumes moved from one not-placed column to another; the real gain is the +33,165.
SPOT-CHECKED RATHER THAN ASSUMED. Baldwin, Camden, Hall, Oconee, Walton and Berrien all land inside Georgia's bounding box. Napoli 40.85,14.29 · Udine 46.06,13.19 · København 55.68,12.56 · Montréal 45.52,−73.62 · nineteen Danish kommuner, all in Denmark. And Vas — the name this whole row started from — is down from 2,146 unplaced to 74, its 2,072 books on vas-county, a place this atlas had already drawn, reused rather than re-invented.
WHAT IS LEFT HERE: 58 names and 1,241 volumes whose collection title names no parent either. Those need evidence that does not exist yet, not a looser rule.
The parent check refused names against themselves, and never asked the divisions
David: «run the refusal audit». The 400 km parent check had grown to 512,285 volumes, 75,097 of them added by my own last two changes, and each time I had called that «the guard working» without testing it. It was not entirely working.
WHAT THE AUDIT FOUND. Every refusal recorded with its distance, its anchor and what anchored it; 4,000 sampled. The distances were not the story — median 1,145 km, only 17% in the marginal 400–600 band, so the threshold itself is sound. The anchors were:
| anchor | resolved to | which is | refusing |
|---|---|---|---|
Kent (GB) | 44.59, −80.59 | Kent, Ontario | Kent in England, 5,792 km |
Alaska | 39.5, −78.77 | Alaska, West Virginia | the state, 5,181 km |
California | 38.3, −76.51 | California, Maryland | the state, 3,788 km |
Alabama | — | Alabama, New York | the state, 1,334 km |
24% of refusals were a name vetoed against itself.
THE CAUSE WAS ONE SLICE. parent_anchor walks areas + path[:-1] — it drops the deepest PATH level because a name cannot vouch for itself. When the path is EMPTY the match came from an AREA, and it then anchored on the very area it was checking. rough_xy resolves a name to its most populous reading, so «Alabama» found a village in New York and refused the state. Its docstring said being wrong there "can only decline to veto"; a wrong anchor is a wrong veto, and that sentence is why nobody looked.
TWO WRONG FIXES BEFORE THE RIGHT ONE, BOTH CAUGHT BY THE GUARD'S OWN TEST CASE. The matcher records why this check exists: «Архангельская» filed under Казань › Чистополь once put 193 books on Arkhangelsk oblast, 1,500 km north.
1. Decline to veto when the readings disagree. Placed 326,120 — and put the Архангельское books back, onto a third one near Moscow. Full Russian coverage now holds hamlets folding to «казань», so a correct, unambiguous anchor looked uncertain and was discarded.
2. Veto only if far from every reading. Placed 316,005, and still leaked 222 books from Samara, Simbirsk and Tatarstan collections onto that Moscow Архангельское. At 207,594 Russian places some hamlet is always near enough to kill a veto — the guard got weaker as the gazetteer got better, which is the wrong direction.
3. Drop the deepest area when there is no path. Symmetric with the path[:-1] already there, and it touches nothing else.
| baseline | after | |
|---|---|---|
| volumes on a place | 3,122,511 | 3,273,254 |
| refused as far from parent | 512,285 | 361,542 |
+150,743 volumes — half what the permissive versions claimed, which is the point: the other 175,000 were refusals that deserved to stand.
TESTED IN BOTH DIRECTIONS. g-arkhangel-skaya-ru is back to 0 and the Samara, Simbirsk and Ufa leaks are gone, while Kent (4,790), Alaska, California and Orange are placed. Brooklyn keeps 103,480, The Bronx 16,307, Winston-Salem 13,572 — no decided placement disturbed. Coordinates spot-checked: Palermo 38.12,13.36 · Braga 41.55,−8.42 · Rio −22.72,−43.09.
AND THEN THE Вятка GAP, WHICH WAS NOT WHERE I SAID IT WAS. I reported 22 books filed Вятка › Елабуга › Архангельское as a gazetteer hole. Looking properly, the hole is wider and sits somewhere else: rough_xy never asked data/admin-divisions.json. It reads the gazetteer, the exonyms and the shelf, and an area level that no settlement answers to returned nothing — so parent_anchor found nothing and the 400 km check never ran at all.
| area level | volumes | |
|---|---|---|
GB England | 40,744 | in admin-divisions all along |
US Forsyth County | 13,572 | likewise |
US Iredell County | 10,971 | likewise |
US Chittenden County | 8,742 | likewise |
416,699 volumes across 764 names. Those books were not placed carefully, they were placed unchecked, which reads identically in the counts and is not the same thing. The file holding every one of those centres was already open four lines above, for a different purpose.
Anchoring on a division now — and only when the name means exactly one, because taking the first of several is the very fault that made Alabama anchor on a village in New York, and an anchor exists to veto, so a wrong one refuses a correct book.
| before | after | |
|---|---|---|
| volumes on a place | 3,273,254 | 3,272,072 |
| refused as far from parent | 361,542 | 362,724 |
−1,182 volumes, and that small number is the result. 416,699 books moved from unchecked to checked; all but 1,182 were already where their division says they should be, and those 1,182 were wrong. Brooklyn's 103,480, The Bronx's 16,307, Winston-Salem's 13,572, Iredell's 11,031, Palermo and Braga: all unchanged.
WHAT IS STILL UNANCHORABLE: 398,360 volumes by neither route — Italian provinces (Verbano-Cusio-Ossola 20,435, Forlì-Cesena 17,843, Pesaro e Urbino 12,362), Upper Austria, and the genuinely historical names: Симбирск 8,917, Бессарабия 8,455. Those are a real gazetteer gap and a different piece of work.
AND ONE DEFECT IN MY OWN CORPUS CHANGE, FOUND ON THE WAY. Row 82 keeps six native-script alternates per place, nat[:6], taken in GeoNames' own order — which is arbitrary. Ulyanovsk's six are Оулиꙗновьскъ, Сембер, Синбирск, Сімбір, Ульск, Ульянаўск: Old Church Slavonic, Belarusian, everything except Ульяновск, the modern Russian name. Two of fourteen major cities tested are missing their own name that way — Ульяновск and Уфа. Not fixed here: it needs the gazetteer rebuilt, and the 806 MB of GeoNames dumps were deleted in the clear-out this morning. Worth doing with a rule that ranks the native forms rather than taking the first six.
Six native names chosen four different wrong ways — the choosing was the bug
Row 85 found this on the way past and I said it needed a rule that ranks the native forms rather than taking the first six. That was the fourth-best of five answers, and three of the others got built before the right one.
THE DEFECT. Row 82 kept six native-script alternates per place, nat[:6], in GeoNames' own order — which is arbitrary. Ulyanovsk's six were Оулиꙗновьскъ, Сембер, Синбирск, Сімбір, Ульск, Ульянаўск: Old Church Slavonic, Belarusian and three historical spellings, and not Ульяновск, the name a Russian document says.
FOUR TRIES, AND EACH LOST A DIFFERENT SET OF NAMES.
| fixed | broke | |
|---|---|---|
| 1. rank by closeness to the ASCII name | Ульяновск | all six slots became Ульяновск, Ульяновськ, Уляновск, Улјановск, Уљановск, Ульянаўск — one name in six national spellings, the exact Wrocław problem the Latin pool is clustered to avoid |
| 2. cluster, as the Latin pool does | Синбирск got a slot | Тверь → Твер: they romanise alike, tie at 1.000, and a stable sort kept the truncated one |
| 3. break ties by length | Тверь | Минск, Кишинёв |
| 4. remove the cap | nothing | — and that was the useful failure |
Removing the cap changing nothing is what showed the cap had never been the problem. Clustering was. Минск and Мінск romanise identically; they are the Russian and the Belarusian spelling, and a register may be written in either. The romanisation is deliberately rough — it cannot tell a soft sign from nothing — so any rule built on it will collapse names that are genuinely distinct.
SO: KEEP THEM ALL. Uncapped, unclustered, ordered modern-name-first for display only. Every entry is a name GeoNames records for that exact place, and the choosing was the mistake.
18 of 19 test names resolve, against 12 of 15 before: Тверь, Калинин, Ульяновск, Уфа, Синбирск, Вятка, Минск, Кишинёв, Бежецк, Київ, Гродна. The one miss is Симбирск, which GeoNames does not hold — it records Синбирск, and that is found.
AND TWO SMALLER THINGS. Уфа is three characters and was being dropped by the len(alt) < 4 rule written to clear out airport codes — a Latin rule the native pool had inherited. And the romanisation exists only to rank: a transliterator that placed a book on resemblance was measured and refused in row 82, and this one never decides where anything goes, only which of a place's own recorded names reads first.
WHAT IT WAS WORTH: +4 VOLUMES. Four. The names it rescued are large cities that FamilySearch mostly files under Latin or that were already placed by another route. It costs 39.8 MB of git, because the gazetteer is committed and gzip pays full price per version.
That is a poor trade on today's numbers and it is still the right file to ship: the data was quietly wrong, CI cannot rebuild it, and the next person to want it would pay 811 MB of GeoNames downloads to find out. Row 83's Cyrillic search would need exactly this. Said plainly rather than dressed up as a win.
Survey France's 108 communal archives — the thin half, and it is not thin the way anyone expected
Row 81 closed the 95 départementales at 89 free. The 108 ac- communales were supposed to be the easy remainder: thin online, mostly not where registers live. The first half of that was right. The second half was the point.
ALL 108 LOOKED AT. 73 NOW CARRY AN ACCESS CLASS — 33 free, 27 onsite, 11 catalogue, 1 mixed, 1 account. The remaining 35 stay unsurveyed, every one with a dated reason written into the row.
THIS IS THE FIRST POPULATION IN THE FILE THAT USES THE WHOLE LEGEND. Row 74 complained that index, mixed and paid were ZERO and had to be drawn dead on the filter. France's communes have now produced the first mixed row in the atlas — Cannes, which says its documents are «consultables sur place ET EN PARTIE sur Internet» — and the first account row that is not a self-service sign-up: Vichy will not let you near its digitised archives until you e-mail the service and it posts you credentials, and the portal it then lets you into still calls itself a «Version d'évaluation».
TWENTY-SEVEN COMMUNAL ARCHIVES PUBLISH NOTHING ONLINE, AND THEY SAY SO THEMSELVES. That is a quarter of the population, and it is a finding rather than a gap in this survey, because in every case the service's own page states the terms:
- Beaune holds the COMPLETE RUN OF COUNCIL DELIBERATIONS SINCE 1525 — and you read it in the left wing of the Hôtel de Ville, the former Ursuline convent, free, on production of ID.
- Manosque has kept EVERY deliberation of its council since 1366.
- Remiremont holds what the mairie produced and received since the 13th century.
- Bourg-Saint-Andéol has tables of baptisms, marriages and burials 1581–1792 plus microfilms of the parish registers and photocopies bound by the local society — Wednesday mornings, ten to twelve.
- Guyancourt's oldest holdings are parish registers 1598–1792; Annecy's cover 700 years and the communes merged into it in 2017, Tuesdays and Thursdays only.
- The Fabrique des savoirs will answer a genealogical enquiry by post if you name precisely dated acts AND ENCLOSE A STAMPED ENVELOPE.
- Le Puy-en-Velay's reading room has eight seats.
AIX-LES-BAINS NAMES THE REASON FOR ITS OWN EARLIEST GAP: «Un incendie en 1739 a détruit les archives antérieures.» Saint-Pierre-et-Miquelon said the same thing about 1992 and Saint-Quentin about the First World War. That is three archives in two rows of this survey volunteering a documented cause for an absence, which is the single most useful sentence an archive can publish.
SAINT-QUENTIN IS THE ROW THAT CHANGES HOW TO USE THE DÉPARTEMENTALES. It holds état civil and décennial tables 1792–1912 with ONE gap — 1799, the register lost in the First World War — and the registers of all twelve parishes, the oldest 1582. The greffe duplicates are free at the Archives de l'Aisne. And then: «Outre certaines lacunes qui n'existent pas dans notre service, en venant notre salle de lecture vous pourrez bénéficier de l'aide du personnel.» The online departmental copy is the LESS complete one. Row 81 spent itself proving the départementales are free and open; this row proves that free and open is not the same as complete, and that the mairie is the fallback nobody thinks of.
FOUR MORE COMMUNES SEND YOU STRAIGHT TO THE DÉPARTEMENT, AND ONE OF THEM IS BLUNT ABOUT WHY. Laval: «une partie importante des archives anciennes (avant 1790) et modernes (entre 1790 et 1982) de la Ville de Laval est encore conservée aux Archives départementales de la Mayenne.» Vitré: the digitised acts of ALL 46 COMMUNES of its territory are at the AD Ille-et-Vilaine. Troyes: «État civil de la ville de Troyes (1535-1922)» is free and downloadable at archives-aube.fr, and the municipal service has merged into the departmental one. Roubaix and Villeurbanne likewise point at the AD Nord and the ADRML. For a genealogist the communal row's job is often to say «not here».
LYON IS THE BEST OF THE 33 FREE ONES AND NOT BECAUSE OF ÉTAT CIVIL. Alongside the registers it publishes the convois funéraires, the fiscal censuses, the enfants abandonnés, and both the entrées AND the décès of the Hospices civils de Lyon, plus the nommées and the lettres de bourgeoisie. Foundlings and hospice admissions are among the hardest records to find anywhere. Bordeaux Métropole has the cemeteries of Bordeaux and Caudéran 1814–1974 and the immatriculation des étrangers. Auxerre and Landerneau both publish burial registers; Landerneau adds the rôles de capitation, the Ancien Régime poll-tax rolls. Rennes searches the hospital registers with the rest. Nantes has the registers of funerary inscriptions with a nominative search. Nancy put 356,000 views under the Licence Ouverte — open data, not merely viewable. Besançon keeps everything on Mémoire vive with the archives, libraries and museums pooled, and warns that series W is off-site and needs 48 hours.
NINETEEN URLS IN THIS ATLAS WERE WRONG, AND FINDING THEM WAS HALF THE WORK. Dijon's was a WordPress blog its authors had deleted. La Rochelle's host now serves nothing but an Apache «Index of /». Meudon's, Colombes's, Aix-les-Bains's, Suresnes's, Poissy's and Valence's all 404ed. And five communes had a whole archive portal the atlas did not know about — Alençon, Thionville, Toulon, Montpellier and La Rochelle — found by simply probing for an archives. subdomain, which cost one DNS query each and should have been done at harvest time.
THE DANGLING-DNS TRAP. That same probe turned up five archives. records that resolve and serve nothing — and one that serves the WRONG THING: archives.ville-forcalquier.fr redirects over port 80 to cohesion.oise.fr, a login page for an unrelated service in the Oise. A harvester that trusted DNS would have filed an Oise intranet login as the archive of a commune in the Alpes-de-Haute-Provence. Resolution is not existence.
THE GATES, AGAIN, AND WORSE THAN IN ROW 81. Nineteen of these hosts answered curl with 403, four with 418 and three with 503 — and almost every one of those was a dedicated archives.* host, which is to say the ones most likely to be real portals. Marseille, Rennes, Strasbourg, Bordeaux, Grenoble, Saint-Denis, Tarbes, Auxerre, Béziers, Landerneau and Sceaux ALL cleared their proof-of-work interstitial within about ten seconds in a real browser, and all eleven were open archives. Metz, Reims, Saint-Maurice and Saint-Ouen did not clear and are recorded as refused — Metz with a support ID, Reims with a flat «403 Forbidden», Saint-Ouen with a haphash challenge still spinning after nineteen seconds. Saint-Ouen's challenge page is served with HTTP 200, so a status sweep calls it healthy.
EIGHT ARE SIMPLY DOWN, with their own words: Antibes and Tourcoing «503 Service Unavailable», Haguenau «maintenance downtime or capacity problems», Roncq «Tous les serveurs qui y sont affectés sont indisponibles ou en maintenance», Quimper returning a literally empty <html><head></head><body></body></html>, and Sevran and Évreux timing out on every address the commune owns. Those are worth retrying; they are not refusals.
WHAT IS LEFT, AND IT IS SMALL. Twenty rows where the url is the commune's own site and no archive page is reachable from it. Each says so, and says what was tried. Every French commune has archives by law; these twenty do not appear to publish anything about theirs, and the atlas now records that rather than leaving the row blank. The next move for them is the FranceArchives directory, which is authoritative and has an entry per service — but its pages need a browser per lookup, and its SPARQL endpoint does not answer queries over HTTP, so that is twenty browser visits and a separate job.
FRANCE IS NOW DONE AS A COUNTRY. 95 départementales and 108 communales, 203 rows, all surveyed or refused with a reason. 162 carry an access class. The country that began this sequence with four providers and two surveyed rows is the best-documented in the atlas.
Bulgaria's 30 and Germany's 31 — and the trick is counting institutions, not rows
Straight after France, the two biggest remaining blocks. Sixty-five rows, and between them about fourteen institutions.
BULGARIA: 30 ROWS, ONE AGENCY, ALL FREE. Twenty-eight of the thirty are branches of the State Agency «Архиви» on a single site with a single policy, so one survey covers them. Its ДИГИТАЛЕН АРХИВ searches each branch separately and returns described items with a full citation — archive > фонд > опис > архивна единица > лист — and the image. It is a selection in the hundreds per archive, not a bulk scan: the keyword «снимка» returns 153 in Plovdiv, 148 in Varna, 74 in Smolyan, 12 in Burgas.
THE REGISTERS ARE MOSTLY NOT IN THE ARCHIVES, AND A LAW SAYS WHY. Civil registration began 1 January 1893; the municipality writes the registers and keeps them for 130 years before transferring them to the state archive (Art. 85(3), Law on Civil Registration). Until that term runs out, certified copies come from the municipality of birth or residence. So for almost anybody alive-adjacent, the state archive is the wrong door — and the agency says as much itself, adding that it cannot certify Bulgarian origin (that is the Executive Agency for Bulgarians Abroad, with citizenship at the Ministry of Justice) and that army service is checked only by the military-historical archive at Veliko Tarnovo.
THE CENTRAL STATE ARCHIVE HAS THE ONE SERIES THE DIASPORA WANTS. Fond 246К, the Bulgarian Exarchate: parish registers of the Bulgarian population across Ottoman Thrace and Macedonia — Eastern and Western Thrace, Aegean, Vardar and Pirin Macedonia — naming individuals and accepted as proof of Bulgarian origin. Compiled 1906–1907, so they record only people born before or during those years, which is the kind of date that saves a reader a month. Inventories 9 and 10 are digitised and free through ИСДА — which resolves to 212.122.187.196 and timed out on four ports, two schemes, curl and a real browser. Recorded unreachable, not closed.
THE NATIONAL LIBRARY WAS ONE UNSURVEYED LINE AND IS THE BEST BULGARIAN SOURCE IN THE FILE. Св. св. Кирил и Методий publishes its digital library free «в пълния им обем», and two of its collections are the Ottoman-era equivalent of parish registers: the ДЕФТЕРИ, tax and population registers listing householders village by village, and the СИДЖИЛИ, the kadı court registers. Plus Bulgarian newspapers 1844–1944, and an Oriental department holding about a million sheets in 342,000 archival units from the 15th century on. I first wrote that the site could not be reached; it answers normally, and the row records the correction rather than hiding it.
GERMANY: 31 ROWS, ABOUT TWELVE PORTALS. 24 catalogue, 4 free, 3 paid, 3 onsite, 1 mixed, 2 unreachable. Niedersachsen's five branches and the Hannover church archive all share Arcinsys, which also carries Bremen and the Kreis, Stadt, Gemeinde, parliamentary, party, church and university archives — one open search across institutions with nothing to do with each other. Baden-Württemberg's six share Online-Findmittel. Sachsen-Anhalt's two share a scopeArchiv portal. Hessen runs its own Arcinsys installation.
THE GERMAN CHURCH BOOKS ARE BEHIND A SUBSCRIPTION AND THE ARCHIVES SAY SO OUT LOUD. Kiel: «Das Landeskirchliche Archiv ist Mitglied im Kirchenbuchportal Archion». Bayern announces that the Nürnberger Kirchenbuchverkartungen — the card indexes that are the shortcut through the registers — went onto Archion at the end of July 2026. Archion is paid. Berlin, Dresden and Hannover publish no online register route at all, and Hannover reads its church books as microfiche at a central station in the Kirchenbuchamt. These are the atlas's first well-evidenced paid rows, and they sit on the most-wanted German source there is.
AROLSEN IS THE COUNTERWEIGHT. collections.arolsen-archives.org searches documents by person — surname, forename, birth name, date of birth, prisoner number — free, no account. For a family deported, imprisoned, conscripted as forced labour or displaced, it is the first place to look.
TWO SMALL TRAPS WORTH REMEMBERING. Thüringen's four branches sit behind a Link11 CAPTCHA that clears in about ten seconds, and their digitised material lives at the university library in Jena on an endpoint whose own path reads stat2_restricted — so the images were not claimed as free on the strength of a link. And seven of these German ids transliterate umlauts (bueckeburg, not buckeburg), so the first write hit seven ids that do not exist; caught only by checking which ids matched, which is now the habit.
WHAT THIS LEAVES. Providers unsurveyed: 117 of 555, 21.1%, down from 60% when row 80 measured it — and 62 of the 117 carry a dated reason, so the never-looked-at population is 55. The tail is FR 42 (all with reasons), BR 6, HR 5, CZ 4, JP 3, PH 3, then ones and twos. The rule this row establishes: before surveying a country, group its rows by host. France's 108 communes were 108 websites and cost two long rows. Bulgaria's 30 archives were one website and cost an afternoon.
Flags is a separate website wearing the same name — fold it in, then put it on the map
David, looking at the two pages side by side: «flags doesnt feel like its part of record atlas? different widths, missing menu bars, etc? can we show them combined, since they really are now?» He is right, and the reason is structural rather than cosmetic.
IT IS NOT STYLE DRIFT. IT IS A DIFFERENT PUBLISHING PIPELINE. site/public/flags/ holds 245 pre-built static HTML files — 210 <country>-flags.html, 34 *-hub.html and an index — which Astro copies into dist VERBATIM. They never pass through Base.astro, so they cannot have the nav, the fonts, the container or the footer: measured on croatia-flags.html, there is no «Explore / The sources» nav, no astro-* class from the layout, and its own ladder of widths — 260, 340, 480, 560, 620, 880, 900px — set independently of the site's. The «Flags» item in Base.astro's nav (line 51) points at /flags/, so the reader walks out of the atlas and the door closes behind them.
THE FIX IS TO RENDER THEM, NOT RESTYLE THEM. The content is already structured data, not just HTML: flag-index.json (210 countries, 34 hubs, each with name, region, hub, colour, crowns[] and an inline SVG flag), flag-eras.json (1,471 eras, 1,332 with years, each with polity, dates and tags) and era-flags.json (1,454 historical flags as inline SVG). An Astro route — /flags/[slug].astro reading those three files — would give every flag page the nav, the widths and the footer for free, and would delete 245 hand-built files. The pages are generated already; they are just generated by the wrong thing.
AND THE TWO QUESTIONS DAVID ACTUALLY WANTS ANSWERED.
1. «is there a way we could start placing every flag on the country shape on the map?»
Most of it exists. site/public/c/ already holds 214 country outlines, and the map already fetches one when a surname names a country. The flags are inline SVG in flag-index.json. Joining them by name gets 169 of 210 on the first try; the misses are nearly all spelling — «Bosnia & Herzegovina» against «Bosnia and Herzegovina», «United States» against «United States of America», & against and — plus Scotland, Wales and Northern Ireland, which have flags and no sovereign outline. A small alias table takes this well past 90%.
MY VIEW ON HOW, BECAUSE THE OBVIOUS WAY LOOKS BAD. Clipping a flag to fill a country's shape is a cartographic trap: the shapes differ by four orders of magnitude in area and wildly in aspect, so Russia reads beautifully, Chile becomes a stripe of one colour, and Malta disappears entirely. What survives every zoom is a two-part treatment: fill the polygon with the flag's dominant colour, which flag-index.json already carries as color, and draw the flag itself as a fixed-size badge at the centroid that does not scale with the country. Then a reader sees the colour field at world zoom and the actual flag at any zoom, and Malta is as legible as Russia. The year control already on the map is the natural way to change WHICH flag is drawn.
2. «a toggleable animation of migratory paths and conquerers showing ins and outs»
THE DATA IS ALREADY THERE AND NOBODY HAS DRAWN IT. The era tags are not one bag; they come in three rows, and one of them is exactly this:
migro-row— 1,250 tags, already in the in/out vocabulary: Immigration 452 · Emigration 343 · Conquered by 252 · Occupied by 81 · Migration 20 · Settlement 11 · Displaced by 6 · Absorbed by 3 · Attempted conquest by 3. Reading like «Occupied by: Soviet Union (1979–1989)», «Conquered by: Mongol Empire (1220s)», «Emigration: mass flight after 2021 Taliban takeover».noble-row— 966 dynasty tags.tag-row— 4,397 everything else.
So conquerors are Conquered by + Occupied by + Absorbed by (336 tags) and migration is Immigration + Emigration + Migration + Settlement + Displaced by (832). Two toggles, straight off the existing data. And 1,279 eras carry a from year with 1,129 carrying to, so the timeline to animate along already exists.
THE HONEST LIMIT, WHICH SHOULD BE SAID BEFORE STARTING RATHER THAN DISCOVERED AT THE END: AN ARROW NEEDS TWO ENDS. Of the 1,250 migro tags, 482 (39%) name a counterpart that matches a country or hub already in the atlas. The other 61% name something with no polygon — «Arab Caliphate (7th–8th c.)», «Venetian coastal enclaves», «Kosovar refugees (1999)», «mass flight after 2021 Taliban takeover». Those are real history and they are not line segments. So the honest build is two layers, not one: drawn arrows for the 482 that resolve, and an in/out marker on the country itself for the rest — an arrow leaving the edge of the shape for an emigration with no destination, entering it for a conquest with no named conqueror. A map that drew only the 39% would quietly imply that the Mongols never came.
ORDER OF WORK. Fold the pages into Astro first, because it is the smallest piece, it deletes 245 files, and it puts the flag data behind a route the map can call. Then flags on shapes, because it is a join and a draw. Then the animation, because it is the only part with a design question in it — and the 61% is that question.
9 OCTOBER, WHILE FOLDING THE PAGES IN: THE ANIMATION ALREADY EXISTS. ALL 210 COUNTRY PAGES HAVE ONE.
Every country page carries an imap — a hand-authored map with a play button and a year slider, with its own CSS vocabulary (imap-play, imap-year, imap-owners, imap-story, imap-key). Croatia's reads: «The same real coastline as the regional hub, zoomed in on Croatia alone and split into the four pieces that were, for significant stretches of history, governed separately … Press play, or drag the slider», and then the rule that makes it good — «The border lines below only solidify where two pieces were actually ruled apart — they dissolve the moment a single government held them together.»
So the hard part of David's third question is not «can we animate rulers over time». It is DONE, 210 times, with a design rule most historical atlases get wrong. The question is only how to lift it onto the main map.
AND THERE IS ONE REAL OBSTACLE, WHICH IS WHY THIS IS NOT A COPY-PASTE. The imap geometry is hand-drawn per page in a LOCAL SVG coordinate space — Croatia's paths read M118,20.2 L130.9,-13.5 … — not latitude and longitude. It cannot be dropped onto Leaflet. The main map would have to drive the real country outlines in site/public/c/ instead, which is the same 214 polygons the flags-on-shapes job needs. That makes the two jobs one job: get the outlines reacting to a year, and both the flag badge and the ruler animation fall out of the same machinery.
It also means the sub-national split the imap does so well — continental Croatia, Dalmatia, Istria, the Military Frontier — has no equivalent in c/*.json, which holds one polygon per modern country. A world map can honestly show WHO RULED A COUNTRY in a given year; it cannot show the Military Frontier without sub-national geometry nobody has. Worth saying now: the country pages will stay the better view for one country, and the world map earns its place by comparison across many.
THE JOIN MEASURED PROPERLY, AND IT IS BETTER THAN THIS ROW GUESSED. Above I wrote «169 of 210 on the first try … a small alias table takes this well past 90%». Two corrections, both in the right direction:
The alias table already exists. site/src/lib/flags.js carries 22 aliases mirroring _ALIAS in build.py, written when hub totals came out short — «Bosnia & Herzegovina» → «Bosnia and Herz.», «Cape Verde» → «Cabo Verde». Using it lifts the match against the PUBLISHED outlines from 169 to 188 of 210 (90%) with no new code.
And the published outlines are not the whole geometry. data/countries.json is keyed by country code and every one of its 237 entries carries rings — the real polygon. site/public/c/ publishes only 214 of them, so 23 countries have geometry on disk that no shard exposes, and they are exactly the ones that came out missing: Malta, Monaco, San Marino, Vatican City, Singapore, Seychelles, Barbados, Grenada, Nauru. Against the full file the match is 197 of 210 — 94% — with nothing to harvest and nothing to alias. Publishing 23 more shards is the whole fix.
WHAT IS LEFT OUT IS NOT A GAP, IT IS A CATEGORY. The thirteen with no geometry are the Kingdom of Hawaii, the Ryukyu Kingdom, the Kingdom of Sikkim, Tannu Tuva, the Dominion of Newfoundland, and the Mongol, Mughal, British, Spanish, Roman, Persian and Ottoman empires, plus Ancient Egypt. None of them has a modern polygon and none ever will. They belong to the animation layer, not the base layer — an empire is a thing that ARRIVES on somebody else's ground, which is precisely the «Conquered by» relation, so they are drawn as the source of an arrow rather than as a shape of their own. That the two halves of this row partition so cleanly — 197 countries that are shapes, 13 empires that are only ever arrows — is the strongest argument yet that the flag layer and the conquest layer are one design and not two.
Keep the fix-it list: remember what was ticked and left as is, and re-apply it when the same file comes back
Closed 8 Nov 2026: shipped in 69e9f90 (remembered fix-list decisions; also carried in the research file). The row still said the code was waiting in docs/research-backup-wip/.
Today it is lost on reload. In the tree viewer (site/public/roots/js/app.js, asIs, ticked) a row marked Leave as is lives in memory only, so closing the tab throws away every decision. The viewer keeps nothing else on purpose, and a test asserts it (roots.test.js).
Proposal. Save the decisions, never the tree: the key of each row (place text and issue code, not a person's name or id), what was decided, and the date. Three places, in this order: (1) a button-pressed Remember on this device that reuses the existing IndexedDB module recent.js, which already keeps up to five files on a button press; (2) the same decisions written into the research file (next item) so they travel; (3) a Download my decisions file for people who clear their browser.
Re-apply. On loading a file whose decisions are known, say "34 decisions from 3 Nov re-applied; show them again", never silently, and keep left as is visible behind a count as it is now.
Check before building. The row key must not carry a living person's name or id: read workbench.js and checklist.js, and extend the privacy test to say exactly which keys may be stored. Say in the interface which list this is.
State (6 Nov 2026). The code is written and tested (13 tests) but not yet in the site: docs/research-backup-wip/ holds it with apply.py and HANDOVER.md, which lists the steps (apply, look at it in a browser, full build, push).
One research file: everything this browser holds, in one dated backup that merges, with a nudge when it is stale
Closed 8 Nov 2026: shipped in 69e9f90 and dfb10af (dated research file that merges, restore, Connection & Backup page) and always-on sync in e4a26cf.
The risk is real and measured. Progress lives in localStorage (ra-progress-v1: lines and volume marks), plus ra-check and ra-recent. A private window, cleared site data, or Safari's seven-day clean-up of unused sites' storage empties it, and /my-research/ is the only way out: a manual Export my research that nobody is reminded to press.
Proposal. (1) Version the file (v:2) and make it hold every piece of state in one place: marks, lines, fix-it decisions, the visiting list, and the date of the last backup. Importing still merges and never replaces, as it does now. (2) Show last backed up N days ago on My research and a quiet banner after a set number of days or marks. (3) Ask the browser to keep the data (navigator.storage.persist()) and say plainly when it refuses. (4) Where the browser allows it, offer Save to a folder (File System Access API) that overwrites one dated file, and on a phone the share sheet to send the file to Mail or iCloud.
Not in scope. No server. This is the cheapest fix for the loss David described, so it goes first.
State (6 Nov 2026). The code is written and tested (13 tests) but not yet in the site: docs/research-backup-wip/ holds it with apply.py and HANDOVER.md, which lists the steps (apply, look at it in a browser, full build, push).
Share research as a pack: everything, one family line or one name, as a file or a link, readable without importing
Closed 8 Nov 2026: shipped in 69e9f90 (share a pack, the /shared/ reader page, Preview what leaves).
What a pack holds. Only what a progress mark already holds: volume, pages walked, state (planned, searched, found, exhausted, blocked), the date, the family line's surname and place. Never a tree, never a given name of a person, never a living person. A Preview what leaves step shows the exact contents before anything is made.
Two shapes. A file for anything large. A link for small packs, carried in the part of the address after the #, which a server never receives, so the site still collects nothing.
A reader page. /shared/ opens a pack read-only (drag in a file, or follow the link): what was searched, what was found, what is still open for this name, with Merge into mine and Continue from here (starts a line at the first unwalked page). That is how someone else carries on the research without an account.
Rules that carry over. A mark is a claim by whoever made it and the page says so: reported by the sender, not checked. Absence is not evidence: a nil is shown as a search done, never as a finding about the name. The verified route stays the family archives' searched.json and the publisher.
State (6 Nov 2026). The code is written and tested (13 tests) but not yet in the site: docs/research-backup-wip/ holds it with apply.py and HANDOVER.md, which lists the steps (apply, look at it in a browser, full build, push).
Accounts: decided on 5 Nov 2026, passphrase sync with no account, kept on Cloudflare
Decision (David, 5 Nov 2026): option B. No account, no email. The browser encrypts the research file with a key made from a generated passphrase, and a small service stores the unreadable blob under an id also derived from the passphrase. We hold nothing readable. The other options (your own Google or Dropbox folder; a real account with a public profile) were set aside, and a profile stays a later step that would need a rewritten privacy page.
Recovery. A new device types the passphrase and restores. A lost passphrase cannot be recovered by anyone, so setup generates a six-word phrase, downloads or prints a recovery sheet and does not finish until it is saved; the dated research file from the item above is the independent backup. Versions kept 30 days; read and write keys separate.
Build passphrase sync on Cloudflare: a Worker, R2 storage, a nightly second copy
Design. A Worker on a subdomain (sync.recordatlas.org, or workers.dev as a stopgap) that accepts a save (checks the write key and a size cap, stores <id>/<timestamp> in R2), serves a restore and lists versions. R2 lifecycle deletes versions after 30 days. A scheduled Worker copies the encrypted blobs nightly to a second provider (Backblaze B2 or a second Cloudflare account). No database: the server cannot read the data, so it never queries it. Accept requests only from recordatlas.org.
In the browser. Web Crypto, a key and an id from the passphrase (a slow key-derivation function), merge on pull exactly as the file import does, synced 2 minutes ago on My research, roll back to an earlier version, and Delete everything I have stored that needs no email.
Needs David. Check where the recordatlas.org DNS is managed; create the Cloudflare token and R2 bucket (David holds the token, the session never enters it); say whether a card on file is acceptable (free tiers should cover it, confirm current limits). The privacy page gets a paragraph: optional sync stores one encrypted file we cannot read, and Cloudflare logs the IP address as any host does.
Order. After the research file and the shared pack, so that sync has one file format to carry. Tests. The server never sees plain text; a wrong passphrase restores nothing and says so; two devices that both changed merge and lose neither side's marks.
Built and tested 6 Nov 2026, not deployed. The Worker is workers/sync/ (config wrangler.jsonc, code src/index.js), the browser side is site/public/js/research-sync.js, the word list site/public/sync-words.json, 12 tests in site/test/sync.test.js against an in-memory R2. Layout in the bucket: meta/<id>, latest/<id> (never expires), v/<id>/<day> (one old version a day; David's lifecycle rule expires v/ after 30 days). The passphrase is six words from a 1,024-word list (60 bits), the keys come from it by PBKDF2 (600,000 rounds) and HKDF, the blob is AES-GCM bound to the id, and the server keeps only a hash of each token. Left: the My research screens (row 125), David's deploy (row 124), a rate-limit rule, a lifecycle rule on the copy bucket, and a copy at a second provider (the second bucket is in the same Cloudflare account, so it guards against a mistake, not against losing the account).
Opt-in community layer: let readers' shared research show what others have walked, kept apart from the checked record
The goal in David's words: research one person does, others can also use. How, without weakening the atlas's one rule. A shared mark is shown as reported by N readers, not checked, in its own layer, never mixed into the family archives' verified marks, and only as counts (a volume, how many readers say it was walked, the latest date), with a threshold so no single reader can be identified from a small place. A reader can withdraw any contribution.
Needs the packs and the sync service first, and a plan for spam and wrong marks. Until then, the way to share is a pack.
Fill the thinnest common names first: David has one form, and 267 of 630 families have three or fewer
Measured on 5 Nov 2026. The given-name catalogue has 630 families; 152 hold a single form and 267 hold three or fewer. David is one form across 14 languages, so a search for it offers no Davide, Davidis, Dávid. Among the biblical and saints' names that fill baptism registers, the thinnest are David (1 form), Marta (4), Marco (6), Ilija/Elias (7), Luca (7), Orsola (8), Simone (9), Ruth (9), Martino (10), Sarah (11), Elena (12) and Madalena (12). Not in the catalogue at all: Ezekiel, Saul, Judah, Ezra, Tamar, Salome, Felicity.
Order of work. (1) The 51 biblical and saints' families, thinnest first, then the missing seven. (2) Then every family with three forms or fewer, ordered by how often a register has met the name (seen, people). Rare names come last.
Method (unchanged). Researcher patches from Wikidata, Wiktionary and Wikipedia only, each form's quote checked on its page (verify.py), merge with merge.py, given_name_variants.py check, then publish. Latin case forms (Davidis, Davidem, Davide) matter most for David, because a priest wrote the case the sentence needed. Meanings for these names come after, 20 of 630 so far.
A read API and a remote MCP server for the whole record search platform, on Cloudflare
Status 8 Nov 2026: phase 1 (the read API and the MCP server) is built and tested in workers/api and is NOT deployed. Deploying it is David's, from workers/api/README.md. Nothing else is waiting on a session.
Today nothing exists. There is no MCP server, API or Worker in the Atlas or the archive repositories (searched 6 Nov 2026), and no MCP server in the user config. The platform is static files: the place and surname shards, the catalogues, the volumes, the providers. The CLI tools (surname_variants.py terms/cover, given_name_variants.py, place_names.py at) are the only programmatic door, and only for sessions on this machine.
Read side first. A Worker on api.recordatlas.org that answers from the published files (fetched and cached, so no database): a surname with its variants and countries, a given name's forms, a place's names at a year, the volumes and archives for a place, the providers for a country, and cover (what to type before writing a nil). Every answer carries asOf and the build id, so a caller can say "published" or "live". A remote MCP server on mcp.recordatlas.org (Cloudflare's Agents SDK, streamable HTTP) exposes the same calls as tools, with the catalogue rules in its server instructions (typography is never a spelling, nothing about a person, cover before a nil), so any app that connects follows them without having our skills.
Rules that carry over. Public data only, rate limited, no passthrough to a third-party source and no credentials of anyone's ever stored. Sits beside the sync Worker but is a separate Worker with its own trust model.
Writes through the API are proposals: checked by the same gates, approved by David, applied through the catalogue tools
Decided 6 Nov 2026 (David). Other people's apps may use the API, possibly hundreds of users. Only David approves a proposal (an admin group may be added later). Git stays the source of truth.
Why not a direct write. The catalogues are written only through scripts/*_variants.py and place_names.py, which refuse a form without evidence and anything that reads like a person, and git is the audit trail. An API that edited data directly would bypass all of that.
Design. Tools such as propose_surname_variant, propose_given_name_form, propose_place_name, log_search (a searched.json row) and propose_provider validate against the same rules as check-data.py (evidence required, positive evidence only, no person-shaped text, the waypoint rule), store a proposal with an id, and answer proposed. A proposal is applied only after approval, through the existing tools, and reports applied, then published, then live: three different states. A proposer can withdraw it.
Because there may be hundreds of users. Keys per app and per user, with a scope (which tools) and a quota; rate limits at the edge; duplicate proposals folded into one with a count of how many people made it; an evidence check that fetches the cited page and finds the form on it (as verify.py does) before David ever sees the proposal; a review page for David behind Cloudflare Access, batch approve and reject with a reason that goes back to the proposer; abuse handling (revoke a key, block a pattern). Proposals never carry a person: a name or date of a living person is rejected at the door.
Order. After the read API. Needs the Cloudflare account.
Set up the Cloudflare account for the sync service, the API and the MCP server
One account, three Workers, separate trust. sync (encrypted personal files, R2), api (public data, read), mcp (the same calls as tools). Subdomains of recordatlas.org, with workers.dev as a stopgap. R2 buckets for the sync blobs and for a nightly second copy; secrets for write keys. David creates the account and the deploy token and holds them; a session never enters a token. First check where recordatlas.org's DNS is managed. Confirm current free-tier limits and whether a card is needed.
Done 6 Nov 2026. recordatlas.org is Active on David's Cloudflare account (records DNS-only, the site and HTTPS unchanged), buckets record-atlas-sync and record-atlas-sync-copy exist (the older rowland bucket is not ours and is never touched), a 30-day rule on v/ is set, and David holds the token record-atlas-workers (Workers, routes, DNS and zone read for this zone; no R2). A session never holds it.
Record Atlas: a «who holds the records» catalogue — custody by parish, record kind and period, with how to ask (design agreed 5 November 2026)
The question that started it, 5 November 2026: the Franciscan friary at Trsat replied that the parish of St George holds registers only from after the Second World War and that everything older is at the State Archives in Rijeka; Google's AI overview said much the same, unsourced. We should not have needed Google, and we should not have needed to ask. This archive's own answer, the 1946 and 1949 rule about where Croatian registers went, already sits in the Atlas (hda883-holdings.json), but it is a paragraph, not a lookup.
What the Atlas has today, and what it lacks. providers.json has 569 institutions, and one of them carries a holdings list (IrishGenealogy.ie). provider-contacts.json has 143 emails, all from Wikidata, none from a reply or an official page. fs-volumes.json says which register is on FamilySearch and where, which is the copy; nothing says who holds the original, who holds a duplicate, who will answer, in what language, or what they need before they will look (DARI's first answer to us, on 5 November, was that the request lacked the applicant's postal address).
The design, in the pattern of the surname and place catalogues.
| piece | what it is |
|---|---|
data/record-custody.json in the Atlas, written only through scripts/record_custody.py | one row per place or parish, faith, record kind, years, holder, role; the role is original, duplicate, civil-registry copy, microfilm, images or index; at PLACE YEAR KIND prints who holds it and in what order to ask; add takes the evidence; check; publish once |
| a status on every row, as for spellings | verified (the holder's own reply or catalogue), catalogue (an official finding aid), rule (a law or a decree, applied to a whole jurisdiction), lead (a secondary source, never a finding: an AI overview, a forum, a tree). A «not held» is allowed only on the holder's own statement, the same rule as a nil |
| observed contacts, beside the Wikidata ones | an email, form, postal address and language that worked, with the date it was seen, what the holder asked for (applicant's address, a fee, the signature of a register) and how long the answer took; one row per holder, not per letter |
| the outcome of every letter written back as evidence | a reply turns a lead into verified (the friary's reply is the first row), and a nil into a dated answer; nothing about a person goes in, only the institution, the register and the period, as in the name catalogues |
| three surfaces | the place panel («Who holds the registers»), a provider page («what they hold, how to ask, what they need first»), and the errands page, whose addressee, language and required fields come from the catalogue instead of from the last letter |
Phases. (0) schema, tool, checker and a user-level skill record-custody, like surname-variants; (1) seed it from this archive's own letters and reads (Trsat, Bakar, Draga, Ledenice, Bribir, Kraljevica, Senj, Pula, the Pazin and Rijeka State Archives) and from the Croatian 1946/1949 rule; (2) the other jurisdictions where one of the family archives has a line (Italy, Slovenia, Austria, Hungary, Uruguay, South Africa), country rule first, parishes after; (3) the three surfaces; (4) outcomes written back from errands.json and searched.json. Not in the plan: a scrape of anyone's AI answer, a scrape of an archive's contact page without a human reading it, and any address inferred from a parent body or a homepage (the rule provider-contacts.json already keeps).
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 283–283); the Atlas is where the work is done, so it is tracked here too.
Seed the custody catalogue from what this archive already knows — Trsat and the Rijeka State Archives first
The first rows of the custody catalogue are already in this archive's mail and logs. Trsat parish registers: the friary says the parish holds only post-war books and the earlier ones are at the Rijeka State Archives (doc, reply of 5 November 2026); the Trsat deaths of 1803–1815 are on no FamilySearch film (1737–1803 and from November 1815 are), so who holds that one book is the open row, and DARI was asked on 4 November and, after it wanted a postal address, again on 5 November. Bakar, Draga, Ledenice, Bribir, Kraljevica: every volume read here has a waypoint, so the images row is known; the originals row is not, and is the first thing a letter to DARI should ask. Pazin: the family's own copy of the Pazin films (DAPA) and the Pazin State Archives. The 1946/1949 rule (registers with entries from 1860 to the civil registry offices, up to 1860 to the archives) is a rule row for every Croatian parish at once, with the quotation from Milić 2017 that the Atlas already holds.
To do, in this archive and then in Record Atlas: list every holder this archive has written to or been answered by (errands.json, searched.json, the requests/ letters), with date, language and what they asked for; write them as rows in the Atlas once the tool exists; until then keep them in the searched row of the letter, not in a new table here (a second copy is how the spellings list drifted).
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 284–284); the Atlas is where the work is done, so it is tracked here too.
A `record-custody` skill: ask the catalogue who to write to before any errand, letter or «not on film» nil
A skill, as there is for names: record-custody, to be run before any errand, letter or email to an archive and before any nil of the form «not on film». at PLACE YEAR KIND says who holds the original, the duplicate and the images, in the order to ask, with the language, the address or form, and what they need first (the postal address DARI wanted, a fee, a signature). add records every reply as evidence; a letter is not finished until its answer is a row. It reads the Atlas, as the surname and place skills do, and keeps no table in an archive.
Why a nil needs it. The Trsat death book of 1803–1815 was written down here as «not on the films» on the strength of the film list alone; the question «who holds it» was first asked by email on 4 November and answered on 5 November by a friary and a registry office, not by FamilySearch. A nil about a register is a nil about where it is, and that is what this catalogue is for. Blocked on the Atlas tool (the row above): write the skill's text now, ship it when record_custody.py exists.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 285–285); the Atlas is where the work is done, so it is tracked here too.
Record Atlas errands: one shared module, so any archive or visitor sees who holds the book, what they need and what others were told — decide the data model and the privacy line
Why this is worth doing, and where it is not. The errands are already the same job in every archive: a gap that reading cannot close, a holder who can, a letter, a wait. They are kept in each archive's own errands.json (40 here, 27 unsent; the Atlas keeps four of its own), so the same holder is written to by several archives who never learn from each other's replies: the postal address DARI wanted on 5 November is a fact every later letter to Rijeka should already carry. The Atlas already receives research sessions and already will know who holds what (rows 100–102); the errand is the missing third thing, and it is the one that turns the catalogue from a reference into a service: a researcher at any archive, or any visitor to recordatlas.org, sees who holds the book, what they need, in what language, what others asked, and what they answered.
What may and may not leave an archive. The line already drawn for the Atlas (surname strings and counts, no given name, no full date, no relationship, no identifier, nobody living) holds, and it decides the design: the letter stays where the person is; the Atlas holds the shape of the ask. A shape is holder · register (place, faith, kind, years) · what is asked (does it exist, where is it, a transcript, a scan, permission to publish) · state · outcome · dates · evidence link. The person's name, the reason and the draft stay in the archive that drafted the letter. Letters are generated in the archive, from a template and a holder profile pulled from the Atlas; the Atlas never holds a draft with a person in it.
Three guardrails, which are the reasons I would say yes and not just «go». (1) Nothing sends from the Atlas, as today: sending is a person's decision, and a shared module must not become a way to mass-mail archives that answer by hand (DARI's office answered ours within a day, and one person reads it). (2) Answered once, published for everyone: the module should show «three researchers asked DARI for the Trsat 1803–1815 deaths; reply of 5 November: …» before it offers to draft a fourth letter, and should bundle every open ask for one holder into one letter; that is the load-reducing reason an archive would not mind it. (3) Institutional contacts only, from the holder's own page or reply; no named staff, no personal address, no scraping.
What it is not. Not a server and not accounts: recordatlas.org is static and my-research is local to a browser, which is the right default for other users (their errands stay in their browser or an exported file; the Atlas's JSON is what they read). Writing back is for sessions, through a script that publishes only the published file from a fresh clone, as publish-research-sessions.py does. A visitor's «tell the Atlas the answer» is a later, optional, reviewed step (a form that opens a pull request), never a live database.
Order of work, one row each below. (1) the data model and the privacy line, agreed by David; (2) the Atlas errand catalogue and the sync script, after the custody tool; (3) bundled letter packs and templates by language, with the holder's required fields as a checklist; (4) the public module on recordatlas.org, local-first; (5) outcomes written back, with turnaround and fee statistics and the skill that ties it together.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 286–286); the Atlas is where the work is done, so it is tracked here too.
Errand catalogue in the Atlas: asks as shapes (holder, register, kind of ask, state, outcome), written by sessions, read by every archive — after the custody tool
Models: scripts/surname_variants.py and scripts/publish-research-sessions.py. A new data/errand-asks.json, written only through scripts/errand_asks.py (add, state, outcome, open --holder DARI, check, publish), published from a fresh clone and touching only that file. Migration: this archive's 40 errands and the Atlas's own four, shape only: the reader of errands.json here sees who, ref and ask, and the Atlas gets the holder id, the register and the kind of ask; why, draft and key stay. A check refuses a given name, a full date of a person, an identifier or a living person in any field, as the name catalogues' checks do. Dedupe by holder + register + ask, so a second archive's identical ask joins the first instead of repeating it.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 287–287); the Atlas is where the work is done, so it is tracked here too.
Holder-bundled letter packs and templates by language, with each holder's required fields as a checklist
One letter per holder, not one per gap: errand_asks.py pack DARI lists every open ask for Rijeka (the Klenovica and Ledenice marriages, the Trsat deaths of 1803–1815, the originals behind every Bakar, Draga and Kraljevica volume read here) with the holder's profile: contact, language, form or email, fee, what they need before they look (DARI: the applicant's postal address with postcode), and the quotation of the last reply. Templates for Croatian, Italian, German, Slovenian and English; the draft is written in the archive, with the person's details, and checked against the holder's checklist before David sends it. Nothing sends from the Atlas.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 288–288); the Atlas is where the work is done, so it is tracked here too.
A public errands module on recordatlas.org: local-first, no accounts, reading the custody and errand catalogues
Same pattern as my-research: the visitor's errands live in their browser or in an exported file, and the page reads the Atlas's published JSON for the holder, the profile, the templates and the answers others have had. Before it offers a draft it says what has already been asked and answered. A «tell the Atlas the answer» button, if it comes, opens a reviewed pull request with the shape only; it is never a live write. The page must say plainly that the archives it names answer by hand, and link their own page first.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 289–289); the Atlas is where the work is done, so it is tracked here too.
Close the loop: outcomes written back, turnaround and fee statistics per holder, and a skill (`record-errands`) that every session runs after sending or hearing back
An errand is not finished until its answer is a row. The outcome codes (found, not held, redirected to another holder, fee, refused, no reply after N days) turn a custody lead into verified (the Trsat friary's redirect to Rijeka is the first example) and a «not on film» nil into a dated answer from the holder. The Atlas shows, per holder, how many asks, how many answered and how long it took, so a researcher can choose the channel that works. The skill is the same shape as surname-variants and record-custody: read first, write with evidence, publish once, say whether it is added locally, published or live.
Raised from the Defranceski archive on 5 November 2026 (its worklist rows 290–290); the Atlas is where the work is done, so it is tracked here too.
Recover marks made before the address moved (daviddef.github.io to recordatlas.org, 6 Oct 2026)
The problem. Browser storage is per address. Anyone who marked volumes at daviddef.github.io/TheMap before 6 Oct 2026 has them under the old address, which now redirects, so no page there can export them. They are probably still in the browser, unreachable.
For David. Run backups/research-2026-11-05/save-browser-storage.js in the console on https://daviddef.github.io/ and on https://recordatlas.org/my-research/, then import the file on My research (it reads that format) and say how many marks and lines each held.
For everyone else. Publish one small page on the old address (the daviddef.github.io user-site repository) that reads the old storage and hands over a research file or a link to /shared/. Only a page on that address can read it.
The 5 Nov backup is in backups/research-2026-11-05/ (all eight archives' searched.json, the Atlas published copies, a git bundle).
Push the Defranceski Family repository: it was 37 commits ahead of GitHub on 5 Nov
Real research (the Ledenice registers, the Worklist custody catalogue) existed only on this machine, plus seven uncommitted files. It is bundled in backups/research-2026-11-05/git-bundles/. A session did not push it because the commits were not its own. Decide, then push; the other seven archives were fully pushed.
Name pages: meanings for the other 610 names, and who carried the name beyond the first 20
Pilot state: 20 names have 36 sourced explanations (data/name-meanings.json) and 20 have bearers from Wikidata (data/name-bearers.json, harvest scripts in scripts/bearers/: add the name's items to map.json, harvest, summarise, check-data). Thin names come first (see the row on common names). A meaning must have its quote found on the cited page; disagreement is shown, not settled.
Stories of names: Magyarisation, the 1787 Habsburg Jewish naming edict, the 1812 Prussian edict; the Zara date for the 1927 story
Two stories are live (Ellis Island; the 1927 Italianisation). Each new one needs sources beyond Wikipedia, numbered notes, and a "what this does not say" list. The 1927 story could not read the decree itself and omits the Zara date; no source gives given-name pairs such as Ivan to Giovanni.
The Archive Kit copy of the name finder still shows empty language cards
A separate repository (SayYourName.astro in the Archive Kit): the Atlas copy hides a language with nothing in it; the Kit copy does not. Merge the change three ways, never overwrite from a peer copy.
Cross-check the 630 given-name families against Wikidata: same-as, different-from, and per-language labels
Wikidata (CC0) has 104,954 given-name items, 25,961 same-as links and 19,037 different-from links. Use them only as lead candidates through verify.py, never as evidence; a form still needs a page that shows it.
Ingest open-licence name counts: Poland and the US first
Checked on 5 Nov, nothing ingested. CC0: Poland dane.gov.pl (given names, surnames of the living), US SSA and Census via data.gov. Credit lines needed: UK ONS (OGL), Norway SSB, Slovenia SURS, Hungary KSH (top 50), Finland DVV (licence seen on a third-party page only: verify). Nothing stated: Croatia DZS, Sweden SCB. No: Meertens. Counts must not describe a person.
Review the hold list of Croatian and Serbian given-name forms
Croatian and Serbian forms are thinner than Slovene in the catalogue and some candidate forms were held back as unverified. Each needs a fetched page that shows it; none is written from memory.
Reading the record: terms data and its gate (first 150 terms of Latin church registers)
Plan: docs/reading-the-record-plan.md. 48 terms are written in data/record-terms.json with a gate in check-data.py (every term sourced, own words, nothing about a person). 100 more to write, from fetched pages only; German, Italian and Hungarian words come after Latin.
Reading the record: term finder and date decoder at /read/
Plan: docs/reading-the-record-plan.md. Built and tested (finder tolerant of endings, i/j, u/v, 7bris; decoder for digits, Roman numerals and Latin words, with its working). Pushed means on the next deploy. Not yet: the French Republican calendar, feast-day dating, Julian/Gregorian by place.
Reading the record: the annotated register page with three layouts (Latin prose baptism, Hungarian tabular births, a printed civil act)
Plan: docs/reading-the-record-plan.md. Drawn by us, tap a field to see what it is, what it does not tell you and how it varies. Needs the layouts file and a source for each field.
Reading the record: the trap list and the Read this line tool
Plan: docs/reading-the-record-plan.md. Misreadings with evidence (uxoris read as ignoris, infans is a child, natum means a different birth day, n.n., sub conditione, delayed marriage). Read this line joins the case forms in the given-name catalogue, so a name found in a register links to its /name/ page.
Reading the record: the learning loop from our own research
Plan: docs/reading-the-record-plan.md. A lesson row with no person in it; a one-off harvest of the eight family archives' notes for recurring patterns; a section in the record-atlas-feedback skill; links from volume and name pages. Each lesson is labelled ours or public. Later one more proposal tool on the API.
Reading the record: practice entries and a handwriting gallery
Plan: docs/reading-the-record-plan.md. Invented entries to read with the answer hidden; handwriting from open-licence sources only, the licence shown. FamilySearch frames stay link-only.
Reading the record: more languages and countries, one pass each
Plan: docs/reading-the-record-plan.md. After Latin church registers: Hungarian, Italian, German (Kurrent), Croatian, Polish; then civil registration (Napoleonic, Austro-Hungarian), Dutch, French, Spanish, Portuguese, Swedish.
Reading the record: whether to ask any guide author for permission to cite or reuse a worked example
Plan: docs/reading-the-record-plan.md. The Belgium handout (Liz Hutchison, BYU Family History Library) is credited with its address. Low priority: nothing is built on any author's text, so nothing waits on this.
Deploy the sync Worker to sync.recordatlas.org (David runs the commands)
Commands are in workers/sync/README.md: put the token in your own shell, dry-run, deploy, check /v1/health, add a rate-limit rule, add the 30-day rule to the copy bucket. A session prepares but never holds the token. Do it after the tests pass on the day, and say 'deployed', not 'live' until health answers.
Wire passphrase sync into My research (make a passphrase, recovery sheet, sync now, restore, older copies, delete everything)
Closed 8 Nov 2026 in the site (Connection & Backup page: six words, Sync now, stop, delete, privacy paragraph; always-on sync in e4a26cf). It does nothing until the Worker is deployed: that is row 124, David's.
Needs the research backup items applied first (docs/research-backup-wip/HANDOVER.md), because sync saves and restores the same research file. Uses site/public/js/research-sync.js. Also: a privacy-page paragraph saying what the server holds (an id, two token hashes, bytes it cannot read) and what it cannot do, and a rewrite of the privacy test that names the modules allowed to keep anything in the browser.
Sunday 8 November: work out what it would take to make this repository private and share the atlas from a new public one
Scoped 8 Nov 2026: docs/private-repo-scope-2026-11-08.md. The deciding fact is Actions minutes: a deploy averages 108 minutes and a private repository on Pro has 3,000 a month (about 28 deploys). Needs David's decision on the three questions at the foot of that note.
David: Record Atlas has become a big deal and may warrant better protection. To scope, not to do yet: what must stay public for the site to work (GitHub Pages from a private repo needs a paid plan, or the site moves to Cloudflare Pages), what the new public repo would hold (built site, data files, scripts worth sharing) and how it is published from the private one, what is in the history that should not be public (the large world-data release asset, family-archive names, any token or key ever committed), the other sessions and archives that push here and the deploy and e2e workflows that depend on the repo name (daviddef/TheMap), the domain and CNAME, the licence for the shared data, and the cost and the rollback.
Harvest more place-name changes, country by country: Australia, South Africa, the USA and others where names are changing now
David: place names are changing now, in Australia, South Africa, the USA and elsewhere, and a record is filed under the name of its year, so the atlas has to follow how each name moved over time. Where it stands, 6 November: /place-names/ holds 375 places and 529 dated names in 52 countries (92 of them Croatian), and 20 of the Australian entries are undated native names marked lead. Most places have no dated name at all; the other layers are undated (the gazetteer's GeoNames alternates on 1.7 million places, and the names on 3,629 places).
To do:
1. Pick the countries where names are actively changing or were changed in living memory: Australia (dual and reinstated Aboriginal names), South Africa (cities, towns and provinces renamed since 1994), the USA (federal and state renamings), then Canada, New Zealand, India, Zimbabwe, the former Soviet states, Ireland and Turkey, in that order of how much a family researcher is likely to hit.
2. Harvest dated names from Wikidata (a name with start and end dates, and the label in each language) and from the national place-name boards, one country at a time, each as a small fetch with the source recorded. Nothing is added without a source and a date; a name with a source but no dates is a lead.
3. Write them with scripts/place_names.py add (never by hand), so the catalogue, the search, the place pages and /place-names/ all pick them up. Record the cause of each change (a ruler, a regime, a reversal), and keep names that changed more than once (a name can go back).
4. Show the change over time on a place page as a timeline, not a list, and let the find box and the map's year slider search under the name in use in that year.
5. Do the same for country and region names in other languages (Hrvatska, Croazia, Kroatien), which no layer holds yet.
6. Family archives do not keep their own tables: the six forms the Defranceski archive has that the Atlas lacks (Golgorizza, Ceppici, Mion, Mione (Ovaro), Telve im Suganertal, Torrington, Connecticut) go in through the same tool.
Not started.
Later: a paid "1-page family deep dive" offered from a finished free story, with a research-cost estimate
Deferred on purpose (David, 9 November): first get the collection and story gathering right, then offer help. The idea: a paid, fixed-scope "1-page family deep dive", the research pages made with the start-family-page skill (Apollonio & Champion, Vidaković & Vrban), offered from a finished free story. Not started; nothing here is to be built until the free half is trusted.\nShape of it. The free story says what the person told us. The deep dive adds what a record shows: a tree split into confirmed and candidate, evidence images with links, the jurisdiction traps for that place and year (data/regions.json), a worklist and a searched list, and where to look next. One status vocabulary on both pages: told / found in a record / candidate / not found. A walked-pages nil counts as a result. The promise is about the work done, not the result; archive fees are passed through.\nA cost estimate comes first, and it is free. From the volumes and page counts the Atlas already holds for the places and years in the story.\nRisks to settle before any price. (1) FamilySearch terms for paid reading through a personal account, the biggest; (2) living people, privacy, and consent to show a page as a portfolio piece; (3) AI misreadings: a human review step and a refund rule; (4) capacity with one account and one reviewer; (5) neutrality: the Atlas never withholds data to push the paid service.\nRoadmap. Free half first (story gathering, status vocabulary, story export as a page); an intake form; three to five manual paid pilots to time and price; automate only what repeats.\nOpen questions. Strangers or genealogy-group people first; whether David reviews every dive.
Start tour: wire the hero search to the real find index (the tour's search box currently matches a small hand-built table, labelled as an example)
The story tour's first scene has a search box. It answers from a few examples built from the catalogues (Fiume, Žubrinić, Salisbury) and sends anything else to /?q=. Wire it to the same find index the map's search box uses, so a typed place or surname answers in the tour itself. Deliberately not in the first pull request (David, 19 Nov 2026). Code: site/public/js/start-tour.js (MATCH), data in site/src/lib/start-data.js.
Nothing in this archive is in that state.
The pages that answer what is known are the map, Countries, What's missing and What changed.