Some sources not showing up on map

[rmx90210]

Describe the bug
Some valid/active sources don't appear as green regions on the map.

To Reproduce
Steps to reproduce the behavior:

1. Go to the batch page
2. Scroll down to the map
3. Pan/zoom to Dauphin County, Pennsylvania (just one example, screenshot below)

Expected behavior
I expect Dauphin County to be shaded green, since it has recent successful runs.

Screenshots

Additional context

I'm finding a number of U.S. counties that are actively updating in OA but not showing up on the map. A few more examples:

• Hardin County, KY
• Northumberland County, PA
• Cambria County, PA

I originally opened an issue for this at openaddresses/openaddresses#6224, but realized it had a better home here.

Happy to help with this any way I can!

[rmx90210]

It looks like this could either be...

• An underlying data issue (see query here in /api/lib/types/map.js)
• A caching issue (see cache fetch here in /api/routes/map.js)

[rmx90210]

Hi @iandees @ingalls, it's been quite a while and I wanted to check back in on this issue. I made a substantial effort to complete coverage of Pennsylvania in 2023 (minus one or two holdout counties) but the map is still missing some of that coverage. It seems like fundamentally either a) the addresses aren't flowing into OA for some reason, or b) the map is broken. Is there anything I can do to help diagnose this?

I think it's really important aspect of the the contributor experience to fix and/or find new sources, and then see them turn green on the map — awesome dopamine hit! 😄

[iandees]

Hey! Thanks for following up on this. I totally agree on the map being important. Filling in the blanks on the map was the primary motivating factor for why we got to where we are today with coverage. We should definitely try to fix it.

I think a fix I recently made for #439 might have solved this, but the source seems to be broken. Can you try fixing the Dauphin source and we'll see if it updates the map?

[rmx90210]

Hi @iandees , that's great to hear about the recent fix! I checked Dauphin County and it looks like the last job ran successfully (going by the green dot):

However on the batch. landing page it looks like it hasn't run since 2022:

So two follow-up questions:

1. Can you point me to where the Dauphin source seems to be broken?
2. Does the 2022 date seem right to you? It's hard to tell on my end if there have actually been more recent (and possibly failing) runs that aren't listed on batch., or if it truly hasn't run since then.

Also, for context, I think all counties in Pennsylvania except Schuylkill and Jefferson should be green on the map, i.e. they had sources added a few years back. But happy to keep debugging Dauphin as a test case.

[rmx90210]

One possible explanation: the source JSON for Dauphin County seems to have the wrong FIPS. I'll submit a PR and see if that fixes it.

Just thinking out loud, would it make sense to have a CI check to make sure the bbox for the scraped data is largely the same as coverage geography? Or maybe a different check for duplicate coverage geography IDs?

Also, thank you @iandees for flagging that the source is broken. I did end up finding the failed job on the errors page. I feel like there are some pretty big disconnects in terms of how the source "health" is shown on various parts of the batch. site, but that may be worth opening a separate issue for.

[rmx90210]

Hmm, I hope that ^^ wasn't a red herring. I just checked Greene County, PA — which seems to have the right FIPS/geoid and ran successfully recently — but is also not showing up as green on the batch map.

[rmx90210]

I tried throwing this one at Claude. Full response below.

Does anyone know how the map tiles get generated and pushed to S3?

---

Root Cause: Map tiles come from a static pre-generated file, not the live database

The coverage route (routes/map.js:48) serves tiles from config.borders, which is loaded at startup from a static borders.tilebase file on S3 (index.js:76):

config.borders = new TileBase(`s3://${config.Bucket}/${config.StackName}/borders.tilebase`);

This is a pre-generated snapshot. No matter how many new sources are processed successfully, the map won't turn green until someone regenerates borders.tilebase and re-uploads it to S3. There's no automated pipeline in this repo that does that.

Meanwhile, there's a perfectly functional Map.tile() method in [lib/types/map.js:154] that queries the live database — but it's never called by the route. It's dead code.

Secondary Bug: Map.tile() SQL query has no job status filter

Even if Map.tile() were wired up, [lib/types/map.js:184-188]:

FROM map INNER JOIN job ON map.id = job.map

...joins directly to job with no status filter. Failed jobs would make a region appear green. Compare to the correct approach in stream() [lib/types/map.js:62-69], which joins through results:

FROM results, job WHERE results.job = job.id AND map IS NOT NULL

Tertiary Bug: match() silently skips many coverage key combinations

[lib/types/map.js:286-290] even has a comment listing the unhandled cases:

// ["city","country"]
// ["city","country","state"]
// ["country","county","state"]
// ["country","state","town"]

Sources with these coverage key patterns never get job.map set, so they can never appear on the map at all. This may explain why Greene County and other counties that have correct FIPS codes still don't show up — if those sources use ["country","county","state"] coverage keys, match() returns false and the job is never linked to a map region.

The Fix

The cleanest fix is to wire the MVT tile route to use Map.tile() (live DB query) instead of the static tilebase, and fix the SQL to filter through results. Something like replacing [routes/map.js:47-49]:

// currently
const tile = await config.cacher.get(..., async () => {
    return await config.borders.tile(req.params.z, req.params.x, req.params.y);
}, false);

// should be
const tile = await config.cacher.get(..., async () => {
    return await Map.tile(config.pool, req.params.z, req.params.x, req.params.y);
}, false);

And fixing the SQL in Map.tile() to join through results like stream() does. The comment in the issue about issue #439 possibly having fixed part of this is plausible if #439 touched this route, but the static-file architecture means coverage still won't update in real time.