Finding and documenting public data sources
Which public records can a project really reach, and how to keep them fresh.
A working sample we built to show how we would solve a problem like this. It is not a delivered project.
In plain terms
A short, real exercise to find out which public sources about building permits and property can be reached for one US city and county, and which cannot. It produced a full extract of permit records, a way to give every record a stable ID, and a safe way to refresh the data without creating duplicates.
- Real data from real public sources, with the original records kept.
- A clear list of what could not be reached, and why.
- Updates add new records without duplicating old ones.
What this does not claim: This was a few hours of work on one county, not a complete inventory. Several sources timed out or blocked automated access, and they are listed as not reached.
Technical details
Discovery run across one city and county: a sources table, a full permit extract with raw records preserved, analysis of which field forms a stable record ID, handling of permits with no suite number, and an idempotent refresh that avoids duplicates. Includes tests and an explicit limits section.
- Python
- Public data portals
Need something like this?
Tell us the problem in your own words. We reply with a plain, practical plan.
Discuss a project like this