Compiling Public Information En Masse
This is a detailed explanation of how I compiled the open records laws for parcel and assessor data for each state, as detailed in the following blog post: Mapping Every Property in the United States. One must realize that for every 10-15 minutes x 50 states adds up to around 10 hours, so it’s best to adopt a strategy. It is a good lesson in how one might collect legal information or information in general.
I started with the places I already knew (California and Iowa), and created a spreadsheet to document things as I went. Of course the next thing to do was google around a bit. The best I could find was a great report called Open Records Laws: A State by State Report, from December 2010 by the National Association of Counties (NACO). I used it to find what exceptions I could (just run a search for “geographic”, and you’ll find some exceptions to “geographic information systems”).
Now, for the remaining states, searching state codes would be enormously difficult and may even be impossibly boring. So I then emailed one place in each state to provide the provision of law that justified any fees, and they would often simply point me to the law, usually consulting their jurisdiction’s attorney. I learned this strategy having received advice from a lawyer at the Electronic Frontier Foundation (EFF) when I tackled California, as was noted in the blog post.
To start, I googled things like “GIS fee schedule”, and then I’d get tons of places that had a list of fees for data. For one place in each state, I would email them, copy-pasting the same email, changing the parts in bold below, and noting in the spreadsheet the people I had contacted.
SUBJECT: provision of Washington state law for charging for public data
Can you please let me know what provision of Washington State Law allows Thurston County to charge for both GIS (eg. parcel shapefile) and Assessor data (eg. assessed value of parcels, etc)? Please provide the precise provision, and a link if you can.
If I did not find any in specific states, I would search for GIS data in that state, such as “county Kansas GIS fee schedule”. If I found one, I’d email them. For each of the remaining states, I would then look up the list of counties by population on Wikipedia, and google the largest ones until I either found a fee or found an email address for their GIS person. Where I still couldn’t find a fee, I would ask all questions all at once. This was the more humorous email, a bit strange for the recipients:
SUBJECT: GIS/Assessor data, and any provisions of NE State Law that allow charging
Does Douglas County provide GIS parcel data and assessor data freely?
If so, can you please point to where I can download both of them?
If not, can you please tell me the costs associated with them, and what provision of Nebraska State Law allows a local government to charge for both GIS (eg. parcel shapefile) and Assessor data (eg. tax roll, etc)?
While the most common response was to simply give the provision I needed, responses varied. Sometimes I really had to dig into the law to understand some provision (such as West Virginia’s “tax maps”). In one case, the state of Nevada responded that I had to fax them (another silly rigid “data policy”, as I explain in the blog article), and I used an online fax service for the same email. And a couple of times, I emailed one state, mentioning another, which was embarrassing and I had to correct.
In many cases there was back and forth – often what they gave me sufficed, but often it was inadequate. They would usually give me a law that said that they were required to provide data at the cost of duplication, which did justify their charge (often $50 or hundreds or thousands of dollars). I would then copy and paste a message explaining that this is a file that is readily available, sitting on a computer or can be exported from a program in the same way one would click “save as”. I would also include the following, explaining what the cost of duplication meant:
The cost of duplication in this case would be either free (sending via dropbox or other file sharing service) or the cost of copying and sending a disk (15-30 mins x hourly rate + a few bucks for disk and shipping). I would of course prefer the first option, it would save us all time.
I also use a super nifty tool called “GoFileDrop” and copy-paste the link:
You can upload it using the link below, like how you attach something in an email (file max is 3GB).
Often they would still stand by their policy, in which case I would then rate and color the state as having an open data policy, and assume that the particular jurisdiction I contacted was in violation of their state’s laws. For the remaining states where I couldn’t find any that were charging (a couple in the south and a few in New England), I looked at the report from NACO above, and filled in the gaps.
Using these methods I was able to compile everything in perhaps 30-40 hours, which is pretty good for 50 states + DC. Of course now that I’m proficient at it, I can do the same type of thing much faster. Though for the recipients it was often a bit like mass produced annoyance, it was the most reasonable way of doing it.