Legal Software: Mill: Following the FISA Court: New Website, New Data

sâmbătă, 7 iunie 2014

Mill: Following the FISA Court: New Website, New Data

Eric Mill has posted Following the FISA Court: New Website, New Data , at his blog.

Here are excerpts from the post:

Short version: The FISA Court has a new website, and @FISACourt is way smarter because I’m turning their entire docket into data.

Longer version:

One year ago [...] the Foreign Intelligence Surveillance Court (also known as the “FISA Court”, or “FISC”), which oversees surveillance policy in the US, began publishing a docket after 35 years of silence.

Ever since, I’ve been following the Court by running @FISACourt on Twitter [...]

@FISACourt is powered by an open source tool I wrote to watch the Court’s public docket site. Within 5 minutes of any change to the docket, the tool automatically posts a tweet with a link to the HTML change, emails me, and texts my personal cell phone. When that happens, I quickly investigate and read anything that was posted, and follow up with a hand-written explanation. [...]

In another step, the FISA Court recently relaunched their site, now at fisc.uscourts.gov. [...]

Naturally, the new website immediately broke @FISACourt. I decided to see it as an opportunity, and rewrote the whole tool to produce a comprehensive, up-to-date dataset of the Court’s public docket.

I’m now doing a proper crawl of the entire FISC website, and saving data for every filing. [...]

That data is in YAML, a humane data format that lends itself well to human readability, and line-by-line diffs.

I’m also downloading the actual PDFs, and watching them for any future changes. As the Court showed with its unexplained withdrawal detailed above (and as the Supreme Court frequently demonstrates), it’s important to watch the public record for revisions of its history.

The data and PDFs are versioned in their own branch, so you can use GitHub’s dedicated Atom feed to watch every changes the system detects, or use the GitHub Contents API to sync up. [...]

If you want to use the FISA Court’s data, or the scraper, or have any questions at all, please open an issue over at the project. It’s all public domain and you don’t need my or anyone’s permission, but I love to help out and talk about this stuff. [...]