Why Net Scraping is Essential to Democracy
The fruits of web scraping – the use of code to collect data and information from websites – are all around us.
People build scrapers that can find every Applebee on the planet, or collect laws and votes for Congress, or track fancy watches for sale on fan websites. Businesses use scrapers to manage their online retail inventory and monitor competitors’ prices. Many well-known websites use scrapers to track the prices of airline tickets and job vacancies, for example. Google is essentially a giant, crawling web scraper.
Scrapers are also the tools of watch dogs and journalists, which is why The Markup filed an amicus letter this week in the US Supreme Court on a case threatening to make scraping illegal.
The case itself – Van Buren v. The United States – is not about cockroaches, but rather a legal issue regarding the prosecution of a Georgian police officer, Nathan Van Buren, who was bribed to look up confidential information in a law enforcement database. Van Buren has been prosecuted under the Computer Fraud and Abuse Act (CFAA), which prohibits unauthorized access to a computer network such as computer hacking, where someone breaks into a system to steal information (or, as in the classic movie “WarGames”) dramatized in the 1980s). “Possibly start the third world war).
In Van Buren’s case, the question arises whether the court will broadly define its problematic activities as “exceeding authorized access” to extract data, which would make it a crime under the CFAA for being on duty was allowed to access the database. And it is precisely this definition that could affect journalists.
Or, as Judge Neil Gorsuch put it during the oral presentation on Monday, in the direction of “perhaps making a federal criminal out of all of us”.
“A law allowing powerful forces like the government or wealthy corporate actors to criminalize unilateral message-gathering activities by blocking those efforts through the terms of service on their websites would violate the first change,” The Markup wrote in ours Letter.
What kind of work is at risk? Here is a summary of the recent journalism made possible by web scraping:
- The Atlantic’s COVID tracking project collects and aggregates data from across the country on a daily basis and serves as a means of monitoring where testing is taking place, where the pandemic is increasing and what racial differences are, who is contracting and who is dying of the virus.
- This project from Reveal scraped extremist Facebook groups and compared their membership lists with those of law enforcement groups on Facebook – and found a lot of overlap.
- The most recent study of the markup on search results by Google showed that it consistently favors its own products, leaving some websites where the web giant itself is scraping information that is fighting for visitors and therefore advertising revenue. The US Department of Justice cited the problem in an antitrust lawsuit against the company.
- A pattern of cookie cutter laws was found in Copy, Paste, Legislate, USA Today, promoted by special interest groups and propagated in legislations across the country.
This article was originally published on The Markup and republished under the Creative Commons Attribution-NonCommercial-NoDerivatives License.