TL;DR: Web scraping's legality depends on several factors: what you're scraping (publicly available facts vs. copyrighted content), how you're scraping (respecting rate limits vs. overwhelming servers), what you do with data (internal research vs. republishing), and jurisdiction (US courts favor data collection, EU GDPR is restrictive). No single law forbids scraping, but contract law (ToS violations), copyright law, CFAA, GDPR, and CCPA create a complex landscape. Following best practices—respecting robots.txt, adding delays, not scraping personal data—mitigates most risk.
The Current Legal Landscape
There is no law that explicitly prohibits web scraping in most jurisdictions. However, multiple overlapping legal frameworks restrict scraping in different ways. The legality of any scraping project depends on specific facts: what is being scraped, how it's being scraped, what's done with the data, and which jurisdiction's law applies.
The general principle: factual data (prices, addresses, product specifications) is harder to protect legally than creative content (articles, photos, code). Data collected passively from public websites is legally safer than data requiring authentication or violating terms of service.
Courts increasingly distinguish between:
- Data scraping (collecting facts): Usually legal if conducted reasonably (respecting rate limits, not using credentials to bypass restrictions)
- Content scraping (copying creative works): Often illegal without permission, regardless of scraping method
- Account harvesting (accessing protected data): Potentially illegal under CFAA if done without authorization
Key Court Cases
hiQ Labs v. LinkedIn (9th Circuit, 2022) was the most significant scraping case to date. hiQ scraped public LinkedIn profiles to build a talent analytics platform. LinkedIn sent cease-and-desist and blocked hiQ's IP. hiQ sued, claiming the CFAA (Computer Fraud and Abuse Act) didn't apply to public data.
The court ruled in hiQ's favor: scraping publicly available data from the internet, even when violating a website's ToS, doesn't violate the CFAA. The CFAA applies to unauthorized access—accessing someone else's computer system without permission. Publicly available data on the internet isn't someone else's computer; it's the public internet.
However, the court didn't address copyright or contract law. LinkedIn could still have sued for copyright infringement or breach of contract (violating ToS). The ruling simply said CFAA doesn't apply to public web scraping.
LinkedIn v. Rapid Intelligence (2014) saw LinkedIn successfully block the competitor using terms-of-service violations and breach of contract, even though the scraping didn't technically violate the CFAA.
Clearview AI lawsuits (2020-2022) resulted in a $100M settlement after scraping billions of photos from public social media without consent. This case involved scraping personal data (photos, biometric data), not just public facts, and directly violated state privacy laws.
Key takeaway: courts increasingly protect publicly available factual data from scraping restrictions, but personal data and creative content remain protected under different laws (privacy, copyright).
Key Regulatory Frameworks
| Framework | Jurisdiction | Impact on Scraping | Key Restriction |
|---|---|---|---|
| CFAA (Computer Fraud and Abuse Act) | USA (federal) | Low (post-hiQ) | Prohibits unauthorized computer access; doesn't apply to public websites |
| Copyright Law (DMCA) | USA (federal) | High (if scraping creative works) | Prohibits circumventing copy protection; applies to protected creative works |
| Terms of Service / Contract Law | USA/Global | Medium | ToS violations can be breach of contract; can't be enforced against non-users |
| GDPR (General Data Protection Regulation) | EU + UK | High (for personal data) | Prohibits processing personal data without consent; applies to EU residents' data globally |
| CCPA (California Consumer Privacy Act) | California | High (for California residents' personal data) | Gives consumers rights to know, delete, opt-out; applies to CA residents' data globally |
| LGPD (Brazilian Data Protection Law) | Brazil | High (for personal data) | Similar to GDPR; applies to Brazilian residents' data globally |
| Copyright (General) | Global (country-specific) | High (for copyrighted content) | Prohibits copying creative works without permission; factual data not protected |
GDPR and European Compliance
GDPR fundamentally restricts scraping of personal data from the EU. "Personal data" is broadly defined: names, email addresses, IP addresses, location data, or any information that could identify an individual.
Under GDPR, you cannot process personal data without a legal basis. Valid bases include:
- Explicit consent (rarely given for scraping)
- Contract fulfillment (you're providing a service to the user)
- Legal obligation (required by law)
- Vital interests (emergencies)
- Public task (government services)
- Legitimate interests (if not overridden by individual rights)
Scraping names and emails from a public website for marketing purposes lacks a valid legal basis under GDPR. This is why Clearview AI was fined: scraping photos and biometric data of EU residents violated GDPR.
However, scraping non-personal factual data (product prices, availability, review ratings) is less restricted. The risk increases dramatically when scraping email addresses, phone numbers, or identifying information.
CCPA and California Law
CCPA gives California residents rights regarding their personal data but is less restrictive than GDPR for data collectors. The law primarily gives rights to individuals (right to know, right to delete, right to opt-out), rather than prohibiting collection outright.
However, scraping personal data of California residents may create CCPA obligations:
- If individuals request their data, you must provide it
- If individuals request deletion, you must delete it (under certain exceptions)
- You must disclose your data collection practices clearly
The restriction is weaker than GDPR but still meaningful. Many companies that scraped data before CCPA was enforced found themselves unable to comply with deletion requests because they had no tracking of where personal data came from.
robots.txt and Site Policy
robots.txt is a convention, not a law. Ignoring it doesn't violate any legal requirement, but it's increasingly treated as important:
- Courts view robots.txt as evidence of intent: If a site explicitly disallows scraping in robots.txt, ignoring it strengthens the case against the scraper in terms-of-service litigation.
- Search engines follow robots.txt: Google respects robots.txt directives. Non-compliance has minimal direct legal consequence but affects SEO and search visibility.
- Rate limit signals: Some robots.txt files specify crawl-delay directives (scrape no faster than X seconds between requests). Following these is good faith effort to minimize server impact.
Best practice: respect robots.txt even though it's not legally required. It demonstrates good-faith effort to minimize impact and strengthens your position if a website owner disputes your scraping.
Best Practices for Legal Compliance
Following these practices dramatically reduces legal risk, even in gray areas:
- Only scrape publicly available data: Don't use credentials to access protected areas, don't bypass authentication, don't scrape private accounts.
- Don't scrape personal data unnecessarily: If you only need prices and product names, don't collect email addresses or identifiers.
- Respect rate limits and robots.txt: Add delays between requests, follow crawl-delay directives, respect disallow rules.
- Don't republish content: If you're scraping articles or creative content, don't republish them as-is. Use data for analysis or aggregation instead.
- Document your scraping policy: Be transparent about what data you collect, how you use it, and how long you retain it.
- Implement GDPR/CCPA controls: If your data includes EU or California residents, implement data deletion and access requests.
- Use proxies ethically: Rotating residential proxies are legal, but using them to hide scraping and circumvent detection is ethically problematic and strengthens legal cases against you.
- Monitor and respond: If a website owner sends a cease-and-desist, cease scraping that site. Continuing after legal notice strengthens their position.
- Consult legal counsel: For significant scraping projects, consult a lawyer familiar with technology and IP law in your jurisdiction.
How Jurisdiction Matters
Scraping the same data is legal in the US, restricted in Germany, and strictly prohibited in France under some circumstances. Key differences:
United States: Post-hiQ, courts favor data collection of public information. CFAA is no longer a major restriction. Copyright law applies to creative works but not facts. Contract law (ToS) can restrict scraping but only applies to parties who agreed to ToS.
EU/UK: GDPR/UK GDPR severely restrict personal data scraping. Data protection authorities are aggressive in enforcement. Even factual data scraping is viewed skeptically if it involves personal identifiers.
Germany/France: Particularly restrictive. German courts have found database rights violations in scraping. French authorities treat certain scraping as intellectual property violations.
China: Very restrictive IP laws. Scraping any data requires government approval in many sectors.
For international scraping operations, the safest approach is to follow GDPR standards globally (most restrictive) and then relax where local law permits.
When to Use Scraping vs. Official APIs
Official APIs are always preferable legally if available. They include explicit terms of use, rate limits, and fair-use policies. However, most websites don't offer public APIs for data that competitors would want (prices, competitor data, inventory).
When an API exists, use it. When it doesn't, scraping is necessary but carries more legal risk. Use residential proxies and best practices to minimize that risk.
Planning a scraping project with legal compliance in mind? Let's discuss how to approach data collection responsibly and sustainably.
