Embed Notice
HTML Code
Corresponding Notice
- Embed this notice@p @phnt @cjd
The scraping industry has become quite diverse regarding vertical integration lately.
There are:
- "legitimate" companies selling residential and mobile proxies
- full service companies where you provide a url and you receive the scrape result, some with an API frontend.
(regarding the full service firms the pricing is directly correlated to anti-bot overcoming techniques)
- There are a couple of sites like multi-vendor marketplaces for scrapers. Vendors will provide the spider program, the site operator will run it, and the user will pay per result
It could be one of these fully automated firms working for OpenAI. Many people have the illusion that these neural algorithms are intelligent monoliths whereas it is usually the combination of many neural nets alongside traditional web infrastructure.