Laravel open source crawler application to build tutorials

First, why is your crawler always pulled by the site?

Brothers who have engaged in data collection must have encountered this situation: just run a good crawler program, suddenly the403 Denial of Accessup, or get a bunch of CAPTCHAs. To put it plainly, your IP is being targeted by the website. Ordinary crawlers with fixed IP frantic request, just like holding a loud speaker in front of people's homes shouting "I want to steal data", not block you block who?

And here comes our savior--proxy IPThe IP address is changed every time a request is made through specialized services such as ipipgo. Through ipipgo this kind of professional services, each request for a different IP address, equivalent to the crawler to wear a myriad of "face masks". For example, the original 1,000 requests in an hour with 1 IP, now replaced by 100 IP turns to send, each IP only sent 10 times, the site wind control system can not detect abnormalities.

Second, hand to build Laravel crawler

Let's not rush into writing code. We need to get our stuff ready:

1. Installing PHP 7.4+ and Composer
2. Create a new Laravel project:composer create-project laravel/laravel crawler
3. Install Goutte, the crawler tool:composer require fabpot/goutte

The core code is really just three pieces (don't let the technical jargon scare you):

// Create a new CrawlCommand.php in app/Console/Commands.
public function handle()
{
    $proxy = 'http://用户名:密码@gateway.ipipgo.com:9020'; // ipipgo's proprietary proxy format
    $goutte = new Client();
    $goutte->setClient(new HttpClient(['proxy' => $proxy]));

    // Specific capture logic...
}

Third, the actual skills of proxy IP

It's not enough to be able to use proxies, theselife-saving techniqueMust be mastered:

Special Reminder: Don't save the timeout! It is recommended to set connect_timeout to 5 seconds and request_timeout to 30 seconds to avoid the whole program getting stuck on a certain IP.

IV. QA time: a must for novices

Q: What should I do if the proxy IP often fails to connect?
A: 80% are using low quality proxies. Recommend ipipgo'sBusiness Level Agent PackageThe IPs in their house have survival detection, and the measured connection success rate can go up to 99.2%.

Q: How to break the collection speed is too slow?
A: Two tricks: ① use ipipgo'smultithreaded agent pool② Enable HTTP persistent connection to reduce the number of TCP handshakes.

Q: How can I tell if a proxy is anonymous?
A: Visit http://httpbin.org/ip, if the proxy IP is returned instead of your real IP, it means that ipipgo's high stash proxy is in effect.

V. Upgrade Play: Distributed Crawler

When the standalone can't hold up, it's time to get on thedistributed architectureUp. Use Redis to do the task queue, multiple servers running at the same time, each machine from ipipgo apply for a different IP segment. This is not a dream to collect millions of data per day, and it is not easy to be targeted by anti-crawling strategies.

One final rant: don't go cheap when choosing a proxy service! Some free agents willStealing response contentorRecord your request dataThe first thing you need to do is to use a regular service provider like ipipgo. With ipipgo this kind of formal service provider, data security is guaranteed, out of the problem can also find technical customer service real-time processing.