Laravel Simple Crawler Application: PHP Framework Implementation

Why are crawlers always blocked? Try this trick!

The old iron engaged in the crawler understand, the most afraid of encountering the target site suddenly give you aIP Blocking Bundle. A couple of days ago, a friend doing e-commerce complained that they use Laravel to do the price comparison crawler, just run two days to be recognized as a robot. This time we should offer our killer - proxy IP service!

And here's the kicker.ipipgoHome services (absolute tap water recommended), their dynamic IP pool is particularly suitable for the need for frequent IP switching scenarios. To give a chestnut, with their API to get the IP address, each request can automatically change the vest, the site simply can not distinguish between a real person or program in operation.

Second, hand to teach you to jack a crawler with an agent

First the whole basic version of the Laravel crawler framework, here with theGuzzleHttpIt's the least amount of work to make a request library:

// Install the required libraries
composer require guzzlehttp/guzzle

// Create the crawler controller
php artisan make:controller SpiderController

The key code goes like this (remember to replace the proxy configuration with the address provided by ipipgo):

public function fetchData(){
    $client = new GuzzleHttpClient([)
        'proxy' => 'http://username:password@gateway.ipipgo.com:端口号'
    ]);

    $response = $client->get('Target URL');
    // Process the crawled data...
}

Proxy IP Configuration Pit Avoidance Guide

Here's the kicker.timeout settingThis is a pitfall! Many newbies forget to set the timeout parameter and the program gets stuck as a result. It is recommended to add it in the Guzzle configuration:

'timeout' => 30, // in seconds
'connect_timeout' => 10

IV. Practical QA session

Q: Can't I just use a free proxy? Why do I need to buy ipipgo?
A: Nine out of ten free proxies don't work! Previously tested, the average survival time of free IP is less than 15 minutes, ipipgo commercial IP poolsAvailability 98%Above that, there is professional technical support.

Q: How do I test if the proxy is working?
A: Add a debugging interface in the code to return the currently used IP address. Or directly use the ipipgo providedIP Detection Interface, enter the command to see the actual exit IP.

V. High-order play: distributed crawler architecture

When large-scale crawling is required, it is recommended to use theLaravel Queue + Multi-Proxy IPThe combo. Split the crawling task into multiple sub-tasks, each sub-task is assigned a different ipipgo proxy channel, so that the efficiency is directly doubled!

Note when configuring task distribution:
1. Use of separate agent configuration for each queue process
2. Setting up a failure retry mechanism
3. Remember to set it up in the ipipgo backend.IP whitelistingTo prevent the authorization from lapsing

One last rant about being a crawler.stop before going too far (idiom); to stop while one canThe first thing you need to do is to set up a reasonable request interval. Don't make people's websites go down, set up reasonable request intervals, with ipipgo's intelligent scheduling function, can accomplish the task but won't get into trouble. If you have any technical problems, please feel free to leave a comment to discuss them, and I'll get back to you when I see them.