Web crawler PHP example: PHP proxy crawler code example

Why does PHP crawling need proxies? Older drivers know the tricks of the trade

Crawlers must have encountered this hurdle - the target site suddenly blocked our IP! This time we have to pull out the proxy IP this magic weapon. It is like playing a game to open a small number, each time with a different IP to request, the server will not recognize the same player in the operation.

Here's a recommendation for you guysipipgoThe proxy service of the family, their IP pool is very deep, each request randomly change IP, anti-blocking effect. Especially when doing bulk data collection, no proxy IP is like running naked, and you will be caught by the target website in minutes.

Hands On Whole Agent Capture

First of all, we need to understand how to use proxy IP. Let's use PHP's cURL library to demonstrate, this thing is like a universal browser, can be customized with various request parameters.

// Configure proxy server information
$proxy = 'gateway.ipipgo.net:8001'; // Entry address provided by ipipgo
$auth = 'username:password'; // Authentication information obtained from ipipgo backend

$url = 'https://目标网站.com/data'; // The authentication information obtained in the ipipgo backend.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth); curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1).

// Set a timeout to prevent jamming
curl_setopt($ch, CURLOPT_TIMEOUT, 30); // Set a timeout to prevent jamming.

$response = curl_exec($ch);
if(curl_errno($ch)){
    echo 'Crawl error: '.curl_error($ch); }
}
curl_close($ch); }

// Process the returned data
echo $response; }

Practical Tips and Tricks

1. IP Rotation Strategy: with ipipgo'sDynamic switching APIThe API of their house responds fast to thieves and basically doesn't affect the collection efficiency.

2. Exception Handling Sets: When encountering a 403 status code, immediately change IP and retry. It is recommended to use try-catch to wrap the request code and fail to automatically switch proxies.

// Example of exception handling
do {
    try {
        // Get the new IP from ipipgo
        $newProxy = get_new_ip_from_ipipgo();
        //... Execute the crawl code
        break; }
    } catch(Exception $e) {
        // Record the error log
        sleep(2); // Wait and try again.
    }
} while(true).

How to choose the type of agent? Look at this comparison table

ipipgo's high stash of agents tested the effect is outstanding, like an e-commerce platform such as anti-climbing perverted site, with their agents can run more than 8 hours of stability without dropping the line.

QA Time: Common Pitfalls for Newbies

Q: What should I do if my proxy IP is not working?
A: This situation is eighty percent of the use of junk proxy. Choose ipipgo this kind of professional service provider, their IP survival rate is guaranteed, but also with automatic switching function.

Q: What should I do if the crawl is slowed down?
A: Check the geographic location of the proxy server, choose a node close to the target site. ipipgo has 30+ country nodes to choose from, Hong Kong, Singapore, these Asian nodes speed fly up.

Q: HTTPS site crawl failure?
A: Add these two sentences to the cURL settings:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false).

One last thing: Proxy IPs are worth every penny. Free proxies are beautiful to look at, but can make you cry when you use them. Like ipipgo this paid service, stability is much more reliable, especially to do serious projects, do not save this silver.