PHP Crawler: CURL Data Collection Scripts

I. Why are crawlers always blocked? Try this method

Anyone who is involved in data collection knows that the biggest headache of writing crawlers with PHP is theIP blockedThe first day I helped a friend to do a price comparison tool. A few days ago to help a friend to do a price comparison tool, just run half an hour to receive 403 forbidden, angry him straight to shoot the keyboard. This time we have to sacrifice the big killer - proxy IP. this thing is like a crawler wearing a myriad of masks, each request for a new face, the site simply can not distinguish between a person is a machine.

Second, hand teach you to install CURL extension

First, make sure that your server has the CURL extension installed (if you don't have it installed, face the wall). Open your php.ini file and find this line:

;extension=curl

Remove the leading semicolon. restart Apache/Nginx. write a test script:

if(function_exists('curl_version')){
    echo 'CURL has been enabled'; } else {
} else {
    echo 'Hurry up and install the extension! ;
}

Proxy IP access code

Here's the point! Use ipipgo's proxy service, their house offersDynamic Residential Agents, measured stability is good. Look at this core code:

$proxy = 'gateway.ipipgo.com:9021'; //proxy server address
$auth = 'username:password'; //get in ipipgo backend

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'destination URL');
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth); curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30); curl_setopt($ch, CURLOPT_TIMEOUT, 30);

$result = curl_exec($ch);
if(curl_errno($ch)){
    echo 'Error: '.curl_error($ch); }
}
curl_close($ch); }

Fourth, avoiding these pits can save two hours

Three common mistakes newbies make:

V. The secret of doubling collection efficiency

Single-threaded crawler too slow? Go multithreaded! Use PHP's curl_multi family of functions with ipipgo'sMulti-Channel AgentThe speed takes off straight away. Remember the settings:

curl_setopt($ch, CURLOPT_FORBID_REUSE, 1); //disable connection multiplexing
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1); //force new connections

VI. Frequently Asked Questions QA

Q: What should I do if my proxy IP suddenly fails?
A: Enable the automatic IP replacement function in the ipipgo background and set it to rotate every 5 minutes

Q: What should I do if I want to capture HTTPS websites?
A: Add it in the curl configuration:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Q: How to optimize the slow agent speed?
A: Prioritize ipipgo'sDomestic BGP line，能控制在200ms以内

VII. Say something heartfelt

After so many years of crawling, the proxy IP thing is really just needed. Maintaining your own IP pool is too much work, so you might as well use a ready-made service. Like ipipgo, which supportspay per volumes, especially friendly to small projects. Finally, I would like to remind you: do collect to comply with the website robots agreement, don't crash the human server!