PHP web crawling example: PHP collection examples

Hands-on teaching you to use PHP to capture data without blocking the number of

The biggest headache of engaging in network collection is that the IP is blocked, especially those sites with strict anti-crawler mechanism. Today we will teach you how to useproxy IPThis artifact, so that our PHP scripts can run stably without rolling over.

What's the point of a proxy IP anyway?

For example, like going to the supermarket to try to eat, if you always use the same plate to rub, the security guards will certainly be on you. Proxy IP is our trick to change the plate, each visit to a different IP address, the site will not recognize the same person in the operation.

This is a must.ipipgoThe service, their dynamic IP pool is particularly large, measured half an hour can change 500 + IP. the key is the automatic authentication function, do not have to manually change the account password, this point is too friendly to lazy programmers.

PHP real-world code to go

// Configure the proxy server
$proxy = 'gateway.ipipgo.com:9021'; // Configure the proxy server in the backend of ipipgo.
$auth = 'username:password'; // get in ipipgo backend

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'destination URL');
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth); curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true).

// Remember to add a delay to prevent blocking
usleep(rand(500000, 2000000));

$result = curl_exec($ch);
curl_close($ch).

Focused attention:Remember to go to ipipgo official website to register to get a free trial package, new users to send 1G flow enough to test. The delay in the code can not save, the site can detect the speed of continuous requests.

Top 3 Tips for Breaking the Counter-Climb

Frequently Asked Questions QA

Q: What should I do if my proxy IP suddenly fails?
A: This situation is eighty percent of the IP is the target site pulled black. Recommended to use ipipgoIntelligent switching modeThe system will automatically filter the invalid IP.

Q: What should I do if I can't get up to speed on acquisition?
A: Try the combination of multithreading + proxy IP pool. ipipgo supports 50 connections at the same time, remember to control the requests not more than 3 times per second.

Q: What should I do if I encounter a CAPTCHA pop-up window?
A: This is the time to reduce the frequency of collection with ipipgo'sLong-lasting static IPservice, such IPs are usually labeled as trusted users.

Guide to avoiding the pit

Two of the easiest things for newbies to fall head over heels for:

1. Didn't set the timeout parameter, causing the script to get stuck
2. Forgetting to deal with web page encoding and outputting garbled code

It is recommended to add it in the curl configuration:

curl_setopt($ch, CURLOPT_TIMEOUT, 15);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');

Finally said a cold knowledge: some sites will detect the mouse movement track, with PHP collection instead of browser automation is more secure. With ipipgo's residential agent, the success rate can reach more than 90%. What do not understand can go directly to their official website to find technical customer service, reply speed is quite fast.