PHP web page capture | CURL real collection case tutorials

Teach you to use PHP to grab data, proxy IP so use to stable!

Engaged in data collection of the old iron should understand, directly with their own server IP to woolgathering site, minutes to be blocked. Last week there was a friend doing e-commerce, he wrote his own crawler script suddenly failed, a check of the logs only to find that the IP was the target site pulled the black. This time we have to move out of our savior - proxy IP.

This is a must.ipipgoThe proxy service of the family, their IP pool is large enough, each request can be changed to a different export IP. I have tested myself, continuous collection of an e-commerce platform for 3 hours without being intercepted, the success rate remains at 95% or more.

PHP Crawl Triple Axe

First on the hard course, directly see how the code plays. When initializing a request with CURL, focus on these two parameter settings:

$ch = curl_init();
curl_setopt($ch, CURLOPT_PROXY, 'proxy IP:port'); // fill in the address provided by ipipgo here
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'Account:Password'); //authentication information generated by ipipgo backend

Many newbies fall prey to timeout settings, suggesting thatConnection timeout set to 8 seconds, transmission timeout set to 25 seconds. When you come across a site that is slow to respond, this setup is effective in avoiding script jams.

Proxy IP practical guide to avoid pitfalls

Here are a few blood lessons:

Special note: When using ipipgo's proxy, remember to put theIP Survival TimeSet it to dynamic mode so that it will automatically change IP for each request, and it has been tested to have the best anti-blocking effect.

Capture Script Optimization Tips

1. Random delay is important, don't use fixed SLEEP time. It is recommended to stop randomly between 1-3 seconds, so that it is more like a real person operation.

2. Don't panic when encountering CAPTCHA, ipipgo's exclusive IP package supports automatic coding service, which can save a lot of work.

3. When storing the results in the database, remember to do the following.Deduplicate Data Filtering. It is recommended to compare the content with MD5 hashes, this method is the most efficient.

Frequently Asked Questions QA

Q: What should I do if my proxy IP is slow?
A: Choose ipipgo's BGP line, they have triple-play backbone nodes, measured latency can be pressed to within 200ms.

Q：How can I continue harvesting if I interrupt in the middle of harvesting?
A: Add a breakpoint function in the script to record the position of the last acquisition. ipipgo's API supports querying the usage record by task ID, which is convenient to retrieve the previous acquisition progress.

Q: How do I get it if I need multi-threaded acquisition?
A: Use pcntl_fork to create sub-processes, each process is assigned a different ipipgo proxy IP. be careful to control the number of concurrency, don't run the server CPU fried.

Finally give a piece of advice: don't be greedy and cheap with free agents, light data is wrong, heavy account is blocked. Like ipipgo regular service providers, although it costs a little silver, but the data quality is guaranteed, there are problems with technical customer service support at any time, this is the right way to engage in the collection.