PHP curl GET Request: Data Collection Code Example

Teach you to use PHP to grab data without blocking IP!

Recently a number of friends doing e-commerce asked me to use PHP to write a crawler always be the target site to block the IP how to do? This matter is in fact with the game to open a small number of a reason.Keep a few more proxy IPs in rotationIt is the right solution. Let's take php curl today to give a chestnut, teach you how to use ipipgo's proxy service to grab data properly.

Basic GET request code

Start with the entire simplest curl request template and add to it later:

$url = 'https://目标网站.com/api';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch); echo $response; echo $response; curl_exec($ch)
echo $response;

This code runs up with the naked like, the target site can catch your real IP in a minute. like to go to the market price cutting without wearing a mask, the stall owner remembered you next time the price directly.

Putting a proxy vest on the request

Here's the kicker! We have to stuff proxy configuration into the code. Here's an example using ipipgo's proxy service, which provides a handy API format for thieves:

$proxy = 'http://用户名:密码@gateway.ipipgo.com:端口';
// A real life chestnut ↓
// $proxy = 'http://vipuser:abcd1234@proxy.ipipgo.com:31028'; // $proxy = 'http://vipuser:abcd1234@proxy.ipipgo.com:31028'; // $proxy = 'http://vipuser:abcd1234@proxy.ipipgo.com:31028'

curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);

Watch out for potholes:

• Remember to urlencode passwords with special symbols.
• The proxy type should match the one given by the service provider
• Set a longer timeout, more than 15 seconds is recommended.

Complete code platter

Save all of what was said before for a piece of code that can be used in practice throughout:

function crawlWithProxy($targetUrl){
    $ch = curl_init();

    // Proxy configuration
    $proxy = 'http://vipuser:abcd1234@proxy.ipipgo.com:31028';
    curl_setopt($ch, CURLOPT_PROXY, $proxy);
    curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP); curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);

    // Request configuration
    curl_setopt($ch, CURLOPT_URL, $targetUrl); // Request configuration.
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // Request Configuration.
    curl_setopt($ch, CURLOPT_TIMEOUT, 20); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // skip https authentication

    $result = curl_exec($ch);
    if(curl_errno($ch)){
        echo 'Error: '.curl_error($ch); }
    }
    curl_close($ch); return $result; }
    return $result; }
}

// Real-world calls
$data = crawlWithProxy('https://目标网站.com/products');

A common rollover scene for older drivers

Why do you recommend ipipgo?

I dare to recommend it only if I have used it in my own home, and these points do top:

1. Exclusive IP pool without crashing
2. Response speed control within 200ms
3. There is professional technical customer service to put out the fire
4. Supporting pay-as-you-go without waste

Real-world QA triple play

Q: How many times will the proxy IP be invalidated?
A: It is recommended to change IP for each request. ipipgo's API supports automatic switching, just add an IP pool polling in the code.

Q: Collecting at a snail's pace?
A: eighty percent of the proxy server is not powerful, replaced with ipipgo's enterprise nodes, the speed can be more than tripled.

Q: How can I prevent anti-climbing?
A: In addition to changing IP, remember to add User-Agent in the request header, and use ipipgo's high stash of proxies for more insurance.

As a final note, data collection is a matter offast, accurate and stableThe first thing you need to do is to choose the right proxy service provider to save you a lot of trouble. Choose the right proxy service provider can save a large part of the heart, ipipgo our team tested, million requests to carry, new users can also whoring test quota, specific to the official website to take a look at it.