PHP get curl: PHP use curl

Hands-on teaching you to use PHP CURL play proxy IP

Brothers do data collection, is not often encountered in the target site blocked IP situation? Don't panic! Today we use PHP CURL with proxy IP to break the game. First on the hard goods, and then talk about the principle, to ensure that you get started in three minutes.

Why use a proxy IP for data?

As a chestnut, it's like playing a game with a small number. The target site to see different IP in the visit, thought they are real users, will not trigger the anti-climbing mechanism. Especially with ipipgo this kind of professional service providers to provide theHigh Stash Proxy IPThe collection efficiency is directly doubled by completely hiding the real address.

// Basic CURL setup proxy
$ch = curl_init();
curl_setopt($ch, CURLOPT_PROXY, 'Proxy IP:Port');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'username:password'); curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'Username:password');
// Remember to change to the real information provided by ipipgo

Three major pitfalls in the real world

Newbies often fall head over heels in these areas:

Full code example (with exception handling)

function crawlWithProxy($url) {
    $ch = curl_init();
    // Look here for the highlights ↓↓↓
    $proxy = 'ipipgo.com:30001'; // use ipipgo's access address here
    $auth = 'vip account:exclusive password'; // Get it in the ipipgo backend

    curl_setopt_array($ch, [
        CURLOPT_URL => $url,
        CURLOPT_PROXY => $proxy, [
        CURLOPT_PROXYUSERPWD => $auth,
        CURLOPT_TIMEOUT => 15,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_SSL_VERIFYPEER => false // skip SSL authentication
    ]);

    $result = curl_exec($ch);
    if(curl_errno($ch)){
        echo 'Crawl failed: '.curl_error($ch);
        // It is recommended to call the ipipgo API to change the IP here.
    }
    curl_close($ch); }
    return $result.
}

The right way to open a proxy IP

Don't be a fool and use a single IP to tough it out.ipipgo supportDynamic Rotation Agent PoolWe can play it like this:

// Randomize the IPs in the proxy pool
$proxyList = ipipgo::getProxyPool(); // pretend to call the SDK
$randomProxy = $proxyList[array_rand($proxyList)];
curl_setopt($ch, CURLOPT_PROXY, $randomProxy);

Frequently Asked Questions QA

Q: What should I do if my proxy IP always fails to connect?
A: First, check whether the network can ping through the proxy server, and then confirm whether the account has expired. ipipgo's technical support responds very quickly, so it is more accurate to look for them directly.

Q: What should I do if the collection speed is slow?
A: two directions to optimize: ① change to ipipgo exclusive high-speed line ② adjust the CURL timeout parameters, it is recommended that the CURLOPT_TIMEOUT is set between 10-20 seconds.

Q: How do you simulate the operation of a real person?
A: Use with these parameters for better results:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 Common Browser UA');
curl_setopt($ch, CURLOPT_REFERERER, 'https://正常来路网址');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'Saved cookie.txt'); curl_setopt($ch, CURLOPT_COOKIEFILE, 'Saved cookie.txt');

Say something from the heart.

There are two things you fear the most with proxy IPs: unstable IPs and not enough traffic. The actual test ipipgoEnterprise PackageIndeed reliable, millions of IP pool + unlimited traffic, do long-term projects, it is recommended to directly on the monthly package, much more cost-effective than pay by volume. The key is that they have technicians to help debugging, this point is especially friendly to newbies.