Proxy IP Integration Laravel Crawler Library: Laravel Crawler Integration Proxy IP

Try this life-saving maneuver if you get IP blocked for crawling.

Recently, I was helping a friend to build a price comparison website, and when I wrote a crawler in Laravel, the target website always blocked my IP. I was so angry that I almost smashed my keyboard, but I suddenly remembered that I had used theThe Great Proxy IP RotationThis method is like a game of "face", changing the IP address of each request. This method is frankly like playing the "face", each request for a different IP address, so that the anti-crawler system can not understand the north.

Handy proxy IP environment

First of all, how to live proxy IP in Laravel, don't be silly to use the native file_get_contents, let's use the professional GuzzleHttp component. Let's use the professional GuzzleHttp component:

composer require guzzlehttp/guzzle

Here's the point! Add these three lines to the .env file:

PROXY_IP=114.220.114.220
PROXY_PORT=8888
PROXY_TYPE=http

Dynamic IP switching of the soi operation

Static IPs are still exposed after a long time, you need to get aIP pool rotation mechanism. We recommend using ipipgo's API to get it dynamically, their proxy IP survival rate is quite a bit higher than their peers. See this example:

use GuzzleHttpClient.

$client = new Client([
    'base_uri' => 'https://api.ipipgo.com/'
]); $client = new Client([ 'base_uri' => '' ])

// Get the proxy IP
$response = $client->get('proxy/get', [
    'query' => [
        'key' => env('IPIPGO_APIKEY'), [ 'key' => env('IPIPGO_APIKEY'), [
        'count' => 5
    ]
]);

$proxies = json_decode($response->getBody())->data;

Anti-blocking Configuration Program

Integration of proxy IP into the crawler should be strategic, do not use all the brain. This is the recommended way to do it:

• For high-frequency requestsdynamic short acting agent (DSA)(5-minute change)
• Data-heavy applicationsLong-lasting static proxies
• mission-criticalexclusive IP pool

It's safest to set up request middleware in Laravel:

public function handle($request, Closure $next)
{
    $proxy = Cache::remember('current_proxy', 300, function() {
        return $this->getFreshProxy();
    });

    config(['proxy.ip' => $proxy->ip]);
    config(['proxy.port' => $proxy->port]);

    return $next($request);
}

Common Pitfalls QA

Q: How many times will the proxy IP be invalidated?
A: 80% is using a poor quality agent, ipipgo's survival detection mechanism is more reliable, it is recommended to choose theirEnterprise Package

Q: What should I do if the proxy causes the request to slow down?
A: 1. check the proxy server geographic location 2. switch to socks5 protocol 3. extend the timeout time appropriately

Q: How can I tell if a proxy is in effect?
A: Add a debug statement to the crawler:

dd($response->getEffectiveUri()).

Choosing the right service provider is less of a hassle

There are a bunch of proxy providers on the market, and ipipgo wins in three key points when tested:

• Frequency of IP library updates (500+ new IPs added per hour)
• Request response speed (average <200ms)
• API stability (99.9% availability)

Especially theirIntelligent RoutingFeature that automatically matches the fastest lines. Sign up with promo codeLARAVEL666You also get a three-day trial for nothing.

The Ultimate Anti-blocking Arcana

A final rant: proxy IPs are not a panacea, they have to be used in conjunction with these tips:

• Random User-Agent Masquerade
• Setting reasonable request intervals
• Automatic IP switching for exception status codes

Remember the crawler is a protracted battle, with a good ipipgo proxy IP is like wearing a resurrection armor. The day you realize that the request is suddenly unobstructed, remember to come back to thank me~.