Laravel proxy IP crawler source code: Laravel proxy crawler source code analysis

Teach you to use Laravel to get proxy IP crawler!

Recently a lot of old iron asked how to use Laravel whole proxy IP crawler, today we will take the guys disk this matter. Let's focus on how to use ipipgo's proxy IP to solve the problem of the crawler was ban, definitely more than those online tutorials are really the same.

Why do you have to use a proxy IP for crawling?

For example, if you take your own IP to climb the website, it is like wearing a fluorescent suit to engage in lurking - exposed in minutes. ipipgo's proxy IP is equivalent to give you a cloak, every time you request to change a vest, the website simply can not recognize who you are.

// Basic crawler configuration (without proxy)
$html = file_get_contents('Target site');
// Getting banned in minutes!

Laravel crawler basic construction

First of all, a simple crawler framework, here with GuzzleHTTP demonstration:

use GuzzleHttpClient.

$client = new Client([['timeout' => 10, 'timeout' => 10, 'timeout' => 10)
    'timeout' => 10, 'headers' => [
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
    ]
]).

// Original request method
$response = $client->get('Target URL');

Integration of ipipgo proxy service

Here comes the point! Use ipipgo's API to extract proxy IP, remember to get it from the official websiteAPI key::

// Interface to get the proxy IP
$proxyApi = 'https://api.ipipgo.com/proxy?apikey=你的密钥';

// Get the list of proxy IPs
$proxyList = json_decode(file_get_contents($proxyApi), true);

// Pick a random proxy
$randomProxy = $proxyList['data'][array_rand($proxyList['data'])];

// Request configuration with proxy
$client = new Client([['proxy' => ['data']]); // configure request with proxy
    'proxy' => [
        'http' => "http://{$randomProxy['ip']}:{$randomProxy['port']}",
        'https' => "http://{$randomProxy['ip']}:{$randomProxy['port']}"
    ],
    'auth' => [$randomProxy['username'], $randomProxy['password']]
]);

Real-world anti-ban techniques

It's not enough to use agents, you have to be strategic:

Frequently Asked Questions QA

Q: What should I do if my proxy IP suddenly fails?
A: It is recommended to turn on ipipgo's automatic IP change function, or switch to an enterprise-level dynamic residential package ($9.47/GB)

Q: Can't get the crawler speed up?
A: Try these tricks:
1. Parallel requests with multithreading + proxy pools
2. Selection of low-latency data center agents
3. Contact ipipgo technology for 1v1 customized solutions

Q: How do I choose a package with a limited budget?
A: Priority is given to Dynamic Residential Standard Edition ($7.67/GB), traffic billing does not hurt. Need fixed IP then upgrade static version.

Say something from the heart.

Engage in crawling this thing, proxy IP quality directly determine the success or failure. I have used seven or eight service providers, ipipgo'sTK LineIs really resistant to build, specializing in dealing with difficult sites. Their client is also a thief easy to use, white people can get started in three minutes.

Finally, remind everyone, although the crawler is good, do not be greedy Oh! Reasonable control of the frequency of requests, with high-quality proxy IP, in order to get the data. What do not understand at any time to go to ipipgo official website to find customer service nagging, technical small brother thief enthusiasm.