PHP Web Crawler: Proxy IP settings to capture data easy tutorials

Why do I need a proxy IP for crawling?

Brothers engaged in data capture understand that the target site is not a fool. To give a chestnut, you use your own broadband connected to the crazy grabbing data, people's servers immediately to your IP booklet. Light is to limit the flow of speed, heavy is permanent black. This time the proxy IP is likecloak of invisibility, changing vests with each request to make the site think it's being accessed by a different user.

Especially for e-commerce price comparison, public opinion monitoring these need to capture high-frequency work, without proxy IP is no different from running naked. Last year, there is a friend doing airfare comparison, the home server IP was blocked to mom do not recognize, and then on the dynamic residential proxy to solve the problem.

Three poses for setting up a proxy in PHP

Here's some real code for the guys, using ipipgo's proxy service as a chestnut. Look at the comments section!

// Method 1: CURL direct connection (good for single requests)
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://目标网站.com");
curl_setopt($ch, CURLOPT_PROXY, "proxy IP:port"); //e.g. 1.2.3.4:8080
curl_setopt($ch, CURLOPT_PROXYUSERPWD, "account:password"); //authentication information provided by ipipgo
$result = curl_exec($ch);

// Method 2: Global proxy (good for batch operations)
stream_context_set_default([
    'http' => [
        'proxy' => 'tcp://proxy IP:port',
        'request_fulluri' => true,
        'header' => "Proxy-Authorization: Basic " . base64_encode("Account:Password")
    ]
]);
file_get_contents("https://目标网站.com").

// Method 3: Guzzle Advanced Play (recommended)
$client = new GuzzleHttpClient([
    'proxy' => 'http://账号:密码@ProxyIP:Port',
    'timeout' => 30
]);
$response = $client->get('https://目标网站.com');

What are the doors to look for when choosing a proxy IP?

There are many proxy service providers on the market, but there are also many potholes. Remember these three indicators:

Reliable service providers like ipipgo are recommended to be prioritized. His family has 200+ countries of local operators resources, but also supports Socks5 protocol. The most critical thing is that the API is easy to extract, and you can get the proxy address and dislike it directly in the code.

Common rollover scene for newbies QA

Q: What should I do if I can't connect to the proxy IP all the time?
A: First check the whitelist settings, many service providers to bind the local IP. if you use ipipgo, remember to add the server public IP in the background.

Q: Crawl suddenly slowed down?
A: It may be that the target website limits the number of concurrency. Suggest to add a random delay in the code, such as usleep(rand(500000,2000000)).

Q: How do I verify if the agent is in effect?
A: Add an echo $_SERVER['REMOTE_ADDR']; in the code to see if the proxy IP address is returned.

Agent Service Provider Hardcore Recommendations

Not to be an old king, ipipgo does have a couple of tricks up his sleeve. His dynamic residential agent supportsvolumetric billingThe package is especially friendly to small projects that are just starting out. If you do cross-border e-commerce business that requires a fixed IP, directly on the static residential package, 35 dollars an IP can be used for a month.

Focusing on their TK line, this is particularly useful for brothers doing business overseas. Previously, a friend of the independent station, with ordinary agents always be wind control, changed the TK line directly doubled the success rate. However, the specific program is recommended to find customer service 1v1 customized, do not blindly toss.

A final reminder:Don't waste your time with free agents! 9 out of 10 of those public proxy lists are phishing. A regular service provider costs money, but it saves the time cost of troubleshooting the problem, so it's a good deal any way you look at it.