http secondary proxy: secondary HTTP proxy chaining configuration

What the hell does this thing do?

Friends engaged in network crawlers should have encountered the IP was blocked the bad things, this time the secondary proxy is like a spare tire car key. Let's say you use the first proxy to access the site, was found abnormal, the second proxy immediately take over the top. To put it bluntlydual insurance, much more stable than a single-tier agent.

Handy Chain

Let's take python crawler as an example and use ipipgo's proxy service as a demonstration. First register an account on the official website to get the API extraction link, for example:

import requests

 Layer 1 proxy
proxy1 = "http://用户名:密码@gateway.ipipgo.com:端口"
 Layer 2 proxy
proxy2 = "http://用户名:密码@gateway.ipipgo.com:端口"

proxies = {
    "http": f"{proxy2} via {proxy1}",
    "https": f"{proxy2} via {proxy1}"
}

response = requests.get("destination URL", proxies=proxies)

Note that you have to replace the proxy address after via with the one you actually applied for. ipipgo's API supports extracting multiple nodes at the same time, which is particularly suitable for this kind of chained configuration.

Configuration Pitfalls

When I was doing price monitoring for an e-commerce client last year, I encountered three typical problems:

Older Driver's Guide to Choosing a Package

There's something to be said for ipipgo's package choices:

• Dynamic residential (standard): Good for data collection newbies, $7+ 1G lasts for ages!
• Dynamic Residential (Business): With customized routing function, essential for cross-border business
• Static homes: Fixed IP is suitable for scenarios that require logging in, although more expensive, but stable

Practical QA

Q: How much more expensive is a second tier agent than a single tier?
A: Actually, the cost is about the same, because the traffic is only counted as part of the actual usage, and the chain is just the forwarding path.

Q: Does the mobile app work with this configuration?
A: ipipgo's client supports chained proxies, just fill in two proxy addresses in the advanced settings.

Q: How do I break the CAPTCHA when I encounter it?
A: It is recommended to add a TK dedicated node in the agent chain, and the rate of CAPTCHA can be reduced to 60%.

Why do you recommend ipipgo?

His family has three killers:
1. Local IP resource pools in 200+ countries around the world, many choices for chain allocation
2. Support the simultaneous extraction of different protocol proxies, no pressure to build a mixed chain
3. There are specialized chain configuration documents, encounter problems at any time to find technical customer service

Recently to help customers do overseas price comparison system, with his dynamic enterprise version of the package to build a three-layer proxy chain, running for half a month without being blocked. If you want me to say, you can really lose a lot of hair by choosing the right service provider.