The case of numerical datasets: an example of numerical data proxy collection

Numerical collection of cartwheels: the embarrassing scene without proxy IPs

Last week, an e-commerce monitoring small brother to find me trolling, he climbed the competitor price data, just grabbed 300 on the blocked IP. the most hilarious thing is that the unlucky child changed three times in a row broadband dial-up, the results of other people's websites directly to his pop-up authentication code pop-up to doubt his life. This is a typicalNaked Runner Acquisition--like going to play hide-and-seek in a fluorescent green jacket and getting caught in minutes.

Anti-Blocking Triple Axe for Proxy IP

That's when it's time to pull out ipipgo's proxy IP, which is the equivalent of giving you the wholeDigital Masked Ball. How exactly does it play out? Look at these three key points:

 Python example (remember to replace your_api_key with the real key)
import requests

proxies = {
    'http': 'http://user:pass@gateway.ipipgo.com:9020',
    'https': 'http://user:pass@gateway.ipipgo.com:9020'
}

response = requests.get('Target site', proxies=proxies, timeout=10)

Notice in the code the9020 portThis is a dedicated channel for ipipgo dynamic homes. More reliable than some platforms randomly open a port 8080, after all, people go is a serious operator line.

A practical guide to avoiding the pit

Here are a few details that are easy to plant:

Data Collection Team Private Configuration

I'll show you our studio.Gold Parameter Configuration::

 Sample configuration in the Scrapy framework
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
    'ipipgo_proxy.middlewares.RotateProxyMiddleware': 100,
}

IPIPGO_API = "https://api.ipipgo.com/v1/getproxy"
POOL_SIZE = 50 Keep 50 available IPs at the same time
ERROR_LIMIT = 3 Immediate replacement of the same IP with 3 errors

This configuration works with ipipgo's API to collect a steady 20-30,000 pieces of data per hour. The point is to setfaulty melting mechanismIf you find an abnormal IP address, immediately cut the backup channel.

White common rollover QA

Q: Why do I still get blocked after using a proxy?
A: Check whether the browser plug-ins are open, some plug-ins will leak the real IP. recommended to use a pure virtual machine environment

Q: How do I choose between the two packages for Dynamic Residential?
A: the standard version of $ 7.67 / GB suitable for small and medium-sized projects, enterprise version of $ 9.47 / GB with exclusive API channel, more stable concurrently with a large amount of

Q: What should I do if my IP breaks in the middle of acquisition?
A: Add an automatic retry mechanism in the code, refer to Scrapy's retry middleware settings above, ipipgo's API returns a new IP as long as 0.5 seconds!

Some solid selection advice

If you're mainly looking for numerical data (such as price, inventory, etc.), go straight to ipipgo'sStatic Home PackageThe first thing you need to do is to get your hands dirty. Although 35 dollars / IP look expensive, but the measured success rate of 12 hours of continuous collection to 98%. than those cheap but always disconnected pheasant IP cost-effective, after all, the cost of time is also money ah.

As a final reminder, many websites now detectMouse movement track, it's not enough to just change the IP to do a behavioral simulation. But that's a topic for another day, so yell if you want to hear about it in the comments section, and we'll talk about it next time.