Web crawling Python script: Python collection code

What's wrong with this thing? Proxy IP to solve crawler headaches

Old iron do data collection must have encountered this situation: just climbed two sites on your IP blocked, angry straight to the keyboard. At this time, we have to invite theproxy IPThis Buddha, in particular, is the one we're going to talk about.ipipgoThe service of the home is as reliable as a northeastern kangaroo.

How did proxy IPs become essential for crawlers?

To give a chestnut, the webmaster is like a supermarket security guard, see the same face (IP) all day in front of the shelves around, not block you block who? Using a proxy IP is equivalent to changing to a different armor every day to go shopping, pay attention to the choice ofHigh Stash Agents, just like the ipipgo family that can even disguise the browser fingerprints, and definitely make the site look fishy.

import requests

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

resp = requests.get('https://目标网站.com', proxies=proxies, timeout=10)
print(resp.text)

How do you pick a proxy IP without stepping into a hole?

The market is full of agency service providers like a vegetable market radish, but we have to pick the best. Focus on these three indicators:

Special reminder: don't use free proxy for cheap, those IPs have been fucked up, slow as a snail, not to mention, but also may contain private goods to steal your data.

A practical guide to avoiding the pit

1. Each requestRandomized switching of agents，别可着一个IP
2. 429 status code encountered firstWe'll fight again when we get some sleep.Recommended Index Exit Strategy
3. Critical data collectionBe sure to buy the paid packageI'm a newcomer to ipipgo and I'm getting 5G of traffic for nothing.
4. Remember to set the timeout parameter, do not let the slow agent drag down the whole program

QA time

Q: What should I do if the proxy IP suddenly fails to connect?
A: Normal phenomenon, ipipgo background will automatically exclude failed nodes, it is recommended to add a retry mechanism in the code, three times can not be connected to change the IP!

Q: Are there any special precautions for collecting overseas websites?
A：选对应地区的代理节点，ipipgo支持30+国家地区定位，日本节点最低能到80ms

Q: How can I prevent being counter-crawled by a website?
A: Proxy IP + random UA + request frequency control three-piece suite, ipipgo'sDynamic Residential AgentsBest for hard rigid anti-climbing systems

Finally, to do crawlers like guerrilla warfare, have to flexibly use the proxy IP this smoke bomb. ipipgo recent activities, the enterprise version of the package to send the IP survival monitoring function, equivalent to each agent installed heartbeat detector, who cool immediately automatic replacement, the degree of worry is comparable to the autopilot.