Python web crawler: Python harvesting solution

Python crawler can not handle the IP blocking? hand to teach you to use proxy IP to break the game

Crawler brothers understand that the biggest headache is the target site suddenly give you an IP ban. Yesterday also ran a good script, today directly out of action. At this time you have to move out of theproxy IPThis rescue, today we will take the actual combat to speak, teach you to use Python + proxy IP to create a King Kong collection program.

Why do I have to use a proxy IP?

To give a chestnut, you go to the same supermarket every day to buy a limited number of goods, the clerk on the third day will certainly recognize you. Web server is also the same reason, the same IP frequent visits, immediately triggered anti-climbing mechanism. At this time it is necessary tomany sets of vests(Proxy IP) rotation, ipipgo family dynamic IP pool can do every request automatically change IP, than manual switching much more efficient.

import requests
from itertools import cycle

 List of proxies from ipipgo
proxies = [
    "http://user:pass@103.ipipgo.com:8000",
    "http://user:pass@104.ipipgo.com:8000".
     ... More proxies
]
proxy_pool = cycle(proxies)

for _ in range(10):
    current_proxy = next(proxy_pool)
    try: current_proxy = next(proxy_pool)
        response = requests.get(
            'https://目标网站.com',
            proxies={"http": current_proxy},
            current_proxy}, proxies={"http": current_proxy}, timeout=5
        )
        print("Successful capture:", response.status_code)
    except.
        print("Current proxy failed, automatically switch to the next one")

A practical guide to avoiding the pit

Just will use the agent is not enough, these details do not pay attention to the car as usual:

Configuration tutorials that even a novice can handle

1. First go to the official website of ipipgo to register, new users to send5000 free trials
2. Generate an API link in the console and copy the proxy address in the code.
3. Plug the following function into your crawler:

def get_ipipgo_proxy().
    api_url = "https://api.ipipgo.com/获取代理的路径" Replace with your own account's
    return requests.get(api_url).text.strip()

Note that replacing user and pass with your own account's authentication information is recommended.environment variableStore sensitive information, don't be stupid and write it in code!

Frequently Asked Questions QA

Q: What should I do if the proxy IP is invalidated while I am using it?
A: This is why to choose ipipgo's dynamic residential agent, their IP survival time is optimized, with automatic replacement mechanism basically will not drop.

Q: How many agents are enough to crawl data?
A: Look at the strength of the target site's anti-climbing, generally small and medium-sized site with the10-20 high quality IPsThe rotation is adequate. ipipgo's pay-as-you-go model is pretty cost-effective, buy as much as you use.

Q: What should I do if I use a proxy and still get recognized?
A: Check these three points: 1) Is the request header with browser fingerprint 2) Is the operation interval too regular 3) Is the IP quality up to standard. It is recommended to go on ipipgo'sHigh Stash Agents, completely hide the real IP.

Finally, the proxy IP is not a panacea, with standardized crawler habits. If you dislike people's servers hundreds of requests per second, even the best proxy can't carry it. Reasonable control of frequency, coupled with ipipgo quality proxy, this is the way of sustainable collection.