Python Web Crawl: Python Proxy Web Crawl

Python crawlers always get blocked? Try this trick

Crawlers brothers understand that the biggest headache is the IP is blocked. Hard-written code to run a sudden break, the server returns 403, the feeling is like playing a game was forced offline. This is the time toproxy IPIt's on - it's like an invisibility cloak for crawlers, so that the target site can't recognize you for who you really are.

How do you load a proxy IP into a crawler?

Taking the most commonly used requests library as an example, just add a proxies parameter to the request and you're good to go. Note that you have to use theHigh Stash Agents, don't use those half-assed generic proxies:

import requests

proxy = {
    'http': 'http://用户名:密码@ipipgo-proxy-server:port',
    'https': 'https://用户名:密码@ipipgo-proxy-server:port'
}

response = requests.get('destination URL', proxies=proxy)

Here's a recommendation for ipipgoDynamic Residential AgentsThe IP pool is stocked with millions of real residential IPs, which are more difficult to recognize than the server room proxies. After registering, you will get the exclusive API link, and you can directly replace the proxy address above.

How do you pair agents with a multi-threaded crawler?

Single-threaded with the agent is too wasteful, you have to work with multi-threaded to take off. It is recommended to use thread pool + agent pool double pool mode, here is a simplified version of the demonstration:

from concurrent.futures import ThreadPoolExecutor
import random

def worker(url): current_proxy = random.choice(ipipgo_proxy_list)
    current_proxy = random.choice(ipipgo_proxy_list) Randomly choose from ipipgo's IP pool.
    try: current_proxy = random.choice(ipipgo_proxy_list)
        response = requests.get(url, proxies=current_proxy, timeout=10)
         Processing data...
    except.
         Automatically remove invalid proxies
        ipipgo_proxy_list.remove(current_proxy)

with ThreadPoolExecutor(max_workers=20) as executor.
    executor.map(worker, url_list)

Be careful to set a reasonable timeout, between 3-10 seconds is recommended. ipipgo's proxies areautomatic fusing mechanismIf you encounter an invalid IP, it will be switched automatically, so you don't have to deal with it manually.

How do I choose a proxy IP type?

Personal experience isResidential Agentsof the most cost-effective. Residential proxy packages like ipipgo's have 100,000+ IPs rotating every day, which is more than enough for small to medium sized projects. If you are engaged in large-scale data collection, it is recommended to choose theirEnterprise Customized Edition, supports pay-per-use.

A practical guide to avoiding the pit

1. Don't use free agents.--Slow, not to mention that many are honeypot traps that specialize in catching crawlers

2. Randomly change the UA before each request, so as not to let the User-Agent reveal itself.

3. Control the frequency of visits, it is recommended that the target site of theaccess intervaladd a random number to

4. Regularly check the availability of proxies, recommended ipipgo comes with theHealth Check API

Frequently Asked Questions QA

Q: Proxy IP becomes slower when I use it?
A: It may be that the IP is speed limited, submit a work order in the ipipgo background, the technical guy will change the new line within 5 minutes!

Q: What if the crawler needs to handle CAPTCHA?
A: ipipgo'sIntelligent Routing AgentSupports automatic CAPTCHA recognition, but you have to pay extra for the premium package.

Q: How can I tell if a proxy is in effect?
A: Visit http://httpbin.org/ip to see if the IP returned is a proxy IP

Why ipipgo?

1. Actual measurements99.2%availability with a packet loss rate of less than 0.31 TP3T
2. Exclusive IP preheating technology, new IP survival time is 3 times longer than peers
3. Supporthourly rateYou don't have to buy a monthly subscription for the temporary program.
4. 7 × 24 hours real customer service, three o'clock in the middle of the night can also find technical support

Finally, to tell the truth, the choice of proxy service provider is like looking for objects, just look at the price is easy to fall into the pit. I've used five or six service providers, and in the end it's ipipgo that's the most secure. Their IP resources are self-owned server room, unlike the second-hand dealers selling second-hand IP, with really worry.