python crawler set ip proxy: code examples and anti-blocking strategy

Hands-on teaching you to use Python to hang proxies to crawl data

Brothers who engage in crawlers understand that it is more common to be blocked IP than to be blackmailed by your girlfriend. Today we will take our own products ipipgo example, teach you how to use proxy IP to save the dog's life. First of all, to tell the truth, the market 90% proxy service providers to the IP quality are like a joke, but our dynamic residential proxy pool of 90 million + real family IP, specializing in anti-climbing mechanism.

 Requests library setup proxy (dynamic residential version)
import requests

proxy = "http://用户名:密码@gateway.ipipgo.com:端口"
proxies = {
    'http': proxy,
    'https': proxy
}

 Remember to keep the session
with requests.Session() as s.
    s.proxies = proxies
    resp = s.get('https://目标网站.com')
    print(resp.text)

Anti-blocking must-kill triple move

Tip #1: IP rotationipipgo's dynamic proxy supports automatic switching, it is recommended to change the IP every 5-10 requests. don't worry about the traffic, we are billed according to the amount of money is much more economical than being blocked.

Tip #2: Camouflage should be in placeUser-Agent don't always use the default, here's an off-the-shelf rotation scheme for you:

user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64) AppleWebKit/537.36..." ,
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..." ,
     Prepare at least 20 different browser versions
]

Tip #3: Pace yourself like a human being. Don't send requests like a jerk, set a random delay of 2-8 seconds. Using time.sleep is too low, try this advanced play:

from random import randint
import time

def human_delay(): time.sleep(randint(3,7) + randint(0,1000)/1000)
    time.sleep(randint(3,7) + randint(0,1000)/1000)

How to choose between dynamic/static proxies?

A practical guide to avoiding the pit

Recently I helped a client to catch an e-commerce platform, and it ran for 72 hours straight without flipping with dynamic agents. The key setting:

• Maximum 15 minutes per IP
• Random jitter in request intervals (don't use fixed values)
• Mixed use of HTTP/SOCKS5 protocols

Don't panic when it comes to CAPTCHA, the smart routing technology in ipipgo's TikTok solution has been tested to work for e-commerce platforms as well. The point is to let the traffic go through the local operator's line, don't do all those fancy cross-country jumps.

Frequently Asked Questions QA

Q: What should I do if the proxy suddenly fails?
A: First check the account authorization, then use the API provided by ipipgo to get the latest proxy list. Dynamic proxies are updated in 30 minutes by default, and it is recommended to actively refresh them for important tasks.

Q: Overseas website latency is too high?
A: Go on the cross-border dedicated line, don't use ordinary proxy hard. The delay of our dedicated line can be reduced to 2ms, which is the same as local access.

Q: Do I need to capture pages rendered by JavaScript?
A: Use the SERP API to take structured data directly , than to write their own crawler to save time. Support 100+ requests per second, also with automatic parsing

Lastly, don't believe in those free agents. Last year, a customer had to use a free IP, the results of the target site reverse traceability, directly received a lawyer's letter. Now with ipipgo static proxy to do competitive analysis, more than half a year without a moth. This is a matter of data collection, stability is much more important than cheap.