Pythonrequests User Agent: Anti-Detection UA Settings

This is how you should play when a crawler encounters UA detection

A lot of partners with Python requests library have encountered this situation: obviously the code is written smoothly, but the target site is like the eye of heaven, always recognize that you are a crawler. Don't panic.User Agent (UA) Masquerade + Proxy IP DuoThat's the right solution.

UA setup is not as simple as changing your vest

Many people think that a random browser UA is the end of the story, but in fact there are many doors. The website will determine the authenticity of the UA by several features, such as:

 Typical error example (don't write it like this)
headers = {'User-Agent': 'Random UA'}

Keep these three things in mind for proper posture:
1. Match the device model with the browser version (don't use Chrome 120 for the whole iPhone)
2. The language and time zone should correspond to each other (US IP with zh-CN language will be exposed)
3. Frequency of requests should be simulated for real people (sudden surge of requests will be blocked)

Proxy IP Selection with Care

It's not enough to change the UA, you have to use it with a proxy IP. There are three types of proxies on the market:

I have to brag about my own product here.ipipgo's high stash proxy poolThe actual test can carry a certain e-commerce platform for 7 consecutive days of testing, the specific configuration of the code to see:

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:9020',
    'https': 'https://用户名:密码@gateway.ipipgo.com:9020'
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0'
}

response = requests.get('https://目标网站', headers=headers, proxies=proxies)

A practical guide to avoiding the pit

Recently, while helping a client with a travel website acquisition, I found three key points:

1. 3-5 am UA switching interval should be greater than 120 seconds (don't ask how you know, it's all tears)
2. Android device UA with a mobile network identifier (e.g. the string of characters in front of MobiLeo)
3. Use of ipipgoDynamic Port AllocationFunctions with higher survival rate than fixed ports 40%

Frequently Asked Questions QA

Q: I set both UA and proxy and still get blocked?
A: Check the purity of the IP, recommend using ipipgo's enterprise-level proxy package, their IP has a real person authentication mechanism

Q: Do I need to change UA for every request?
A: look at the site wind control level, ordinary site 2-3 hours to change once on the line, the key is to keep the equipment characteristics consistent

Q: How does ipipgo's proxy ensure stability?
A：They use cellular network dynamic IP pool, each request is a new export IP, pro-test continuous collection of 8 hours without a break

Upgrade Play Recommendations

For the ashes players, tryUA Fingerprint Confusion: Split common UA into component libraries and generate new UA by random combination every time. with ipipgo's IP rotation strategy, the success rate is directly pulled full.

 Example of component-based UA generation
device = ['Windows NT 10.0', 'Macintosh; Intel Mac OS X 10_15_7']
browser = ['Chrome/120.0.0.0', 'Safari/537.36']
ua = f "Mozilla/5.0 ({random.choice(device)}) AppleWebKit/537.36 ({random.choice(browser)})"

As a final, heartfelt remark, the essence of anti-detection is tohuman-computer confrontationIt's important to keep your technology up to date. It's important to keep technology up to date, like ipipgo's tech team updates their library of anti-anti-crawl strategies every week, which saves you a lot of heartache compared to tossing around on your own.