Request Feature Randomization: a Library of User-Agent Rotation Strategies

First, why is your crawler always recognized? First look at the pit

Engaged in data collection of the old iron should have encountered this situation: obviously changed the IP address, the target site can still accurately identify the crawler behavior. At this time, many people will wonder--How could you get caught after changing your IP address? The problem is actually that your requests are characterized too regularly!

For example, if you go to the supermarket to buy something, although you change different clothes every day (proxy IP), but every time you carry the same schoolbag, walk the same route, the security guards do not stare at you to stare at who? Website protection system is throughUser-Agent, Request Frequency, Cookie Characteristicsthese details to identify anomalous traffic.

Second, the core play of User-Agent rotation

Here's a trick to teach you:Dynamic UA library + smart switching. Not simply get dozens of UA randomly selected, but according to the characteristics of the target site to match the configuration:

For example, when using ipipgo's proxy service, it is recommended that you include in the request headerRandom generation of device modelsThe function. Their API supports the automatic generation of UA that matches the current IP's locale, avoiding the embarrassing situation where a US IP hangs the UA of a Xiaomi phone.

Third, the golden combination of proxy IP and UA

It's not enough to change IPs, you have to learndouble randomization::

1. Get a new IP via ipipgo before each request
2. Automatically matches corresponding UA based on IP location
3. Randomly selected version numbers from the common UA library

Focusing on step 2, for example, if you get a residential IP in Guangdong, you should use theCommon cell phone models in Guangdongof UA. ipipgo's Smart Routing feature automatically associates geographic information, saving you a lot of work over manual maintenance.

IV. Practical guide to avoiding pitfalls (with code snippets)

Here's a Python example, note the comments section:

 Get dynamic proxy from ipipgo
def get_proxy().
    return requests.get('https://api.ipipgo.com/getProxy').json()

 Smart UA Generator
def generate_ua(ip_info):
    if ip_info['isp'] == 'mobile': return f "Mozilla/5.0 (Linux;);".
        return f "Mozilla/5.0 (Linux; Android {random.choice(['10','11'])}...)"

 Example request
proxy = get_proxy()
headers = {
    'User-Agent': generate_ua(proxy),
     Remember to add other randomization parameters
}

V. Frequently Asked Questions QA

Q: How many UA libraries do I need to have enough?
A: Not the more the better, the key to look at theVersion Distribution. It is recommended to maintain around 200 mainstream UA's, distributed proportionally by browser market share.

Q: How do I choose a package for ipipgo?
A: For small projectsSpirit Edition(5GB/day) is enough, large-scale acquisition directly on the enterprise customization package, their IP survival time is 3 times longer than others.

Q: Will I be recognized as using a proxy?
A: It can basically be avoided with high anonymity proxies + proper UA strategy. ipipgo'sResidential IP PoolAll are real person device IPs that are hard to recognize with the methods in this article.

As a final reminder, some sites will detectFont Rendering Differencessuch higher order features. This is the time to get on ipipgo'sBrowser environment simulationService, but that's a whole other topic.