Amazon Data Crawling (Python): Amazon Agent Crawler Development

Why do I have to be on a proxy to do Amazon data crawling?

Old iron must have encountered, with Python script just grabbed a few pages of Amazon on the jump out of the CAPTCHA, serious direct IP blocking. these days to do e-commerce data monitoring, who do not have a few agents in the hands of the pool? To cite a chestnut, our team last year with the native IP to catch price data, the results of 3 days on the blacklist, and then changed the ipipgo residential agent is as stable as the old dog.

The best thing about proxy IPs is thatMake the server think you're a real person visiting. For example, if you use a dynamic residential IP and change your home broadband address in a different region for each request, Amazon's anti-crawl system won't be able to tell if it's a real person or a machine.

Practical configuration proxy crawler

Here is the whole Python example for the guys, using the requests library + ipipgo proxy. Focus on the auth parameter settings, many people fall in this piece:

import requests

 API extraction link from ipipgo backend
proxy_api = "https://api.ipipgo.com/getproxy?type=dynamic&count=1"

def get_proxy():
    resp = requests.get(proxy_api)
    return f"{resp.json()['ip']}:{resp.json()['port']}"

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36...'
}

proxies = {
    'http': f'socks5://{get_proxy()}',
    'https': f'socks5://{get_proxy()}'
}

try.
    response = requests.get(
        'https://www.amazon.com/dp/B08J5F3G18',
        proxies=proxies,
        headers=headers,
        timeout=15
    )
    print(response.text[:500]) Print the first 500 characters to see the effect.
except Exception as e.
    print(f "Rollover: {str(e)}")

Pothole Point Reminder:Don't use free proxy! We have tested more than two dozen service providers in the market, and finally used ipipgo's TK line to solve the problem of the U.S. product page loading incomplete.

Agent selection doorway

To give you a comparison table, different business needs correspond to different agent types:

In particular.TK LineThis thing is specially optimized for overseas e-commerce platforms, and the actual test grabbed Amazon's picture loading speed is more than 3 times faster than ordinary agents.

QA session

Q: Why am I still blocked even though I set up a proxy?
A: 90% of the probability is that the User-Agent is not randomly replaced, it is recommended to change the browser fingerprint every 50 requests.

Q: How much IP volume is needed per day?
A: Look at the collection frequency, generally 5 requests per second, if the dynamic residential package to choose 7.67 yuan / GB is enough to use!

Q: What should I do if I encounter a 403 error?
A: immediately check three points: 1. whether the proxy is in effect 2. whether the request header with a cookie 3. IP purity (with ipipgo's detection tool to check)

How to choose a ipipgo package

They have three levels of packages:
- Dynamic Standard Edition: suitable for small teams just starting out, $7.67/GB cabbage price
- Dynamic Enterprise Edition: with request priority guarantee, a must-have for grabbing seconds of data
- Static residential IP: account registration to raise the number of this choice, 35 dollars an IP with a whole month!

Finally said a riotous operation: the ipipgo client loaded on the cloud server, with selenium to do distributed collection, pro-tested at the same time open 200 browser instances have not been blocked. Specific configuration program can find their technical brother to ready-made scripts, said to read this article can also send half an hour of test time.