Amazon Product Review Dataset: Product Review Datasheet

Why do I have to use a proxy ip for Amazon review data?

Do e-commerce friends know, want to analyze competitors have to stare at the evaluation of goods. But directly climb Amazon data, nine times out of ten will be blocked IP. last month I helped a friend to get a mother and baby products evaluation analysis, the local IP just grabbed 200 data, click was blocked, so angry that he almost smashed the keyboard.

That's when it's time to rely onProxy ip pool rotationto break the ice. The principle is simple:Every time you ask for a new "vest"., making the platform think it's being visited by a different user. It's like if you go to the supermarket to try something out and change your jacket each time, the clerk won't recognize you as the same person.

import requests
from ipipgo import get_proxy Here we use the ipipgo SDK.

def scrape_amazon_reviews(product_id).
    proxy = get_proxy(type='https', country='us') auto assign US residential ip
    headers = {'User-Agent': 'Mozilla/5.0'} remember to disguise the browser

    try: response = requests.get()
        response = requests.get(
            f'https://www.amazon.com/product-reviews/{product_id}',
            proxies={'https': proxy},
            headers=headers,
            timeout=10
        )
        return response.text
    except Exception as e.
        print(f'Scrape error, automatically switching ip to retry | error message:{str(e)}')
        return scrape_amazon_reviews(product_id) auto retry mechanism

The three big pits of choosing proxy ip, 90%'s people have been planted

Agent services on the market are mixed, I have seen the most outrageous case: a company bought a low-priced agent package, the results of the 50% ip are in the Amazon blacklist. Here to teach everyone to avoid the pit:

Hands on data messing with ipipgo

After signing up for a ipipgo account, focus on these two features:

1. Smart rotation model:Set every 5 requests to automatically change IP, with random UA header, pro-test catch 3 hours without being banned!

2. Failure to retry mechanism:Automatically switch IP to retry when encountering CAPTCHA, more than 10 times more efficient than manual processing

 Configuring an Intelligent Rotating Policy
from ipipgo import RotatingProxy

proxy_config = {
    'strategy': 'smart_rotate', smart mode
    'requests_per_ip': 5, 5 times per IP
    'retry_times': 3, fail to retry 3 times
    'geo_target': 'us-west' Specify the US west IP.
}

with RotatingProxy(proxy_config) as proxy.
     Your crawler code...

Frequently Asked Questions QA

Q: Can I get sued by Amazon for using a proxy IP?
A: As long as no malicious attacks are involved and robots.txt rules are followed, it is legal to simply collect public data. ipipgo's service agreement also explicitly prohibits illegal use.

Q: How many IPs are needed to be sufficient?
A: For 10,000 comments per day, it is recommended to prepare 500+ high-quality residential IPs. ipipgo's business package just includes a quota of 600 IPs per day, and it also sends automatic replenishment for failed requests.

Q: How do I break the CAPTCHA when I encounter it?
A: Don't tough it out! Immediately reduce the frequency of requests and switch ipipgo'sHigh Stash Residential IPThe code is a standardized code, which can be used in conjunction with automated coding services (note that this is a separate purchase).

A little bit of heartfelt experience.

Last year, I helped a big 3C manufacturer do a competitive analysis with ipipgo'sCity-level targeted IPA phenomenon was found: users in Los Angeles care more about product design, and New Yorkers are more concerned about functional parameters. This kind of geographically differentiated data can't be captured with ordinary agents.

A final reminder for newbies:Don't buy a junk proxy on the cheap.I'm sorry, but I'm not sure if I'm going to be able to do this! Previously, a friend was greedy for cheap, and the result was pitched by the supplier - the IPs given were all marked by Amazon, and the account was blocked just after the program was started, which was a loss.