Amazon customer review dataset: product review data

Getting an e-commerce data headache? Try this wild trick

Do Amazon merchants have recently been worried: customer reviews of this gold mine how to dig? The official interface restrictions, not to mention the direct climb and easy to be blocked. Last year, I helped my friend to engage in store optimization, and found alocal method-Used proxy IPs with automation tools and froze the competitor's 3,000+ bad reviews.

Data collection three big pitfalls, there is always a pitfall to you

1. IP blocking: Amazon is like a DUI check for frequently visited IPs, catching one and blocking another!
2. Captcha Hell: Sudden pop-up CAPTCHA interrupts the acquisition process
3. Data mutilation: Comments are not displayed in full in some areas

 For example, the normal crawler code
import requests
url = 'Amazon product link'
response = requests.get(url) This will get you banned the next day!

How to choose a proxy IP so as not to pay IQ tax

There are a variety of agency services on the market, so we recommend focusing on these three points:

Special mention to ipipgo'sIntelligent SwitchingFunction, can automatically change IP + change UA header, than manual operation to save a lot of heart. The last time I collected comments from a German station, I swiped for 8 hours in automatic mode without interruption.

Teach you to build a collection system by hand

1. Sign up for a ipipgo account to receive 500M test traffic.
2. Generate API keys in the background
3. Modify the crawler code:

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}

 Remember to add random delays and simulate mouse scrolling
response = requests.get(url, proxies=proxies, timeout=10)

Frequently Asked Questions QA

Q: How can I break it if I keep getting asked to verify?
A: two ways: ① reduce the collection frequency ② use ipipgo's high stash of residential IPs

Q: What should I do if I get disconnected halfway through the acquisition?
A: add a retry mechanism in the code, ipipgo backend can be set to automatically switch nodes

Q: What if I need to capture multiple comments?
A: Select the global node of ipipgo, remember to add the corresponding language parameter in the request header

Tell the truth.

Proxy IP is not a panacea, but it is really the most reliable solution at the moment. Recently, I found that some merchants have started to usedistributed acquisition: 10 crawlers + 100 IP rotation, with ipipgo's traffic pool management, daily average can pick 50,000 pieces of data has not been blocked. The cost of this play is a little high, but it is suitable for big sellers who want to do in-depth analysis.

Finally, to remind the novice: do not buy cheap junk proxy, I have seen someone with a free IP library, the results of the data back are all garbled. Reliable service providers like ipipgo, although it costs a little money, but can save a lot of time to toss.