Amazon Web Crawl: Compliance Product Capture

Hands-on with proxy IPs to mess with Amazon product data

Recently, many friends who do e-commerce are asking how to safely pickpocket Amazon product information. In fact, it is not difficult to say that this matter is not difficult, the key to find the right method. Today we will nag how to use proxy IP compliance to engage in data, by the way, Amway a reliable tool ipipgo.

Why do you always get pulled by Amazon?

Many newcomers come to use their own network to brush the page violently, and the result is that the account is GG in less than two days. Amazon's anti-climbing system is stricter than the neighborhood gates, mainly looking at three indicators:Access frequency, IP attribution, device fingerprints. Especially with frequent visits from domestic IPs, you'll be labeled a bot in minutes.

 错误示范：采集
import requests
for page in range(1,100):
    response = requests.get(f'https://www.amazon.com/s?page={page}')
     这样搞必封号！

How to choose a reliable proxy IP

There are all sorts of agency services on the market, so keep these hard indicators in mind:

Here must boast ipipgo's dynamic residential IP pool, their home IP are real home network, each request automatically change IP, pro-test run for three days in a row has not been blocked.

Real-world configuration tutorials

Take Python as an example, and use the ipipgo API to implement smart switching:

import random
import requests

def get_proxy().
     Get dynamic proxies from ipipgo
    proxy_list = [
        "us-wa.ipipgo:8000",
        "de-ber.ipipgo:8000",
        "jp-tky.ipipgo:8000"
    ]
    return {'http': f'http://{random.choice(proxy_list)}'}

response = requests.get(
    'https://www.amazon.com/dp/B08J5F3G18'.
    proxies=get_proxy(),
    timeout=10
)

Focused attention:You must change your IP before each request, and it is recommended to set a 3-5 second interval, so don't fool Amazon.

White Frequently Asked Questions QA

Q: Is it safe to 100% with a proxy IP?
A: There is no absolute safety, but in conjunction with ipipgo'sIntelligent Rotation + Request Interval ControlThe risk of 90% blocking can be minimized.

Q: How to control the acquisition frequency?
A: It is recommended that the single product page does not exceed 50 times per hour, and the search results page does not exceed 20 times, depending on the heat of the category

Q: What should I do if I encounter a CAPTCHA?
A: Immediately stop the collection of the current IP, ipipgo background will automatically isolate the abnormal IP 12 hours

Why recommend ipipgo

I've been using it for over 6 months and found a couple of really nice features:
1. Global 200+ countries IP switch at any time
2. Automatic clean-up of failed nodes
3. Separate IP pools optimized for Amazon
4. 5G of free traffic for newcomers to test the waters
Especially theirIP Quality Inspection SystemThe ability to filter IPs flagged by Amazon in advance is a feature you really don't see elsewhere.

As a final rant, the core of compliant capture is theSimulation of real-life operationThe first thing you need to do is to use the right tools and control the pace of the process. Don't be greedy, use the right tools + control the pace, in order to get the data in the long run. If you don't understand anything, you are welcome to come to ipipgo's official website to find customer service girls, they are much more than I would say.