Web site data capture: crawler proxy IP configuration program

The Proxy Doorway You Must Know to Engage in Data Crawling

The friends who engage in website data crawling understand that the most headache is to be the target site blocked IP. yesterday, the old king next door is still spitting, his crawler program just ran for half an hour, the server IP was blacked out, so he could only squat in the engine room to manually change the line. At this time if you can use a proxy IP, which is not as bad as this?

Proxy IPs are, to put it bluntlyInvisibility cloak for reptilesThe first is to make the website think that each request is operated by a different user. However, there are various types of proxies on the market, and it's even worse if you don't choose the right one. For example, to do e-commerce price monitoring, using a data center IP is easy to be identified, this time you have to use a residential IP is reliable.

Three Tips for Choosing the Right Proxy IP Type

Based on our experience of doing programs for thousands of companies, the main three dimensions to look at when choosing an agent are these three dimensions:

1. There is a difference between movement and static:
Dynamic IP is suitable for high-frequency crawling (e.g., ticket-grabbing scripts), where the IP is automatically changed every 5-15 minutes; static IP is suitable for scenarios where the login state needs to be maintained (e.g., social media monitoring).

2. Priority is given to dwellings:
Residential IPs come from real home broadband, and anti-climbing strategies are the hardest to recognize. Dynamic residential packages like ipipgo's, at more than 7 bucks for 1 G of traffic, the price/performance ratio hangs with the peers.

3. Protocol matching:
Newbies are recommended to use HTTPS protocol directly, saving effort and not tossing. Older drivers can use Socks5 protocol, the transmission speed is faster. Here is a Python configuration example:

import requests

proxies = {
    'http': 'http://user:pass@gateway.ipipgo.com:9020',
    'https': 'http://user:pass@gateway.ipipgo.com:9020'
}

resp = requests.get('destination URL', proxies=proxies)

A practical guide to matching rabbits (hand-held version)

Using the Scrapy framework as an example, add these lines to settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

IPIPGO_PROXY = "http://user:pass@gateway.ipipgo.com:9020"

def process_request(request, spider).
    request.meta['proxy'] = IPIPGO_PROXY

Be careful to putuserrespond in singingpassSwitch to the key you got in the ipipgo backend. It is recommended to add an exception retry mechanism in the code to automatically switch IP nodes when encountering a 403 error.

Avoiding the Pit Q&A

Q: Proxy IPs are not working when I use them?
A: 80% of the use of poor quality proxy pool. ipipgo's residential IP survival cycle are more than 12 hours, the background can also check the IP availability rate.

Q: Will I be blocked for having multiple threads open at the same time?
A: Depends on the type of proxy package. Dynamic Residential (Enterprise Edition) supports 500 concurrency, and normal packages are recommended to control within 50 threads.

Q: Do I need to maintain my own IP pool?
A: Just use the API interface of ipipgo to automatically assign a new IP for each request. code example:

import random

def get_proxy().
    proxy_list = requests.get("https://api.ipipgo.com/dynamic").json()
    return random.choice(proxy_list)

How to choose a money-saving package

Right-sized according to the size of the business:
- Individual small projects: dynamic residential (standard) $7.67/GB
- Enterprise-level acquisition: $9.47/GB for dynamic residential (enterprise) (with high concurrency privileges)
- Long-term monitoring needs: $35/IP/month for static homes

Lastly, I would like to remind newbies not to trust those free agents. We have received a lot of cases, customers cheap with free IP, the result of the data did not catch, but was implanted mining scripts. Regular service providers have a traffic audit mechanism, such as ipipgo's dedicated line are operators directly signed, the security of this piece of pinch dead.