Crawler Proxy Server: Crawler Proxy Server Tutorials

Hands on with building your own crawler agent pools

Brothers engaged in crawling know that the anti-climbing mechanism of the site is now more and more ruthless. Yesterday can run the program, today may give you a blocked dead. This time you need a proxy server toFake Real IP, making the target site think that a different person is operating on each request.

There are many ready-made proxy services on the market, but building your own is more flexible and affordable. Here we teach you to useipipgo Dynamic Residential ProxyDoing a live demo, their resource pool is large enough that the probability of being blocked is much lower.

Don't be sloppy with your prep.

First, prepare a cloud server (1 core 2G enough), the system recommended CentOS7. note that to chooseOverseas NodesThe most important thing to remember is that domestic servers are prone to being banned by association. Here is a pit to remind: don't buy those shared IP web hosting cheap, you must use independent IP cloud server.

 Install the base tools
yum install -y gcc python3-devel
pip3 install proxypool

Four Steps to a Practical Build

1. Go to the official website of ipipgo to register for an account, select theDynamic Residential (Standard) Package, $7+ 1G traffic is enough for testing. Find the API extraction link in the backend, it looks like this:

https://api.ipipgo.com/get?key=你的密钥&count=20

2. Configure the proxy pool program (here with the open source proxypool transformation):

 Modify config.py
API_URL = 'The API link you got above'
VALID_CHECK_INTERVAL = 60 check availability every minute

3. Start the service remembering to open the firewall port:

firewall-cmd --add-port=5032/tcp --permanent
systemctl restart firewalld
nohup python3 main.py > /dev/null 2>&1 &

4. Call the proxy pool in the crawler code:

import requests
def get_proxy(): return requests.get("").json().get("proxy")
    return requests.get("http://你的服务器IP:5032/get").json().get("proxy")

 Example of use
resp = requests.get(url, proxies={"http":get_proxy()})

See here for tuning tips

- come across403 errorDon't panic. Go to ipipgo and switch backstage.Socks5 protocoltry out
- High-concurrency scenarios recommend upgrading toEnterprise Edition Dynamic Residential9 more than 1G to support higher concurrency
- Automatically restart proxy pool scripts at 3am to avoid memory leaks
- When collecting European and American websites, add the API link to the&country=usdesignated area

Guidelines on demining of common problems

Q: What should I do if the proxy IP survival time is too short?
A: Set the detection interval to 30 seconds, and at the same time turn on the ipipgo backgroundLong-term model(corporate packages required)

Q: What if I need a fixed IP for login?
A: Switch to $35/monthStatic Residential IPThe IP can be used for a full 30 days.

Q: The IP returned by API is not available?
A: First check the whitelist settings, ipipgo need to bind the server IP to call the API

Why ipipgo?

theirTK line agentDo cross-border e-commerce data collection is particularly stable, before a friend to do independent station, with this program daily mining 300,000 data has not been closed. The key is fast customer service response, the last midnight encounter technical problems, actually 10 minutes on remote assistance to get it done.

Lastly, I would like to remind newbies: don't run large file downloads on proxy servers! There is a buddy to take the proxy pool under the movie, 1 hour to the package flow used up, this operation blood loss. Do collect to control the request frequency, with User-Agent random is the king.