PythonHTTP Proxy Server: Build Tutorials

Hands-on with Python to build a proxy server

Recently, many friends asked how to build a proxy server to use, especially to do data collection, multi-account management of these businesses. Today we will use Python to get asimple and practicalof HTTP proxies, focusing on how to use them in conjunction with ipipgo's premium proxy IPs.

Get ready for your stuff.

First make sure you have Python 3.6 or higher on your computer. You need to install a third-party library calledproxy.py, this thing is specifically designed to mess with proxy services. Hit this at the command line:

pip install proxy.py

If the installation gets stuck, try adding-i https://pypi.tuna.tsinghua.edu.cn/simpleChange domestic sources. I have a tip here, remember to update to the latest version after installing, some older versions have weird bugs.

Base model agent construction

new constructionbasic_proxy.pyfile, write these codes below:

from proxy import Proxy

def main(): proxy = Proxy(port=8866)
    proxy = Proxy(port=8866)
    proxy.start()

if __name__ == '__main__'.
    main()

After running it your proxy server is in thePort 8866Running. Fill in your browser settings when you use127.0.0.1:8866It will work. But there are two problems with this barebones version: first, it uses the local IP, and second, it's easy to be stolen without authentication.

Advanced Version: Dynamic IP + Authentication

It's time to bring out theProxy services for ipipgoIt's a good idea to get an API key from their website first. First go to their website and get an API key, the free trial package is enough for testing. Rewrite the code:

import requests
from proxy import Proxy, ProxyHandler

class CustomHandler(ProxyHandler).
    def get_upstream_proxy(self).
         Get a new IP for each request
        ip_data = requests.get(
            "https://api.ipipgo.com/get-proxy",
            params={"type": "http", "count": 1}
        ).json()
        return (ip_data['ip'], ip_data['port'])

    def handle_request(self, data).
         Add a simple authentication
        if b'Secret-Key: mypassword' not in data: return self.client.
            return self.client.send(b'HTTP/1.1 403 Forbiddenrr')

Proxy(port=8866, handler=CustomHandler).start()

There are three highlights of this release:
1. each request automatically change ipipgo IP, not afraid of being blocked
2. must be used with a specified key header
3. Going to ipipgo's high stash of proxies, hiding the real IP more thoroughly

Real-world configuration parameters

This configuration table is recommended for tuning:

A guide to common pitfalls

Q: What should I do if I can't connect to the agent all the time?
A：先检查防火有没有放行端口，再试试curl命令curl -x http://localhost:8866 http://ip.ipipgo.com/ipSee if the returned IP is ipipgo's

Q: What happened to the sudden slowdown?
A: It may be a local network problem, or the selected IP node has high load. It is recommended to add an IP scoring mechanism in the code to automatically eliminate the slow nodes

Q: How do I prevent people from stealing?
A：除了代码里的密钥认证，最好在服务器防火设置只允许指定IP访问代理端口

Why ipipgo?

Maintaining an IP pool by yourself is too much work, buying a server, doing verification, dealing with bans...all these troublesome things ipipgo has taken care of for you. The actual test of their home three advantages:
– full coverage: 300+ city routes nationwide
– responsive：平均<80ms
– protocol-rich: HTTP/HTTPS/Socks5 full support

Especially if you are doing long-term data collection, it is recommended to go directly to theirDynamic Residential IPThe simulation of real user behavior is not easy to be identified. New users register remember to receive 2G traffic trial, enough to measure the effect.

A few final words.

Proxy server building is not difficult to say, but to long-term stable operation also need to pay attention to:
1. Periodic updating of authentication keys
2. Monitoring IP usage
3. Setting up traffic alerts
4. Reactivation of services during the low morning peak period

If you can't figure it out, ipipgo actually has a ready-made proxy API that you can call directly to save yourself the trouble of maintaining it. However, if you are playing by yourself, you should not have any big problems according to the above tutorial. Encountered jammed place, their technical customer service response is quite fast, you can go to the official website to find online support.