python http proxy server: Python build HTTP proxy server

Hands-on with Python to build a proxy staging area

Recently, many friends who do data capture asked Lao Zhang, want to toss a proxy server and afraid of trouble. Today, we will take Python to start, the whole can actually run a HTTP proxy service. Don't panic, even if you're a beginner, follow the steps to ensure that it can be done.

Why build your own agent?

For example, you have a lot of proxy IPs to test the quality, you can't manually configure them one by one, right? Your own service is like an intelligent sorter, which can automatically switch between different IPs for testing. Another example is that some services require specific protocol conversion, the ready-made tools on the market may not be appetizing.

Here's the point:

The core advantages of self-built proxy servers areComplete control over where the traffic goes. You are free to add logging, request filtering, and other customization features, just like putting surveillance cameras on the data channels.

Get ready for your stuff.

Let's use the http.server module from the Python standard library for the base, and then install the requests library to handle the requests. Open cmd and hit this:

pip install requests

By the way, get the API documentation for ipipgo ready, you'll need to use their proxy pool for testing later. Remember that their extraction address looks like this:

https://api.ipipgo.com/getproxy?key=你的密钥

Basic Proxy Setup

First the whole can forward prototype, 20 lines of code to get it done:

from http.server import BaseHTTPRequestHandler, HTTPServer

class ProxyHandler(BaseHTTPRequestHandler):: do_GET(self).
    def do_GET(self).
         This is where the magic happens
        import requests
        resp = requests.get(self.path,
                          proxies={'http': 'proxy address provided by ipipgo'})

        self.send_response(resp.status_code)
        for k,v in resp.headers.items()::
            self.send_header(k, v)
        self.send_header(k, v)
        self.wfile.write(resp.content)

server = HTTPServer(('', 8888), ProxyHandler)
server.serve_forever()

After running it, the browser sets the proxy to 127.0.0.1:8888, and access to the web page will go out through the proxy IP of ipipgo. This beggar's edition is simple but complete.

Add a few useful buffs to the agent

The basic version is only a toy, we need to add something real:

1. Automatic IP changeover device

def get_new_proxy().
     Call ipipgo's API to get a new IP.
    return requests.get('ipipgo's API address').json()['proxy']

2. Small book of requests

Add a write log function in the handler to record which IP visited what website, so as to facilitate the subsequent analysis of the success rate.

3. Flow rate limiting valve

Use the TIME module to control the transmission speed and prevent overloading the IP. Especially when using dynamic residential IP, this feature can avoid extra chargebacks.

Playing with the ipipgo service

I have to brag about ipipgo's three best things here:

Their TK line is especially powerful when doing certain overseas business. Previously, a friend doing e-commerce used this program, and the success rate of the request soared directly from 60% to 92%.

Frequently asked questions on demining

Q: What about the snail-like agent speed?
A: First check if you are using a free IP, then confirm the protocol type. Remember to select the same geographic node when using ipipgo's static residential IPs

Q: The code reports SSL certificate error?
A: Add verify=False to requests, but this is not recommended for formal environments.

Q: How do I choose the right package?
A: the small amount of data selected dynamic standard version, the need for fixed IP selection of static, enterprise-class high concurrency directly to find their technology to customize the program

Suggested directions for upgrading

Join if you want to be more professional:
1. Proxy IP health check module
2. Automatic retry mechanism for failed requests
3. Traffic consumption statistics
These are all readily available in the ipipgo developer documentation.

Lastly, a self-built proxy server is like keeping a pet, you have to maintain and update it regularly. If you are too lazy to toss, directly use ipipgo ready-made client tool is more worrying, they have that one-click IP switching function is really fragrant.