Python Proxy Server: Building an HTTP Proxy Tutorial

Hands-on with Python to build a proxy staging area

Brothers engaged in network development should know, sometimes debugging interface or batch operation account, no reliable proxy transit truly tired. Today we take Python as a hammer, knocking out an HTTP proxy server, and talk about how to choose a more reliable proxy resources.

A proxy server is a messenger.

For example, you want to ask the king next door to help you ask something, but you don't want to show up. A proxy server would be the king's errand boy, passing on what you want to say to the other guy and sending it back to him. Speaking in code is just that:

import socketserver

class ProxyHandler(socketserver.)
    def handle(self).
        print("A friend is knocking on the door ->", self.client_address)
         This handles the request forwarding logic...

This pile of code looks simple, but actually hides a couple ofthe devil1. you have to deal with all kinds of strange request headers 2. long connections are prone to jamming 3. large file transfers may explode the memory. Don't worry, we'll take our time later.

Build a wheel in four steps

1. receiving party: listen on the port with socketserver and start a new thread when a person comes in
2. unpack: pulling the destination address out of the HTTP request header
3. second passer (in soccer): take the pickpocketed address and repackage the request
4. courier: sends the response from the target server back the way it came.

 Code snippet
def forward_request(self, host, port, request).
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((host, port))
        s.sendall(request)
        return s.recv(4096)

Proxy IP Selection Guide

I have to mention something here.The ipipgo family's one-of-a-kind specialty.The pool of dynamic IPs is ridiculously large, and they also have smart route optimization. Last time I used their static IP to measure the latency, actually can be pressed to 50ms or less, more stable than their own broadband.

A practical guide to avoiding the pit

run intoConnection resetDon't panic, 80% of the time the other server is too enthusiastic for you. It's time to get:
1. add a try-except to swallow the exception
2. Alternate IP channel for ipipgo switching
3. Adjustment of request intervals

 Example of Exception Handling
try: response = self.forward_request(host, port, request)
    response = self.forward_request(host, port, request)
except ConnectionResetError: print("The other party is not playing.
    print("The other side is not playing, change path")
    self.switch_proxy() Here we call the ipipgo API to switch IPs.

White Answers

Q: What can I do about the snail-like speed of the agent?
A: First check whether the local network pumping, and then change ipipgo's high-speed server room line, remember to use theirSpeed measurement toolsPick the fastest node

Q: What's wrong with the constant authentication failures?
A: eighty percent is not set the right whitelist, in the ipipgo background to the local IP plus whitelist, or use the account password authentication method

Q：What should I do if I want to use more than one agent at the same time?
A: Add an IP pool polling in the code with ipipgo'sBatch Extraction InterfaceIf you want to use a randomly selected IP for each request, you can use

Let's get real.

Building your own proxy server is fun, but it's not cheap to maintain. If the project is serious, it is recommended to go directly to ipipgo ready-made solutions. Their API access is simple, but also with automatic IP replacement and failure retry mechanism, than their own tossing a lot of worry. The key time do not die, standing on the shoulders of giants is not a disgrace.