http proxy server: HTTP proxy server setup full strategy

Hands-on teaching you to play with HTTP proxy servers

Recently, many friends who do data collection find me complaining, saying that the website anti-climbing is getting more and more fierce. Last week, the old king just wrote a good crawler script, this week will not be able to catch the data. In fact, this thing, set up a HTTP proxy server can be solved. Today we will nag how to get a proxy server, by the way, a reliable proxy service provider.

What exactly is a proxy server?

Simply put, it's a middleman that relays web requests for you. For example, if you want to access a certain website, you first let the proxy server knock on the door, and it moves the data back and forwards it to you. This has two benefits:One is to hide the real IP(math.) genusSecond, break through access restrictionsThe following is a list of some of the most popular websites in the world. Note that the access restriction here refers to the limitation of the frequency of requests from certain websites to a single IP, not any special operation.

Do-It-Yourself Agents

Here's how to do it with Python'shttp.serverModules to give a chestnut. First install a tripartite library:

pip install PySocks

Then create a newproxy_server.pyDocumentation:

import socketserver
import http.server

PORT = 3128
class MyProxy(http.server.SimpleHTTPRequestHandler)::
    SimpleHTTPRequestHandler. def do_GET(self).
        self.send_response(200)
        self.end_headers()

if __name__ == '__main__': with socketserver.
    with socketserver.ThreadingTCPServer(('', PORT), MyProxy) as httpd:
        print("Proxy running, port:", PORT)
        httpd.serve_forever()

Once it's running, fill in the browser settings127.0.0.1:3128It will work. However, this toy agent is only for testing, and you really need to use a professional solution for production.

The right way to open a professional player

Maintaining a proxy server on your own is too much work, so here's a good recommendationipipgoof ready-made services. Their home has three particularly useful axes:

Extracting proxy IPs with their API is massively convenient, to give you a chestnut:

import requests

proxy = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}

response = requests.get('target site', proxies=proxy)

A practical guide to avoiding the pit

1. Don't panic if your proxy fails.: It is recommended to randomly select 3-5 IPs per request as spares
2. Timeout settings are important: Don't use the default timeout, adjust it to 3-10 seconds depending on the business scenario
3. Remember to disguise the request header: Remember to change the User-Agent often, and don't use those crappy fields!

Frequently Asked Questions QA

Q: How to choose between dynamic and static IP?
A: need to maintain a long-term login state choose static, ordinary collection with dynamic more cost-effective

Q: What should I do if I encounter 403 forbidden?
A: First check if the request header is complete, then try another IP. Use ipipgo's enterprise package for a higher success rate

Q: How can I tell if a proxy is in effect?
A: Visitshttp://httpbin.org/ipSee if the returned IP changes

One final rant, don't just look at price when choosing a proxy service. The likes of ipipgo can provide200+ country resourcesIt's still possible.1v1 Customized SolutionsThe only thing that is really reliable. Especially their TK line, do cross-border e-commerce friends used to say really fragrant, specific how fragrant method you try to know.