How to set up a reverse proxy for a dynamic ip?

Teach you to play reverse proxy with dynamic IP handily

Recently, a number of e-commerce friends and I complained that the use of ordinary servers to do data collection is always the target site blocked IP. this is something I have too much experience, last year, our team to do competitive analysis of the time, every day to change a few hundred IP in order to ensure that the data capture. Today with the guys nagging how to build a reverse proxy with dynamic IP, especially with Nginx upstream unique configuration skills.

Three great uses for dynamic IP reverse proxies

Many people think that reverse proxy is a server thing, in fact, with dynamic IP is really fragrant:

1. Collecting data is not afraid of blocking IP (personally tested with ipipgo's dynamic residential IP, the collection of 3 consecutive days without being intercepted)
2. do load balancing automatically switch fault nodes
3. to the crawler program to wear a cloak of invisibility (remember to choose to support HTTP/SOCKS5 dual-protocol proxy)

Nginx upstream dynamic configuration in four steps

Here's a pitfall to be aware of: don't use the outdated static configuration methods on the web! We have to use resolver to implement dynamic updates:

http {
    resolver 8.8.8.8 valid=30s; DNS server should be a reliable one
    upstream dynamic_proxy {
        zone upstream_dynamic 64k; server proxy.ipipgo.com:8000 resolve; fill in the proxy name here
        server proxy.ipipgo.com:8000 resolve; here is the access point provided by the proxy provider
    }

    server {
        location / {
            proxy_pass http://dynamic_proxy; proxy_set_header Host $host; proxy_set_header
            proxy_set_header Host $host.
        }
    }
}

Here's the point!valid=30sThis parameter has to be adjusted according to how often the proxy IP is changed. For dynamic residential packages like ipipgo, it is recommended to set it to update once every 5 minutes, and their IPs can be available for up to 6 hours on average.

A unique technique for dynamic IP preservation

Don't rush to reboot after the configuration file change, try this trick to make the IP pool update automatically:

 Script to clean up invalid IPs at regular intervals
! /bin/bash
while true
do
    curl -X POST "http://localhost/upstream_conf?upstream=dynamic_proxy"
    sleep 300 Align with the proxy package IP expiration date.
done

It's even better with ipipgo's API. Their interface response time is measured within 200ms, more than double that of many service providers.

Guidelines on demining of common problems

Q: What should I do if I always get a 502 error after configuration?
A: Ninety percent of the time the protocols aren't right, check three points:
1. Nginx version is not 1.19.4 or above
2. Whether the proxy service provider supports the HTTP protocol (standard dynamic package from ipipgo is recommended)
3. Firewalls are not releasing proxy ports

Q: How can I tell if the dynamic IP is in effect?
A: Add a variable to the Nginx log:

log_format proxy_log '$remote_addr - $upstream_addr';

Seeing different IP addresses change frequently is right.

Why we recommend ipipgo Dynamic Residential Proxy

After using seven or eight service providers, I finally chose ipipgo for three main reasons:

1. support per traffic billing (small teams can save 2/3 of the cost per month)
2. Exclusive city-level positioning function (essential for localized business)
3. automatic capacity expansion when the traffic fluctuates greatly (saved life during the double eleven)

There's also a hidden perk to their Enterprise package - you can use both dynamic and static IPs, which makes it especially convenient to do AB testing.

The Ultimate Reminder to Avoid Pitfalls

One final note for newbies:
① Test phase first with pay-as-you-go package (ipipgo new users get 1G traffic)
② Overseas business remember to choose the corresponding region of the agent node
③ Be sure to configure dual service provider backups for important projects (although we haven't had a failure in two years with ipipgo)

In accordance with this program configuration, our team's collection success rate from 47% directly soared to 92%, the key is no longer need to get up in the middle of the night to change the IP. What specific questions are welcome to tease, see will be back.