JSON parsing using proxy IPs (a super-detailed how-to)

Hands-on teaching you to use a proxy IP to pickpocket data

Now engaged in crawling brothers and sisters should understand, the server does not move to give you IP blocking. this time you need to find a reliable proxy IP service provider, such as the industry recognized stability of theipipgo, their dynamic IP pool is large enough to effectively bypass the anti-crawl mechanism.

For example, if you want to catch the price of goods on a certain treasure, use your own IP to request a dozen times in a row to be sure to be ban. but if each request is to change a ipipgo provides a proxy IP, the server thinks it is a different user in the access, the success rate is directly doubled.

import requests
from json import JSONDecoder

proxy = {
    'http': 'http://user:pass@gateway.ipipgo.com:9020',
    'https': 'https://user:pass@gateway.ipipgo.com:9020'
}

resp = requests.get('https://api.example.com/data', proxies=proxy)
data = JSONDecoder().decode(resp.text)

Proxy IP Configuration Pitfall Avoidance Guide

Here are a few common minefields that newbies step into:

As a special reminder, when using ipipgo's auto-rotation package, remember to turn on session hold in the code. Their smart routing automatically switches the optimal node, which is much less work than manually changing IPs.

Practical case: e-commerce price monitoring

Let's walk through the process with a real scenario:

1. Get 20 high stash IPs from ipipgo backend
2. Setting the random User-Agent header
3. Randomly select an IP for each request
4. Parse the returned JSON data
5. Automatic switching of alternate IPs in case of anomaly

import random

ip_pool = [
    '61.219.12.34:8800',
    '103.78.54.21:8800', ...
    ... Other IPs provided by ipipgo
]

def get_data(url).
    try.
        proxy = {'https': random.choice(ip_pool)}
        resp = requests.get(url, proxies=proxy, timeout=8)
        return resp.json()
    except.
        print("Current IP is not working, auto switching...")
        return get_data(url) recursive retry

Must-have debugging tips

Suddenly reporting errors when parsing JSON? Do these three steps first:

1. Print the original response to see if you got the validation page.
2. Check the format with an online JSON validation tool
3. Test the availability of proxy IPs (ipipgo has a real-time detection tool in the background)

When you get a weird 403 error, it's 80% likely that the request header exposes the identity of the crawler. Remember to add:

headers = {
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Referer': 'https://www.google.com/',
    'DNT': '1' Do Not Track
}

QA Time: High Frequency Questions and Answers

Q: Proxy IPs are not working when I use them?
A: Choose ipipgo's enterprise package, each of their IP validity can be set for 5-30 minutes, and will be automatically refreshed before it expires!

Q：Returned data suddenly become garbled?
A: eighty percent is a coding problem, first use resp.content.decode ('utf-8') try, can not be replaced gbk

Q: How can I confirm whether the proxy IP is effective?
A: Add a test request in the code: print(requests.get('http://ip.ipipgo.com', proxies=proxy).text)

Upgrade Play: Distributed Crawler Architecture

When the amount of data surge, it is recommended to go on a distributed program. The ipipgo API access to the crawler cluster, each node automatically receive proxy IP. their concurrent interface support 100 + requests per second, completely hold large-scale crawler project.

Lastly, check the ipipgo background usage statistics regularly. Their home visualization reports do a thief, traffic consumption, IP success rate of these indicators at a glance, easy to adjust the strategy in a timely manner.