Python json.loads: proxy IP-assisted parsing of web JSON data

When the crawler meets JSON data, proxy IP can help what?

Many just learned to crawl partners have encountered this situation: obviously got the web page to return the data, open a look at all the dense JSON strings, this time we have to ask out of ourjson.loads()to help. However, it is not enough just to be able to parse, if the website finds out that you visit frequently, it will block your IP in a minute. This is where proxy IPs come in, especially for sites likeipipgoThis reliable service provider allows you to create countless "alters" like the Monkey King pulling out hairs.

import requests
import json

 Proxy configuration with ipipgo
proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('https://api.example.com/data', proxies=proxies)
data = json.loads(response.text) key parsing steps
print(data['results'][0]['price'])

Proxy IP use three-piece suite

If you want proxy IP and JSON parsing to work well together, these three potholes should not be stepped on:

Practical case: capture e-commerce prices

Suppose you want to monitor the price fluctuation of a commodity, and the regular operation may be limited by the flow. Use ipipgo's high stash proxy with the following code to get a steady stream of data:

def get_price(product_id):: {'User-Agent': 'Mozilla/5.0'} Fake Browsers
    headers = {'User-Agent': 'Mozilla/5.0'} fake browser
    try: resp = requests.get()
        resp = requests.get(
            f'https://api.shop.com/products/{product_id}',
            proxies=proxies,
            timeout=5
        )
        return json.loads(resp.content)['currentPrice']
    except json.JSONDecodeError: print("JSONDecodeError", "JSONDecodeError").
        JSONDecodeError: print("Parsing exception, validation mechanism may have been triggered.")
        return None

Frequently Asked Questions QA

Q: Why is it still recognized even if I use a proxy?
A: It may be that the IP quality is not good, it is recommended to choose ipipgo's exclusive IP package, to avoid multiple people sharing lead to the characteristics of the repeated

Q：json.loads()报错咋处理？
A: First print the raw data to see if it is a validation page, you can use theresponse.content.decode('unicode_escape')View garbled content

Q: How to ensure the speed of data acquisition?
A: ipipgo's domestic BGP line latency can be controlled within 50ms, with connection pooling technology for better results!

The doorway to choosing a proxy service

The market is a mixed bag of agency services, so it's important to recognize three hard indicators:

• IP survival time > 6 hours (ipipgo Enterprise Edition supports 24-hour long-lasting IP)
• Simultaneous online IP number >500,000 (ipipgo actual available IP over 2 million +)
• HTTPS/Socks5 dual protocol support (this is something that many small vendors fail to do)

Finally, a tip: add IP health check module in the crawler script to test the proxy connectivity regularly. If you encounter response timeout, you can automatically pull new IPs from the API of ipipgo, so that the whole system can run stably for a long time. After all, data collection is like guerrilla warfare, flexible change of position is the key to victory.