Python requests to get JSON: API data processing

A. Why crawl data is always pulled? Try this method

Engaged in data collection of the old iron are sure to have encountered such a situation: with the requests library just grabbed two pages of data, IP on the target site off the small black house. At this time, don't be in a hurry to smash the keyboard.proxy IPIt's your saving grace! It's like playing a game and opening a small number, changing your vest and continuing to work.

For example, some e-commerce site's anti-climbing mechanism thieves, the same IP access to a dozen consecutive times to trigger the alarm. At this time, if you use ipipgo's dynamic proxy pool, each request for a new export IP, the other server can not distinguish between real people or programs, naturally, you will not be blocked.

import requests
from itertools import cycle

 List of proxies provided by ipipgo (example)
proxies = [
    "http://user:pass@gateway.ipipgo.com:30001",
    "http://user:pass@gateway.ipipgo.com:30002".
    "http://user:pass@gateway.ipipgo.com:30003"
]
proxy_pool = cycle(proxies)

for page in range(1, 50): current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
        resp = requests.get(
            "https://api.example.com/data",
            proxies={"http": current_proxy},
            timeout=10
        )
        print(resp.json())
    except Exception as e.
        print(f "Rollover with {current_proxy}:", str(e))

Second, the proxy IP configuration of the three pits, 90% newbies have planted

1. Authentication Information Omission: Many brothers directly write an IP address on the end, the result returned 407 error. ipipgo proxy need to fill in the username and password, the format ishttp://用户名:密码@GatewayAddress:Port

2. Improperly set timeout: Some proxy nodes may be slow to respond, without the timeout parameter, the program will be stuck. It is recommended to set a timeout of 5-15 seconds according to business requirements.

3. Missing Exception Handling: Network requests are inherently unstable, especially when using proxies, it is more important to do a good job of retrying errors. It is recommended to use retry decorator to realize automatic retry mechanism.

Third, JSON data processing skills in practice

After getting the JSON data returned by the API, don't be in a hurry to store it directly in the database. First do this several processing:

1. Data Cleaning: Extracting key fields with jsonpath is much easier than parsing them manually. For example$...priceAbility to quickly extract all prices

2. Outlier Filtering: When encountering null values or incorrectly formatted data, log and skip in a timely manner

from jsonpath_ng import parse

def process_data(json_data).
     Extract product name and price
    name_expr = parse('$..productName')
    price_expr = parse('$..price')

    results = []
    for match in name_expr.find(json_data):
        product = {'name': match.value}
        price_match = price_expr.find(json_data)
        if price_match.
            product['price'] = float(price_match[0].value)
        results.append(product)
    return results