News Data API: News API Calls and Proxy Settings

Why are news data crawls always blocked?

Brothers who have engaged in news data collection understand that the biggest headache is that the target site suddenly gives you a403 Denial of AccessThe first thing I did was to get the news crawler to work for my friend. Last week I helped a friend debugging news crawler, obviously no problem with the code, but even grabbed half an hour quasi-IP blocked. later found that the site are now learning fine, see the high-frequency access to the direct black IP segments, regardless of whether you're a real person or a machine.

This is the time to offer up the godsend that is the proxy IP. Simply putKeep changing the crawler's "armor"., making the site think that it is visited by different users. Like you go to the supermarket to try to eat, can not let the same person try to eat 100 times, right? If you change your clothes and go back, the clerk won't recognize you.

Hands-on: Putting a Proxy Vest on the News API

Here's an example using Python's requests library. Pay attention to the location of the proxy parameter settings, just like the courier parcel sticker, you have to stick in the right place to be delivered:

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:端口',
    'https': 'http://用户名:密码@gateway.ipipgo.com:端口'
}

 Pretend to be accessed by a normal user
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
}

response = requests.get(
    'https://newsapi.org/v2/top-headlines',
    params={'category': 'technology'},
    headers=headers,
    proxies=proxies,
    headers=headers, proxies=proxies, timeout=10
)

The key points are in these places:

• Proxy address with account password (don't write it directly in the code, put it in an environment variable for more security)
• User agent masquerading as a browser
• Don't set the timeout too short, 5-10 seconds is recommended

Choosing a proxy IP is like buying groceries

Agency services on the market are a mixed bag, here are a few easy to step on the pit:

Here's an honorable mention for our own productsipipgoThe dynamic residential IPs are especially suitable for news gathering. There is a cold knowledge: many news websites will push different contents according to the geographic location of the visiting IP, using his family's IP resources in 200+ countries around the world, you can collect more comprehensive news data.

QA Time: Frequently Asked Questions for Newbies

Q: Will proxy IPs slow down the collection speed?
A: good proxy service latency control within 200ms, faster than human access. ipipgo's TK line measured average response of 180ms, does not affect the efficiency of the

Q: What if I need to manage multiple agents at the same time?
A: Directly use the API they provide to obtain IP pool, code samples are available on the official website. Remember to set the automatic switching frequency, it is recommended to change the IP every 5-10 requests.

Q: What should I pay attention to when gathering overseas news?
A: Focus on the quality of the cross-border line of the agent service. ipipgo's cross-border line is a direct connection to the operator, unlike some service providers to bypass the third country, the freshness of the data is guaranteed!

Saving program: how to choose ipipgo packages

Right-sized according to the size of the business:

• Small-scale test: dynamic residential standard version, more than 7 yuan 1G traffic enough to run tens of thousands of requests
• Long-term stable collection: static residential IP, 35 bucks a month without worrying about IP failure
• Enterprise-level requirements: directly to customer service for a customized solution, able to deploy IP resources on demand

As a final reminder, using a proxy is not a get-out-of-jail-free card. Or to comply with the website robots agreement, control the collection frequency. After all, we are serious about data collection, do not get hung up on their servers. Encounter CAPTCHA don't hard just, appropriate add a little interval, with the proxy IP to use better results.