News crawler: news crawler set proxy tutorial

Teach your news crawlers to put on a cloak and dagger.

The old iron engaged in news gathering know that the website anti-climbing mechanism is now more and more refined. Especially the kind of 24 hours non-stop to catch data, minutes on the IP blocked. this time to rely on proxy IP to be "stand-in actor", today we will talk about how to give the crawler to wear a good layer of invisibility cloak in the vernacular.

Why does your crawler always get caught?

Many newbies will be wondering at first:Obviously the code is fine, why did it suddenly strike? In fact, there is a website background "blacklist book", found that the same IP frequent visits will be pulled black. To cite a chestnut, just like the supermarket tasting staff always go to the same booth to get food, security must be suspicious ah.

Proxy IP Selection Guide

There are two main types of agents on the market:

Dynamic Residential IP - Like a face-changing Szechuan opera singer, he changes his face every time he visits.
Static Residential IP - It's like being an undercover agent who lurks for a long time, suitable for scenarios where you need to log in steadily

News gathering recommended with dynamic IP, especially like ipipgo's dynamic residential packages, more than 7 yuan 1G traffic is affordable enough. If enterprise-level projects, his family more than 9 dollars of enterprise version more resistant to build.

Proxy Configuration in Three Steps

Take the Python requests library as an example:

import requests

 Proxy information from the ipipgo backend
proxy = {
    "http": "http://用户名:密码@gateway.ipipgo.com:端口",
    "https": "http://用户名:密码@gateway.ipipgo.com:端口"
}

response = requests.get('https://目标新闻网站', proxies=proxy, timeout=10)
print(response.text)

Pay attention to change the username and password to the account you registered in ipipgo, the port number can also be found in their background. It is recommended to set the timeout parameter, don't let the program die waiting.

Guide to avoiding the pit (QA session)

Q: What should I do if I use a proxy or get blocked?
A: Check if the IP pool is too small, we suggest you choose a service provider like ipipgo which covers 200+ countries. If it doesn't work, get their tech guy to customize the solution.

Q: Do I need to open an agent for wee collection?
A: Don't save this money! Websites are now monitored 24/7, and it's easier to get caught in the middle of the night grabbing data.

Q: How can I tell if a proxy is in effect?
A: Add a detection function in the code, such as visiting ipinfo.io to see if the returned IP address changes.

How to choose a ipipgo package

There's a hidden perk in their house--Free debugging traffic for new users on their first rechargeI've been looking for a customer service girl to help me with this. Technical support is really reliable, the last time I mention work orders at 3 am actually someone back...

Say something from the heart.

Proxy IP is not a panacea, with random access intervals, User-Agent camouflage these tricks. If the budget is enough, it is recommended to go directly to ipipgo's enterprise version of the package, after all, the timeliness of the news data can not be delayed. Encountered a special site can not handle, do not fight, let their technical team out of the customized program more worry.