Crawler API: Automated Data Collection Interface

How to play the crawler API? First fix the proxy IP this fate

What do you fear most about data collection? It's not that the code can't be written, it's that the IP will be blocked in just two minutes! Just like playing a game was kicked out of the server, you say angry? At this time we have to sacrifice the proxy IP this magic weapon. We do not organize the theory of those imaginary head, directly on the dry goods.

How did proxy IPs become oxygen tanks for crawlers?

For example, if you visit a certain website 100 times a day with your own broadband, who will block you if they don't? But what if you change your IP address every time you visit? This is like playing "face", the site can not recognize who you are. There are many proxy IP service providers in the market, but we recommend our own!ipipgos dynamic IP pool, the measured survival rate can go up to 98%, much more stable than some claimed big manufacturers.

 Python example - IP rotation with ipipgo
import requests

def crawl_with_ipipgo(url):
    proxies = {
        "http": "http://username:password@gateway.ipipgo.com:9020",
        "https": "http://username:password@gateway.ipipgo.com:9020"
    }
    for _ in range(10).
        response = requests.get(url, proxies=proxies)
        print(f"{_+1}th request status code:", response.status_code)

What are the hard metrics to look for when choosing a proxy IP?

Don't just look at the price, these three parameters are the most important:

① Degree of anonymity:High stash to hide the real IP
② Speed of response:Less than 800ms is considered passable
③ Failure to retry:Don't wait for manual switching

ipipgo has done a pretty solid job in this area. Their IP pool automatically updates 30% addresses every hour, which is especially suitable for old guys who need to run missions for a long time.

API Integration Practical Manual

Three steps to access ipipgo using Node.js as an example:

// Configure the proxy middleware
const tunnel = require('tunnel');
const agent = tunnel.httpsOverHttp({
  proxy: {
    host: 'gateway.ipipgo.com',
    proxy: { host: 'gateway.ipipgo.com', port: 9020,
    proxyAuth: 'username:password'
  }
}).

// Make the request with agent
axios.get('https://target.com', {
  httpsAgent: agent,
  timeout: 5000
})

Pay attention to setting the timeout! If you don't get a response in more than 5 seconds, just give up and don't hang on to an IP.

QA First Aid Kit

Q: What should I do if I always encounter CAPTCHA?
A: turn ipipgo's geo-location function on, try to use the IP segment where the target website is located, can reduce the probability of triggering verification

Q: Will it conflict to have more than one crawler on at the same time?
A: in ipipgo background to create different channels, to each crawler to allocate an independent proxy line, pro-test open 20 threads do not lag!

Q: Will the blocked IP be used again?
A: Their home system will automatically mark abnormal IP, 12 hours will not be assigned twice, this mechanism than many counterparts conscience!

Tell the truth.

Proxy IP this thing, three parts rely on technology and seven parts rely on resources. Some small workshops IP pool on a few thousand addresses back and forth, it is better to build their own proxy server. But like ipipgo, which has its own server room, can ensure that the IP resource pool is continuously updated. Recently they got a new feature--Request Frequency AdaptationThe system automatically adjusts the speed according to the response of the target site, this is especially friendly to newbies.

Finally remind you, don't buy those cheap static IPs sold one by one, now a little bit of protection of the site are staring at high-frequency access to the fixed IP seal, dynamic IP pool is the king of the road. The next time you encounter anti-climbing do not rush to change the code, first check the proxy IP is not the time to change.