Book dataset: Publication Metadata CSV

When book datasets meet proxy IPs: the pitfalls you must know about

The old iron of data collection know how difficult it is to get a complete CSV of publication metadata. The website anti-climbing mechanism is getting more and more ruthless, not moving to block IP. last week I helped publishers to do data collection, just grabbed 300 records IP was pulled black, so angry that I almost fell on the keyboard.

It's time to move outproxy IPThis big killer is up. The principle is simple:Rotate requests with different IPsThe first thing you need to do is to make the site think that it is a normal user visit. But in practice, some details do not pay attention to the car as usual.

Practical: using proxy IP to collect book metadata

Take a real case: to catch a book site'sISBN number + title + publisher + publication dateThese four fields. Straight to the Python code:

import requests
from bs4 import BeautifulSoup

proxies = {
    'http': 'http://ipipgo-12345:password@gateway.ipipgo.com:9020',
    'https': 'http://ipipgo-12345:password@gateway.ipipgo.com:9020'
}

response = requests.get('destination URL', proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')
 Subsequent parsing of the field code...

Here's one.lesson learned through blood and tearsDon't use free proxies! I used a certain free proxy before to save myself some trouble, and the result:

Why do you recommend ipipgo?

The in-house team has tested 7 agency service providers in the market and finally locked in the three hardcore advantages of ipipgo:

1. exclusive IP pool: Individual IP segments for each account to avoid "clashing" with other users.
2. Success Guarantee: Commitment to 99.5%+ request success rate
3. The protocol supports full: HTTP/HTTPS/Socks5 Full Compatibility

Especially theirIntelligent RoutingThe function can automatically select the fastest node. Last time when collecting foreign language book data, the speed of switching nodes is more than 3 times faster than manual.

Frequently Asked Questions QA

Q: What is the appropriate acquisition frequency setting?
A: It is recommended that a single IP does not exceed 15 requests per minute, with ipipgo's rotation strategy can be mentioned 30 times per minute

Q: What should I do if I encounter a CAPTCHA?
A: ipipgo's high stash of IP can reduce the probability of CAPTCHA triggering, really encountered when it is recommended to: 1) reduce the collection speed 2) replace the IP segment

Q: What do I need to know about data storage?
A: It is recommended that the field containcollect a timestamprespond in singingUsing IPTwo columns to facilitate subsequent troubleshooting

One final rant: doing data collection is like fighting a guerrilla war.Flexible IP switching + control of request cadenceThat's the way to go. Use a good ipipgo this kind of professional tools, can save at least 50% tossing time. Recently, their family is doing activities, new users to send 10G traffic package, the need of the old iron may try.