Recruitment Data Platform: Indeed/Glassdoor Data Source

The biggest headache of working with recruitment data

Old timers who do data crawling for recruitment platforms know that sites like Indeed and GlassdoorCrawler-proofing is stricter than thief-proofing.. The day before yesterday just wrote the script, today prompted a 403 error, IP directly be blacklisted. What's even better is their CAPTCHA system, sometimes even the human eye can't tell whether it's a letter or an abstract painting.

A customer doing headhunting system complained to me that their team had to manually change the IP address seven or eight times a day. The worst time is 3:00 a.m. was woken up by the boss phone, because the crawler program is stuck, resulting in the next day no data available. This kind of bitter experience, engaged in people know how devastating.

Proxy IPs are the real deal.

Want to steadily capture recruiting data with a focus onMasquerading as normal user accessThe secret is threefold. Here to say a real case: a HR SaaS company with ipipgo's residential agent service, the data acquisition success rate directly from 37% soared to 92%. secret on three:

Be careful with these two parameter settings when using ipipgo:Don't request less than 5 seconds between requests(math.) genusDon't use a single IP for more than 30 minutesThe first thing you need to do is to get your hands on a script and go to a hotpot. I've seen people run off to fondue with scripts on, only to come back and find their IP blocked, so don't make this kind of low-level mistake.

Hands-on configuration

Here's a chestnut in Python, remember to fill in the authentication information provided by ipipgo:

proxies = {
  "http": "http://用户名:密码@gateway.ipipgo.com:端口",
  "https": "http://用户名:密码@gateway.ipipgo.com:端口"
}
response = requests.get(url, proxies=proxies, timeout=10)

Focus on three pits:
1. Don't use free proxies, the speed is as slow as a snail, not to mention that the data may also be intercepted!
2. Don't fight hard when encountering CAPTCHA, you should go to the coding platform.
3. clear cookies regularly, don't let the website remember your crawler characteristics

Guidelines on demining of common problems

Q: Why do I still get blocked with a proxy IP?
A: Check if the IP switching frequency is not enough or the requests are too intensive. It is recommended to use ipipgo'sauto-rotation modeThe new IP will be used for every request.

Q: How much IP volume do I need to prepare to be sufficient?
A: 200-300 high-quality IPs are enough for 10,000 pieces of data per day. ipipgo's packages includeDynamic IP PoolNo need to maintain quantities yourself

Q: What should I do if I encounter Cloudflare protection?
A: Disguise the request header in its entirety, especially User-Agent and Referer. ipipgo'sBrowser FingerprintingYou can take care of these parameters automatically

Three Iron Rules for Choosing a Service Provider

1. Look at the IP purity: many agents' IPs have been labeled by job boards as early as ipipipgoResidential IP PoolUpdated three times a week
2. Measure the response speed: less than 800ms can only be used, do not believe those bragging about 200ms
3. Check the protocol support: must also support HTTP/HTTPS/SOCKS5, some of the old system only recognize specific protocols

One last bit of cold knowledge: the frequency limit for Glassdoor visits isBy stateThe. Using ipipgo to switch between IPs in different regions can pick up 30% more data than a fixed IP. These are our real test out of the experience, take it directly use don't be polite.