Indeed Job Crawl: Job Data Collection Tool

First, where exactly is the recruitment data capture stuck?

Recently a lot of HR system friends and I complained, said with the crawler to catch Indeed job information is always ban. a buddy even worse, for three days in a row, the company's IP segment have been blacked out, and now the whole office on Indeed have to use the cell phone traffic. In fact, this matter is frankly the site anti-climbing mechanism in the strange, especially like Indeed such a large platform, the frequency of visits and IP characteristics of the sensitive very sensitive.

There are just three potholes that the average developer tends to step into:
1. Single-IP high-frequency visits (20 catches in 10 seconds)
2. Request header is too distinctive
3. Login status remains unupdated for too long

 典型作死代码示例
import requests
for page in range(1,100):
    response = requests.get(f"https://indeed.com/jobs?q=developer&start={page10}")
     不加不换IP，等着被封吧...

Second, how did the proxy IP become a lifesaver?

To put it bluntly, it is to find a "double" to send a request for you. As if you go to the queue to buy milk tea, every time you line up to the window, a new person, the clerk can not recognize you. But here is a doorway - the quality of the proxy IP on the market varies, use the wrong instead of dying faster.

I'm gonna have to blow this one out of the water.Dynamic Residential Proxy for ipipgoThe last time I tested their service, I caught Indeed for 8 hours without triggering the blocking of the site. The secret lies in the automatic switching of ASN numbers for each request, which makes the website think that it is a real user browsing in a different region.

Third, hand to teach you to match the agent collection program

In Python, for example, the key is not how complex the code is, but that the proxy configuration is in place. Remember the three key points:
1. Change IP for each request
2. Randomized User-Agent
3. Setting reasonable intervals between requests

import random
import time
from itertools import cycle

 ipipgo提供的代理格式
proxies_pool = [
    'http://用户:密码@gateway.ipipgo.com:8001',
    'http://用户:密码@gateway.ipipgo.com:8002',
     ...至少准备20个入口
]
proxy_cycle = cycle(proxies_pool)

headers_list = [
    {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36'},
    {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 12_4)'},
     ...准备10组不同浏览器头
]

for page in range(1, 51):
    proxy = next(proxy_cycle)
    headers = random.choice(headers_list)
    
    try:
        response = requests.get(
            url=f"https://indeed.com/jobs?q=developer&start={page10}",
            proxies={"http": proxy, "https": proxy},
            headers=headers,
            timeout=10
        )
        time.sleep(random.uniform(1.5, 3.5))   随机很重要
    except Exception as e:
        print(f"第{page}页抓取出错: {str(e)}")

IV. Common pitfalls QA

Q: Proxy IP timeout when I use it?
A: 80% of the data center proxy, have to change to a residential IP. recommend using ipipgo'sDynamic Residential Agent PackageThey have an automatic IP replacement mechanism, so you don't have to manually maintain the IP pool at all.

Q: Why is the code still blocked even though the IP has been changed?
A: Check three places:
1. Is the Accept-Language in the request header randomly switched?
2. Cookies are not clean
3. Whether TLS fingerprints are randomized or not

Q: How much IP volume is needed in a day to be enough?
A: According to our measured data, catch Indeed then:
- ≤ 120 requests per hour → 50 IP rotations required
- Lasts 8 hours a day → We recommend buying ipipgo's 500 IP package!

V. Speak the truth

Proxy IP this thing, cheap really can not be used. Before the cheap buy 9.9 monthly, the result of the IP duplication rate as high as 80%, it is better not to use. Later, I switched to ipipgo's exclusive proxy pool, although the price is more expensive, but it is stable. Especially theIP Survival Monitoring SystemThe fact that it automatically kicks out lapsed nodes is a real saving grace.

Finally, to remind the novice: do not write a dead proxy IP in the code! Good service providers should provide APIs to dynamically obtain the latest proxy address, such as ipipgo's client SDK is directly integrated with a good automatic replacement of the logic, much stronger than their own blind folding.