Indeed Job Crawler: Job Data Collection Solution

First, why is it so hard to get Indeed recruitment data?

The biggest headache of catching Indeed's job postings is theIP blockedI'm not sure if you're a fan of this site. This website is like a door god, catching the same IP wildly scratching to give you a blacklist. Last month, my colleagues do not believe in evil, with their own computers even climbed 3 hours, the results of the entire office network can not login Indeed, the boss almost smashed his monitor...

Here's a real-world data comparison:

(Note: the test environment for the daily collection of 50,000 data volume)

Second, how to choose the proxy IP not to step on the pit?

Proxy IP on the market with Taobao 9.9 shipping data line like, look at the cheap with up to death. Choose the proxy to recognize three hard indicators:

1. Sufficient number of IPs: With 10 million IP pools like ipipgo, the site won't even recognize it with a new vest for every request!
2. Switching speed is fast enough: It's better to change IPs automatically in milliseconds instead of waiting for half a minute for the whole IP change.
3. geographically accurateFor example, if you're trying to capture a US post, and you're assigned an Indian IP, you're immediately exposed.

I have used a certain agent before, the IP is quite a lot, but once the job information is mixed with Bengali, then I realized that their IP positioning is not accurate at all. After I switched to ipipgo, I can specify the city level IP location, and the localized jobs are very stable.

Third, hand to teach you to ride the collection system

Here's a Python example that uses the requests library + ipipgo to implement smart IP switching:

import requests
from itertools import cycle

 List of proxies from ipipgo (remember to replace the API key with your own)
PROXY_LIST = [
    "http://user:pass@gateway.ipipgo.com:8000",
    "http://user:pass@gateway.ipipgo.com:8001".
    ... More proxy nodes
]
proxy_pool = cycle(PROXY_LIST)

def get_jobs(keyword):
    for _ in range(3): fail retry 3 times
        proxy = next(proxy_pool)
        try.
            resp = requests.get(
                f "https://www.indeed.com/jobs?q={keyword}",
                proxies={"http": proxy, "https": proxy}, timeout=10
                timeout=10
            )
             Parsing the page code...
            return data
        except Exception as e.
            print(f "Failed to crawl with {proxy}, switching IPs automatically...")
    return None

Watch out for two potholes:
1. request frequency is not too hard, it is recommended that every request 3-5 times on the initiative to change the IP
2. add a random delay of 0.5-2 seconds to make it look like a real person is doing it.

IV. Frequently Asked Questions QA

Q：Why was I blocked even though I used a proxy IP?
A: 80% of them are using shared IP pool, too many people use the same segment. ipipgo's exclusive IP pool can avoid this problem, each user has a separate IP segment.

Q: What should I do if my proxy IP responds slowly?
A: Choose a proxy service that supports HTTP2 protocol, like ipipgo's high-end nodes with dedicated line transmission, which is more than 3 times faster than ordinary proxies.

Q: What if I need to grab a global position?
A: In the background of ipipgo directly select the multinational IP hybrid mode, will automatically assign the corresponding country's export IP, pro-test to catch the Australian post with the local IP success rate skyrocketed.

V. Complete list of maintenance techniques

1. IP Health Check: Automatically test IP availability in the early hours of each day, eliminating failed nodes
2. traffic equalizationDon't use a single IP for a long time, set a daily traffic limit for a single IP.
3. anomaly monitoring: When 5 consecutive IPs are invalidated, the SMS alarm is triggered immediately.
4. UA camouflage: Work with ipipgo's IP replacement to randomly change User-Agents each time.

Finally said a trick: the ipipgo API access to the crawler scheduling system, real-time response speed according to automatically switch the optimal node. Our project team with this set of programs, has been running a stable 7 months did not turn over, every day jerking a million pieces of data to play with.