Yad2 Crawler: Israel Real Estate Data Collection

Hands-on with proxy IP to catch Yad2 real estate data

Recently, a lot of friends doing overseas real estate analysis are asking how to capture the data of Yad2, the largest real estate platform in Israel, in a stable manner. Today, let's talk a little bit of real, teach you how to use a proxy IP to avoid anti-climbing, smooth and smooth to get the data in hand.

Why do I have to use a proxy IP?

Yad2 has a feature that is very sensitive to the frequency of visits. Last year, a buddy used his own IP to grab data for three days in a row, and the result was directly beingIt's been closed for a whole month.The most important thing is that they will limit the display content according to the IP address. What's even more troublesome is that they will also limit the display content according to the IP address, if you don't use the local IP address, some keywords will not be displayed at all.

That's when it's time to rely onResidential agent for ipipgoIt's a good idea. Their family has 3000+ local IP resources in Israel, the real test each IP can be used for 5-7 hours without turning over. The most critical thing is that these IPs are real home broadband, more than one grade more reliable than the server room IP.

Three Steps to Real-World Configuration

Here's an example of how to quickly deploy an agent in Python:

import requests

 Proxy information from ipipgo
proxy = {
    'http': 'http://用户名:密码@il.ipipgo.com:9020',
    'https': 'http://用户名:密码@il.ipipgo.com:9020'
}

 Request headers with random UA
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124'
}

resp = requests.get('https://www.yad2.co.il/realestate/rent',
                   proxies=proxy,
                   headers=headers, timeout=15)
                   timeout=15)

pay attention toRandomly switch UA for each request, don't use those crappy UA libraries. It is recommended to prepare your own 20-30 common browser UA rotation.

The tawdry operation of avoiding detection

Here are a few practical lessons to share:

Focusing on frequency control, it is recommended thatNo more than 3 requests per minute from a single IPThe API of ipipgo supports automatic IP switching, and it is recommended to set the IP to be changed every 50 requests, so that it is stable and does not waste resources.

Frequently Asked Questions

Q: Is it okay to use a free proxy?
A: Never! I've tried about ten free proxies, but they are either slow or have a short survival time. Once with a free IP to catch data, the results are returned to the false data, white toss a night.

Q: How many IPs are needed to be sufficient?
A: According to 8 hours a day, 50-80 quality IPs are enough. ipipgo's package has a "Middle East Exclusive Package", which is the most cost-effective way to catch Yad2.

Q: What should I do if I encounter a CAPTCHA?
A: Two options: either on the coding platform (high cost) or on theSmart Agents for ipipgo, some of their IP segments come with CAPTCHA capability.

How to choose agency services

You have to look at a few hard indicators to choose an agency service:

• IP survival time > 4 hours
• Single IP cost <$0.3/hour
• Dedicated national/city-level IP pools available

This is something that ipipgo does really well, especially with theirReal-time monitoring of IP availabilityRecently, I found out that they also have a "Cold Country Speedy Opening" service, which allows you to open an exclusive channel within 2 hours in a niche area like Israel.

Lastly, I'd like to remind you that data capture is a matter of time and effort. Don't try to be fast, set a good random delay (1-3 seconds), with a good quality proxy, in order to get long-term stable data. Once I was lazy and didn't set the delay, the result was that I was blocked more than 20 IPs in one night, and I lost a lot of money...