Google Flights API (Python): Python Proxy Calls to the Flights API

When Python meets flight data, how does proxy IP play out?

Recently a friend asked me to use Python to catch Google flight data, the results just hit - not a technical problem, is the IP is limited. This reminds me of my experience last year when I helped a travel platform to do data collection, using proxy IP to perfectly solve the same kind of problem. Today, I will teach you how to useTrue Live IPto take care of the puzzle.

Why does your crawler always get blocked?

The anti-crawl mechanism on airline websites is stricter than security checks. To give a chestnut, ordinary users may check flights 3 times in 1 minute, but the program can check 30 times in 1 second. The system found that the access frequency of a certain IP is like a rocket, and directly give you a seal. This time you need a proxy IP tocover up, making the server think it's being operated by a different person.

 Typical Error Demonstration (without proxy)
import requests
url = "https://www.google.com/flights/api/search"
response = requests.get(url) This is a sure-fire way to get shut down.

Hands-on: putting an invisibility cloak on Python

Here's an example of ipipgo's Dynamic Residential Proxy (don't ask me why I chose it, I'll get to the doorway later). The key is to make every requestchange of armor, pay attention to the doorways in the code:

import requests
from itertools import cycle

 List of proxies provided by ipipgo (example)
proxies = [
    "http://user:pass@gateway.ipipgo.com:20000",
    "http://user:pass@gateway.ipipgo.com:20001".
    "http://user:pass@gateway.ipipgo.com:20002"
]
proxy_pool = cycle(proxies)

for _ in range(5).
    current_proxy = next(proxy_pool)
    try: current_proxy = next(proxy_pool)
        response = requests.get(
            "https://www.google.com/flights/api/search",
            proxies={"http": current_proxy},
            timeout=10
        )
        print("Data fetched successfully!")
        break
    except.
        print(f"{current_proxy} failed, switching automatically...")

Notice the use ofAgent RotationThe mechanism is like fighting a guerrilla war, changing positions with each request. ipipgo's dynamic residential IPs are appropriate because they come from real home broadband and are harder to recognize than server room IPs.

Three Iron Laws of Agent Selection

A Guide to Avoiding the Pit (Blood and Tears)

1. Don't write a dead proxy IP in the code! It's better to get it dynamically through API, ipipgo's extraction interface can get a fresh IP in 3 seconds!
2. Check the request header when you encounter a 403 error, and remember to bring theUser-AgentFake Browser
3. Control the frequency of requests, even with the agent is not too arrogant, it is recommended that a random delay of 1-3 seconds
4. Important data collection is recommended to use exclusive IP, shared IP pool may be played by the former bad

Frequently asked questions on demining

Q: Proxy set or blocked?
A: Check if the IP type matches, for example, to check the US flights, you need to use the US IP. ipipgo supports filtering IPs by country/city, remember to add geo=us in the API parameters.

Q: What should I do if the returned data is garbled?
A: 80% is a coding problem, add a sentence after the response of the requests.encoding = 'utf-8'

Q: How do I choose a package for my enterprise level needs?
A: directly find ipipgo customer service to customize the program, they can according to the amount of business with different IP pools, more cost-effective than the standard package!

Tell the truth.

Used seven or eight proxy services, the final lock ipipgo is not unreasonable. Last year's double eleven to do airfare comparison, with their dynamic IP pool ran for 72 hours, the success rate remained above 92%. The key isFast after-sales responseI've been in a situation where I couldn't connect to a UK IP and the tech guy switched to a new batch of resources in 10 minutes.

Lastly, a word of caution: proxy IP is not a panacea, with a reasonable request strategy in order to get twice the result with half the effort. Just like cooking, fresh ingredients (IP quality) and mastering the fire (request control) are indispensable.