PowerShell Web Request: Invoke-WebRequest Tutorial

First, how does a PowerShell web request actually play?

Anyone who has ever used Windows knows that PowerShell is a gem of a tool. Today we're going to focus onInvoke-WebRequestThis command, especially in combination with proxy IP scenarios. For example, when you do data collection, you often encounter website anti-climbing mechanisms, and this time you have to rely on proxy IP to renew your life.

 The most basic usage looks like this:
$response = Invoke-WebRequest -Uri "http://example.com"
$response.Content

However, the light with this is easy to be the target site blocked IP, this time we have to pull out our killer -Proxy services for ipipgo. Their dynamic IP pools are updated quickly and are especially suitable for scenarios that require frequent IP switching.

II. Putting a proxy vest on the request

Hooking up a proxy to PowerShell is actually super simple, with the focus on the-ProxyThis parameter. There is a small pitfall to note here: if the proxy needs account password authentication, you have to use the-ProxyCredentialParameters.

 Example of proxy setup with password
$proxy = "http://123.45.67.89:8888"
$cred = New-Object System.Management.Automation.PSCredential ("username", ("password" | ConvertTo-SecureString -AsPlainText -Force))
Invoke-WebRequest -Uri "https://target.com" -Proxy $proxy -ProxyCredential $cred

If you find it troublesome to enter your password every time, you can tryAPI services for ipipgo. They provide proxy addresses with authentication parameters and just stuff them directly into the URI, eliminating the authentication step.

Third, the actual combat anti-IP blocking strategy

Here we share two code templates for real scenarios:

 Option 1: Randomly switch proxy IPs (with ipipgo's API)
$ipList = (Invoke-RestMethod "https://ipipgo.com/api/get_proxies").data
foreach ($page in 1..100) {
    $randomProxy = $ipList | Get-Random
    Invoke-WebRequest -Uri "https://target.com/page/$page" -Proxy $randomProxy
}

 Option 2: Automatic Exception Retry
try {
    $response = Invoke-WebRequest -Uri "https://敏感网站" -Proxy "http://ipipgo动态代理"
} catch {
    Write-Host "Triggered wind control, changing IP..."
    $newProxy = (Invoke-RestMethod "https://ipipgo.com/api/rotate_ip").proxy
    $response = Invoke-WebRequest -Uri "https://敏感网站" -Proxy $newProxy
}

IV. First aid kit for common problems

Q：Why does the agent return a 407 error when it works?
A: Ninety percent of the authentication information did not pass the right, check three points: 1) username and password there are no spaces 2) port number is not right 3) proxy protocol is http or https. ipipgo with ipipgo then suggest directly copy their background generated by the complete proxy address.

Q: How do I break the request timeout?
A: Add first-TimeoutSec 30参数延长等待时间。如果还不行，到ipipgo后台换个低的机房节点，他们家的国内BGP线路响应速度能控制在200ms以内。

Q: How can I tell if a proxy is in effect?
A: There's a tart operation - first visit ipipgo's verification interface without proxy, it will return the real IP; then visit with proxy and compare whether the two IPs are different or not.

V. The Complete Guide to Avoiding Pitfalls

1. When encountering SSL certificate errors, add-SkipCertificateCheckParameters (PowerShell 7+ only)
2. When you need to save a cookie, use the-SessionParameters to create a session object
3. Handling of large file downloads must be added-OutFileparameter to avoid memory overflow

 Download the file in the correct position
Invoke-WebRequest -Uri "http://大文件地址" -Proxy $proxy -OutFile "D:/downloads/file.zip"

Lastly, I'd like to apologize.ipipgo recently went live with a PowerShell-specific SDK!The first one is to use Invoke-WebRequest, which encapsulates the functions of automatic retry and intelligent IP switching, and saves much more effort than bare Invoke-WebRequest. New users register to send 5G traffic, enough to test scripts run half a month.