Track vulnerabilities like this one.
Sign up free to get alerted when software you use is affected.
4.3
Scrapy Fails to Switch Proxy When Redirected to Different Scheme
published 2 years ago
Summary
Scrapy's redirect feature doesn't switch between HTTP and HTTPS proxy settings as it should. This could be a security risk if you have different proxy settings for each scheme. To fix this, update to Scrapy 2.11.2 or create custom middlewares to handle redirects correctly.
What to do
- Update scrapy developers scrapy to version 2.11.2.
Affected software
| Ecosystem | Vendor | Product | Affected versions |
|---|---|---|---|
| pip | scrapy developers | scrapy |
< 2.11.2 Fix: upgrade to 2.11.2
|
Original advisory text
Scrapy's redirects ignoring scheme-specific proxy settings
### Impact
When using system proxy settings, which are scheme-specific (i.e. specific to `http://` or `https://` URLs), Scrapy was not accounting for scheme changes during redirects.
For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around.
If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue.
### Patches
Upgrade to Scrapy 2.11.2.
### Workarounds
Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) and the `HttpProxyMiddleware` middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.
### References
This security issue was reported by @redapple at https://github.com/scrapy/scrapy/issues/767.
When using system proxy settings, which are scheme-specific (i.e. specific to `http://` or `https://` URLs), Scrapy was not accounting for scheme changes during redirects.
For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around.
If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue.
### Patches
Upgrade to Scrapy 2.11.2.
### Workarounds
Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) and the `HttpProxyMiddleware` middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.
### References
This security issue was reported by @redapple at https://github.com/scrapy/scrapy/issues/767.
Internet-facing
60 days
Internal
At next upgrade
- Not known to be exploited
- Needs hands-on effort to exploit (estimated)
- Gives an attacker partial control (estimated)
Timeline
Published14 May 2024
Updated6 Mar 2026
First seen6 Mar 2026
Sources
GHSA-jm3v-qxmh-hxwv · GHSA
Track software like this
Free during beta