Proxies or proxy servers are intermediary servers that are positioned between users' devices and the rest of the internet. The proxy landscape is constantly changing, with new proxies coming online every day. This fact makes detecting proxies one of the hardest and most challenging problems to solve. Currently, detecting proxies involves checking if an IP address belongs to a known datacenter, comparing browser timezones with IP geolocation, and/or analyzing HTTP headers like X-Forwarded-For or Via for irregularities. Other methods involve using software services, reviewing server logs, and checking IP blocklists.

Proxy Detection Methods

Checking if an IP address belongs to a known datacenter

Checking if an IP address belongs to a known datacenter involves using WHOIS lookup tools or commercial IP geolocation services to identify the owner, specifically looking for hosting providers such as AWS or DigitalOcean rather than residential internet service providers (ISPs). Key indicators include the ISP name, ASN (Autonomous System Number) registration, and specialized "Usage Type" data often labeled as DCH (Data Center/Web Hosting/Transit) in geolocation and lookup tools.

For WHOIS lookup tools, use sites like ARIN, WhatIsMyIP.com, or command-line tools (whois <IP>). For IP geolocation services, providers like IPLocate.io offer APIs for automated detection. For ASNs, use a service like RIPEstat - a large-scale information service and open data platform.

Comparing browser timezones with IP geolocation

Comparing browser timezones with IP geolocation requires websites to read the timezone currently set on a device's operating system, look up the incoming IP address for its expected timezone in a geolocation database like MaxMind or IP2Location, then compare the timezone obtained from the browser with the timezone obtained from the geolocation database. If an IP address indicates that a user is in London (GMT), but the browser reports a timezone indicating that a user is in New York (EST), that user could be flagged as a proxy.

Analyzing HTTP headers

Analyzing HTTP headers is a simple and direct way for web servers to identify proxies. Intermediaries often inject specific metadata into the request to help the server understand the traffic's origin and/or path. Below are some commonly analyzed HTTP headers:

  • X-Forwarded-For - the de-facto standard. This HTTP header contains the original client's IP address and a list of any subsequent proxies it passed through.
  • Via - used by many proxies (like Squid or Apache) to signal their presence. This HTTP header includes the protocol version and the hostname or IP of the proxy.
  • Forwarded - the modern, standardized version of X-Forwarded-For. This HTTP header bundles the "for" (client IP), "by" (proxy IP), "host", and "proto" (HTTP/HTTPS) into one field.
  • Proxy-Connection - an older, non-standard header used by some clients to manage keep-alive sessions when routing through a proxy.
  • X-Real-IP - similar to X-Forwarded-For, often used by Nginx or other load balancers to pass the original client's IP to the backend server.

Analyzing HTTP headers applies mostly to Transparent (Level 3) proxies and Anonymous (Level 2) proxies. Elite / High Anonymity (Level 1) proxies strip all proxy-specific headers and other information that could cause requests to be identified as coming from a proxy. Elite / High Anonymity (Level 1) proxies often go undetected using HTTP header analysis.

Using software services

Software services and APIs are the industry standard for real-time proxy detection, especially for identifying Elite (Level 1) proxies or residential proxies. These services maintain massive, daily-updated databases of IP addresses classified by risk and connection type. Professional services like IPQualityScore, IPinfo, and MaxMind provide a "risk score" or "fraud score" for any incoming IP address by checking it against several data points:

  • ASN Classification - identifies if an IP address belongs to a residential ISP (low risk) or data center like AWS or DigitalOcean (high risk).
  • Connection Type - flags whether an IP address is a known VPN exit node, Tor relay, or public proxy.
  • Residential - identifies an IP address as belonging to real home users but are being used by proxy networks.
  • Reporting - flags an IP address for malicious activity like spamming, web scraping, or failed login attempts.

Software services use proprietary data collection methods to build their databases, including direct subscriptions to major VPNs like NordVPN or ExpressVPN to map their IP ranges, setting up honeypots to identify new proxy servers as they come online, and infrastructure scans that continuously scan the internet for open ports (like 1080 or 3128) that indicate a proxy server is running on a specific host.

Blacklist Domains is a software service that brings a unique approach to proxy detection, maintaining a database of reverse DNS names and performing analysis on those names, then comparing the reverse DNS name of a given IP address against the internal database of reverse DNS names. Blacklist domains is good for any kind of proxy detection, including residential proxy detection and Elite / High Anonymity (Level 1) proxy detection. Detect a proxy today.

Reviewing server logs

Reviewing server logs involves identifying specific headers (X-Forwarded-For, Via, Forwarded) and spotting suspicious traffic patterns in access logs.

Even for Elite proxies, server logs can reveal their presence through behavioral anomalies such as:

  • IP-to-User Ratio - a single IP address associated with many different session IDs, User-Agents, or account logins is likely to be a proxy or VPN exit node.
  • Data Center ASN - if an IP address in the server logs belongs to a hosting provider like AWS, DigitalOcean, or Hetzner rather than a residential ISP, the visitor is almost certainly using a proxy or VPN.
  • Geographical Mismatches - server logs that show a single user session jumping between drastically different geographical locations in a short timeframe can indicate a proxy being rotated.
  • High Latency / Duration - abnormally long connection durations or high Time to First Byte (TTFB) can indicate the extra hop introduced by a proxy server.

Industry standard tools for parsing and visualizing proxy-related data include ELK Stack (Elasticsearch, Logstash, Kibana), GoAccess, and Splunk.

Checking IP blocklists

Blocklists (DNSBLs, RBLs) are databases that track IP addresses and flag connections from known "risky" sources like VPNs, Tor nodes, and data center proxies.

Checking IP blocklists involves getting the IP of a device when it connects to a server, and checking the IP against a blocklist. If a match is found, the connection is flagged or denied.

Blocklists can be static or dynamic: static blocklists focus on persistent threats, while dynamic blocklists update in real-time to catch rapidly rotating proxies.

Conclusion

Detecting proxies is one of the hardest and most challenging problems to solve, and proxy detection solutions must evolve to keep pace. Current methods for proxy detection outside of software services are difficult to implement and tenuous. Even with software solutions, proxy users are still confident and getting away with illicit activities.

At Blacklist Domains, we are confident in our ability to keep ahead of the proxy curve. Our proxy detection solution is simple and robust, and keeps criminal proxy users on their toes. Try our proxy detection today.