Proxy Rotation Best Practices: Avoid Bans and Maximize Success Rates ZentisLabs Blog

Rotating proxies is not hard. Rotating proxies well is surprisingly nuanced. The difference between a scraper that gets 95% success rates and one that gets banned after 200 requests often comes down to rotation strategy — when to rotate, how often, how to handle sessions, and how to detect when an IP has been flagged. This guide covers everything.

When to Rotate IPs

The most common mistake is rotating too aggressively or not enough. Here's the mental model:

Rotate per request when: you're making high-volume requests to the same domain and each request is independent (e.g., checking prices across thousands of SKUs).
Use sticky sessions when: you have multi-step workflows that require session continuity (login → navigate → extract). Switching IPs mid-session is an immediate red flag to most anti-bot systems.
Rotate on error when: you receive a 403, 429, or detect a CAPTCHA. Don't keep hammering a target from an IP that's already been flagged.
Rotate on time threshold when: using sticky sessions, set a maximum duration (10–30 minutes) and rotate even without errors — long-lived sessions from a single IP look suspicious.

Understanding Sticky Sessions

ZentisLabs (and most quality providers) support sticky sessions — a mechanism that pins your traffic to the same exit IP for a defined duration, even as the rotation pool changes around you.

python

import requests
import uuid

# Generate a unique session ID for each logical "user" session
def get_sticky_proxy(session_id: str, duration_minutes: int = 10) -> dict:
    """Returns proxy config that pins to the same IP for duration_minutes."""
    proxy_url = (
        f"http://USER:PASS_session-{session_id}"
        f"@gate.zentislabs.com:7777"
    )
    return {"http": proxy_url, "https": proxy_url}

# Independent tasks → rotating (new IP per request, no session ID)
rotating_proxy = {"http": "http://USER:PASS@gate.zentislabs.com:7777",
                  "https": "http://USER:PASS@gate.zentislabs.com:7777"}

# Multi-step workflow → sticky session (same IP for entire flow)
session_id = str(uuid.uuid4())[:8]  # e.g., "a3f8c91b"
sticky_proxy = get_sticky_proxy(session_id)

# Now do your multi-step flow:
session = requests.Session()
session.proxies = sticky_proxy

# Step 1: Load homepage (establishes cookies)
session.get("https://example.com")

# Step 2: Log in (same IP as step 1 — critical)
session.post("https://example.com/login", data={"user": "x", "pass": "y"})

# Step 3: Navigate to target page (still same IP)
result = session.get("https://example.com/protected-data")

# Same IP used throughout — session stays valid

Rotation Strategies

1. Round-Robin (Basic)

Cycle through a fixed list of proxies sequentially. Simple but predictable — if the pattern is detected, all IPs in your list get flagged simultaneously.

python

proxy_list = ["http://user:pass@ip1:port", "http://user:pass@ip2:port", ...]
proxy_index = 0

def get_next_proxy():
    global proxy_index
    proxy = proxy_list[proxy_index % len(proxy_list)]
    proxy_index += 1
    return proxy

2. Random Selection

Select randomly from your pool. Less predictable than round-robin, but you may hit the same IP multiple times in a row by chance.

python

import random

def get_random_proxy(proxy_list):
    return random.choice(proxy_list)

3. Gateway-Based Rotation (Recommended)

Use a rotation gateway like ZentisLabs — each request is automatically assigned a different IP from a pool of millions. No list management, no recycling, and the gateway handles IP health automatically.

python

# No list needed — the gateway handles rotation automatically
proxy = {"http": "http://USER:PASS@gate.zentislabs.com:7777",
         "https": "http://USER:PASS@gate.zentislabs.com:7777"}

# Every request automatically gets a different residential IP
for url in urls_to_scrape:
    r = requests.get(url, proxies=proxy, timeout=20)
    # Different IP used each time

4. Weighted Rotation

Track success rates per IP and weight selection toward IPs that have been working. Avoid IPs that recently triggered errors.

python

from collections import defaultdict
import random

class ProxyRotator:
    def __init__(self, proxies):
        self.proxies = proxies
        self.success_counts = defaultdict(int)
        self.failure_counts = defaultdict(int)
        self.cooldown = {}  # IPs cooling down after failures

    def get_proxy(self):
        available = [p for p in self.proxies if p not in self.cooldown]
        if not available:
            self.cooldown.clear()  # Reset cooldowns if all proxies cooling
            available = self.proxies
        
        # Weight by success rate
        weights = []
        for p in available:
            total = self.success_counts[p] + self.failure_counts[p]
            success_rate = self.success_counts[p] / total if total > 0 else 0.5
            weights.append(max(0.1, success_rate))
        
        return random.choices(available, weights=weights)[0]

    def record_success(self, proxy):
        self.success_counts[proxy] += 1

    def record_failure(self, proxy, cooldown_seconds=60):
        self.failure_counts[proxy] += 1
        import time
        self.cooldown[proxy] = time.time() + cooldown_seconds

Detecting When an IP Gets Banned

Don't just check HTTP status codes. Sophisticated sites return 200 with a CAPTCHA or honeypot response. Build a proper ban detector:

python

def is_blocked(response) -> bool:
    """Detect ban/block responses, including soft blocks that return 200."""
    
    # Hard blocks
    if response.status_code in (403, 429, 503):
        return True
    
    # Redirect to challenge page
    if "challenge" in response.url or "verify" in response.url:
        return True
    
    content = response.text.lower()
    
    # CAPTCHA indicators
    captcha_markers = [
        "g-recaptcha", "h-captcha", "cf-challenge",
        "please complete the security check",
        "verify you are human",
        "access denied",
        "unusual traffic",
    ]
    if any(marker in content for marker in captcha_markers):
        return True
    
    # Content anomaly: expected data missing
    # (customize for your target)
    if "expected-element-class" not in content and response.status_code == 200:
        return True
    
    return False


def scrape_with_rotation(url, max_retries=5):
    for attempt in range(max_retries):
        proxy = get_next_proxy()  # Your rotation method
        try:
            r = requests.get(url, proxies={"https": proxy}, timeout=20)
            if not is_blocked(r):
                return r
            print(f"Blocked on attempt {attempt+1}, rotating IP...")
        except Exception as e:
            print(f"Error: {e}, rotating IP...")
    
    raise Exception(f"Failed after {max_retries} attempts with IP rotation")

Rotation Patterns That Trigger Detection

Even with good proxies, these patterns will get you flagged:

Same request headers, different IP every time: A constant User-Agent across thousands of IPs is a fingerprint. Rotate your User-Agent alongside your IP.
Machine-speed timing: Requests at exactly 500ms intervals, every time, from thousands of IPs screams automation. Add random jitter: time.sleep(random.uniform(0.5, 3.0)).
IP switching mid-session: If you've started a session (loaded a page, set cookies) and then switch IPs, the session–IP mismatch is an immediate red flag.
Subnet clustering: Rotating between IPs in the same /24 subnet (256 IPs) means when one gets flagged, the whole subnet often does. Good rotation spreads across diverse subnets.
Missing common headers: Real browsers send 15–20 headers. If you send 3 headers from thousands of IPs, that's a fingerprint pattern that doesn't correlate with any real browser.

Header Rotation

python

import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
]

def get_headers(accept_language: str = "en-US,en;q=0.9") -> dict:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": accept_language,
        "Accept-Encoding": "gzip, deflate, br",
        "DNT": "1",
        "Connection": "keep-alive",
        "Upgrade-Insecure-Requests": "1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "none",
    }

# Rotate both proxy AND headers on each request
r = requests.get(url, proxies=get_proxy(), headers=get_headers(), timeout=20)

Using ZentisLabs API for Programmatic Rotation Control

python

import requests

# ZentisLabs API: manage rotation programmatically
ZentisLabs_API_KEY = "your_api_key"
BASE = "https://api.zentislabs.com/v1"

def get_pool_stats():
    """Check your current pool health and usage."""
    r = requests.get(f"{BASE}/pool/stats",
                     headers={"Authorization": f"Bearer {ZentisLabs_API_KEY}"})
    return r.json()

def set_rotation_mode(mode: str):
    """
    mode options:
    - "per_request": new IP for every request (default)
    - "per_session": sticky based on session ID in username
    - "per_minute": rotate every 60s regardless of requests
    """
    requests.patch(f"{BASE}/rotation/mode",
                   json={"mode": mode},
                   headers={"Authorization": f"Bearer {ZentisLabs_API_KEY}"})

def get_geo_specific_proxy(country: str, city: str = None) -> dict:
    """Get a proxy pinned to a specific geographic location."""
    target = f"country-{country}"
    if city:
        target += f"_city-{city.replace(' ', '').lower()}"
    proxy_url = f"http://USER:PASS_{target}@gate.zentislabs.com:7777"
    return {"http": proxy_url, "https": proxy_url}

⚡ Key rotation principle: Match your rotation strategy to your session structure. Independent requests → rotate freely. Multi-step flows → sticky sessions. Error → immediately rotate and back off. Time threshold → rotate proactively before detection occurs.

When to Rotate IPs

The most common mistake is rotating too aggressively or not enough. Here's the mental model:

Rotate per request when: you're making high-volume requests to the same domain and each request is independent (e.g., checking prices across thousands of SKUs).
Use sticky sessions when: you have multi-step workflows that require session continuity (login → navigate → extract). Switching IPs mid-session is an immediate red flag to most anti-bot systems.
Rotate on error when: you receive a 403, 429, or detect a CAPTCHA. Don't keep hammering a target from an IP that's already been flagged.
Rotate on time threshold when: using sticky sessions, set a maximum duration (10–30 minutes) and rotate even without errors — long-lived sessions from a single IP look suspicious.

Understanding Sticky Sessions

ZentisLabs (and most quality providers) support sticky sessions — a mechanism that pins your traffic to the same exit IP for a defined duration, even as the rotation pool changes around you.

python

import requests
import uuid

# Generate a unique session ID for each logical "user" session
def get_sticky_proxy(session_id: str, duration_minutes: int = 10) -> dict:
    """Returns proxy config that pins to the same IP for duration_minutes."""
    proxy_url = (
        f"http://USER:PASS_session-{session_id}"
        f"@gate.zentislabs.com:7777"
    )
    return {"http": proxy_url, "https": proxy_url}

# Independent tasks → rotating (new IP per request, no session ID)
rotating_proxy = {"http": "http://USER:PASS@gate.zentislabs.com:7777",
                  "https": "http://USER:PASS@gate.zentislabs.com:7777"}

# Multi-step workflow → sticky session (same IP for entire flow)
session_id = str(uuid.uuid4())[:8]  # e.g., "a3f8c91b"
sticky_proxy = get_sticky_proxy(session_id)

# Now do your multi-step flow:
session = requests.Session()
session.proxies = sticky_proxy

# Step 1: Load homepage (establishes cookies)
session.get("https://example.com")

# Step 2: Log in (same IP as step 1 — critical)
session.post("https://example.com/login", data={"user": "x", "pass": "y"})

# Step 3: Navigate to target page (still same IP)
result = session.get("https://example.com/protected-data")

# Same IP used throughout — session stays valid

Rotation Strategies

1. Round-Robin (Basic)

Cycle through a fixed list of proxies sequentially. Simple but predictable — if the pattern is detected, all IPs in your list get flagged simultaneously.

python

proxy_list = ["http://user:pass@ip1:port", "http://user:pass@ip2:port", ...]
proxy_index = 0

def get_next_proxy():
    global proxy_index
    proxy = proxy_list[proxy_index % len(proxy_list)]
    proxy_index += 1
    return proxy

2. Random Selection

Select randomly from your pool. Less predictable than round-robin, but you may hit the same IP multiple times in a row by chance.

python

import random

def get_random_proxy(proxy_list):
    return random.choice(proxy_list)

3. Gateway-Based Rotation (Recommended)

python

# No list needed — the gateway handles rotation automatically
proxy = {"http": "http://USER:PASS@gate.zentislabs.com:7777",
         "https": "http://USER:PASS@gate.zentislabs.com:7777"}

# Every request automatically gets a different residential IP
for url in urls_to_scrape:
    r = requests.get(url, proxies=proxy, timeout=20)
    # Different IP used each time

4. Weighted Rotation

Track success rates per IP and weight selection toward IPs that have been working. Avoid IPs that recently triggered errors.

python

from collections import defaultdict
import random

class ProxyRotator:
    def __init__(self, proxies):
        self.proxies = proxies
        self.success_counts = defaultdict(int)
        self.failure_counts = defaultdict(int)
        self.cooldown = {}  # IPs cooling down after failures

    def get_proxy(self):
        available = [p for p in self.proxies if p not in self.cooldown]
        if not available:
            self.cooldown.clear()  # Reset cooldowns if all proxies cooling
            available = self.proxies
        
        # Weight by success rate
        weights = []
        for p in available:
            total = self.success_counts[p] + self.failure_counts[p]
            success_rate = self.success_counts[p] / total if total > 0 else 0.5
            weights.append(max(0.1, success_rate))
        
        return random.choices(available, weights=weights)[0]

    def record_success(self, proxy):
        self.success_counts[proxy] += 1

    def record_failure(self, proxy, cooldown_seconds=60):
        self.failure_counts[proxy] += 1
        import time
        self.cooldown[proxy] = time.time() + cooldown_seconds

Detecting When an IP Gets Banned

Don't just check HTTP status codes. Sophisticated sites return 200 with a CAPTCHA or honeypot response. Build a proper ban detector:

python

def is_blocked(response) -> bool:
    """Detect ban/block responses, including soft blocks that return 200."""
    
    # Hard blocks
    if response.status_code in (403, 429, 503):
        return True
    
    # Redirect to challenge page
    if "challenge" in response.url or "verify" in response.url:
        return True
    
    content = response.text.lower()
    
    # CAPTCHA indicators
    captcha_markers = [
        "g-recaptcha", "h-captcha", "cf-challenge",
        "please complete the security check",
        "verify you are human",
        "access denied",
        "unusual traffic",
    ]
    if any(marker in content for marker in captcha_markers):
        return True
    
    # Content anomaly: expected data missing
    # (customize for your target)
    if "expected-element-class" not in content and response.status_code == 200:
        return True
    
    return False


def scrape_with_rotation(url, max_retries=5):
    for attempt in range(max_retries):
        proxy = get_next_proxy()  # Your rotation method
        try:
            r = requests.get(url, proxies={"https": proxy}, timeout=20)
            if not is_blocked(r):
                return r
            print(f"Blocked on attempt {attempt+1}, rotating IP...")
        except Exception as e:
            print(f"Error: {e}, rotating IP...")
    
    raise Exception(f"Failed after {max_retries} attempts with IP rotation")

Rotation Patterns That Trigger Detection

Even with good proxies, these patterns will get you flagged:

Same request headers, different IP every time: A constant User-Agent across thousands of IPs is a fingerprint. Rotate your User-Agent alongside your IP.
Machine-speed timing: Requests at exactly 500ms intervals, every time, from thousands of IPs screams automation. Add random jitter: time.sleep(random.uniform(0.5, 3.0)).
IP switching mid-session: If you've started a session (loaded a page, set cookies) and then switch IPs, the session–IP mismatch is an immediate red flag.
Subnet clustering: Rotating between IPs in the same /24 subnet (256 IPs) means when one gets flagged, the whole subnet often does. Good rotation spreads across diverse subnets.
Missing common headers: Real browsers send 15–20 headers. If you send 3 headers from thousands of IPs, that's a fingerprint pattern that doesn't correlate with any real browser.

Header Rotation

python

import random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
]

def get_headers(accept_language: str = "en-US,en;q=0.9") -> dict:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Language": accept_language,
        "Accept-Encoding": "gzip, deflate, br",
        "DNT": "1",
        "Connection": "keep-alive",
        "Upgrade-Insecure-Requests": "1",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "none",
    }

# Rotate both proxy AND headers on each request
r = requests.get(url, proxies=get_proxy(), headers=get_headers(), timeout=20)

Using ZentisLabs API for Programmatic Rotation Control

python

import requests

# ZentisLabs API: manage rotation programmatically
ZentisLabs_API_KEY = "your_api_key"
BASE = "https://api.zentislabs.com/v1"

def get_pool_stats():
    """Check your current pool health and usage."""
    r = requests.get(f"{BASE}/pool/stats",
                     headers={"Authorization": f"Bearer {ZentisLabs_API_KEY}"})
    return r.json()

def set_rotation_mode(mode: str):
    """
    mode options:
    - "per_request": new IP for every request (default)
    - "per_session": sticky based on session ID in username
    - "per_minute": rotate every 60s regardless of requests
    """
    requests.patch(f"{BASE}/rotation/mode",
                   json={"mode": mode},
                   headers={"Authorization": f"Bearer {ZentisLabs_API_KEY}"})

def get_geo_specific_proxy(country: str, city: str = None) -> dict:
    """Get a proxy pinned to a specific geographic location."""
    target = f"country-{country}"
    if city:
        target += f"_city-{city.replace(' ', '').lower()}"
    proxy_url = f"http://USER:PASS_{target}@gate.zentislabs.com:7777"
    return {"http": proxy_url, "https": proxy_url}

Proxy Rotation Best Practices: Avoid Bans and Maximize Success Rates

When to Rotate IPs

Understanding Sticky Sessions

Rotation Strategies

1. Round-Robin (Basic)

2. Random Selection

3. Gateway-Based Rotation (Recommended)

4. Weighted Rotation

Detecting When an IP Gets Banned

Rotation Patterns That Trigger Detection

Header Rotation

Using ZentisLabs API for Programmatic Rotation Control

Ready to get started?

Related Articles

How to Set Up a Rotating Proxy in Python, Node.js, and Bash (2025 Guide)

Best VPS for Web Scraping in 2025: Performance Benchmarks

Deploy Ollama on a VPS: Run LLMs Privately in 10 Minutes

Proxy Rotation Best Practices: Avoid Bans and Maximize Success Rates

When to Rotate IPs

Understanding Sticky Sessions

Rotation Strategies

1. Round-Robin (Basic)

2. Random Selection

3. Gateway-Based Rotation (Recommended)

4. Weighted Rotation

Detecting When an IP Gets Banned

Rotation Patterns That Trigger Detection

Header Rotation

Using ZentisLabs API for Programmatic Rotation Control

Ready to get started?

Related Articles

How to Set Up a Rotating Proxy in Python, Node.js, and Bash (2025 Guide)

Best VPS for Web Scraping in 2025: Performance Benchmarks

Deploy Ollama on a VPS: Run LLMs Privately in 10 Minutes