Overview

Scrapy's built-in HttpProxyMiddleware can route a request through one HTTP proxy, but production scraping with Smartproxy usually needs two things: a residential pool for geo-targeted or ban-resistant requests, and a datacenter pool for high-throughput requests. This tutorial shows how to add a custom downloader middleware that selects a Smartproxy endpoint per request, rotates between residential and datacenter pools, and retries common proxy errors without hard-coding credentials in your spider.

Smartproxy supports HTTP and SOCKS5; this guide uses HTTP/HTTPS proxy endpoints because Scrapy's built-in proxy middleware expects HTTP proxy URLs.

Prerequisites

A Linux machine or container with Python 3.10 or newer.
A Smartproxy account with residential and/or datacenter proxy credentials.
Endpoint details from the Smartproxy dashboard: proxy host, port, username, and password for each pool you plan to use.
pip and venv available.

Steps

Step 1 — Create a virtual environment and install Scrapy

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install scrapy python-dotenv

Step 2 — Store Smartproxy credentials as environment variables

Do not commit credentials to your repository.

export SMARTPROXY_RES_HOST='your-residential-host'
export SMARTPROXY_RES_PORT='your-residential-port'
export SMARTPROXY_RES_USERNAME='your-residential-username'
export SMARTPROXY_RES_PASSWORD='your-residential-password'

export SMARTPROXY_DC_HOST='your-datacenter-host'
export SMARTPROXY_DC_PORT='your-datacenter-port'
export SMARTPROXY_DC_USERNAME='your-datacenter-username'
export SMARTPROXY_DC_PASSWORD='your-datacenter-password'

If you prefer a .env file, load it before Scrapy starts. Keep the file out of version control.

Step 3 — Create the Scrapy project

scrapy startproject smartproxy_scrapy
cd smartproxy_scrapy
scrapy genspider example example.com

Step 4 — Add a Smartproxy rotation middleware

Create smartproxy_scrapy/middlewares.py or append to the generated file. The middleware sets request.meta['proxy'] before Scrapy's HttpProxyMiddleware runs.

import os
from urllib.parse import quote

from scrapy import signals
from scrapy.exceptions import NotConfigured
from scrapy.utils.project import get_project_settings


class SmartproxyRotationMiddleware:
    '''Route Scrapy requests through Smartproxy residential or datacenter endpoints.'''

    def __init__(self, settings):
        self.pools = {
            'residential': {
                'host': settings.get('SMARTPROXY_RES_HOST'),
                'port': settings.get('SMARTPROXY_RES_PORT'),
                'username': settings.get('SMARTPROXY_RES_USERNAME'),
                'password': settings.get('SMARTPROXY_RES_PASSWORD'),
            },
            'datacenter': {
                'host': settings.get('SMARTPROXY_DC_HOST'),
                'port': settings.get('SMARTPROXY_DC_PORT'),
                'username': settings.get('SMARTPROXY_DC_USERNAME'),
                'password': settings.get('SMARTPROXY_DC_PASSWORD'),
            },
        }

        if not any(pool['host'] and pool['port'] for pool in self.pools.values()):
            raise NotConfigured('Set at least one Smartproxy pool host and port.')

    @classmethod
    def from_crawler(cls, crawler):
        middleware = cls(crawler.settings)
        crawler.signals.connect(middleware.spider_opened, signal=signals.spider_opened)
        return middleware

    def spider_opened(self, spider):
        spider.logger.info('Smartproxy rotation middleware enabled')

    def process_request(self, request, spider):
        pool_name = request.meta.get('proxy_pool', 'residential')
        pool = self.pools.get(pool_name)

        if not pool or not pool['host'] or not pool['port']:
            return None

        host = pool['host']
        port = pool['port']
        username = quote(pool['username'] or '', safe='')
        password = quote(pool['password'] or '', safe='')

        if username and password:
            proxy_url = f'http://{username}:{password}@{host}:{port}'
        else:
            proxy_url = f'http://{host}:{port}'

        request.meta['proxy'] = proxy_url
        request.meta['smartproxy_pool'] = pool_name
        return None

Step 5 — Enable the middleware before Scrapy's proxy middleware

Edit smartproxy_scrapy/settings.py.

DOWNLOADER_MIDDLEWARES = {
    'smartproxy_scrapy.middlewares.SmartproxyRotationMiddleware': 749,
}

HTTPPROXY_ENABLED = True
DOWNLOAD_TIMEOUT = 30
RETRY_HTTP_CODES = [403, 407, 429, 500, 502, 503, 504]
RETRY_TIMES = 2
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 4

Scrapy's built-in HttpProxyMiddleware is ordered at 750, so 749 ensures your middleware sets the proxy URL first.

Step 6 — Choose residential or datacenter per request

Use meta={'proxy_pool': 'residential'} for geo-targeted or high-anonymity pages. Use meta={'proxy_pool': 'datacenter'} for high-throughput, stable pages.

import scrapy


class SmartproxyDemoSpider(scrapy.Spider):
    name = 'smartproxy_demo'
    start_urls = ['https://httpbin.org/ip']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={'proxy_pool': 'residential'},
                callback=self.parse,
                dont_filter=True,
            )
            yield scrapy.Request(
                url,
                meta={'proxy_pool': 'datacenter'},
                callback=self.parse,
                dont_filter=True,
            )

    def parse(self, response):
        yield {
            'pool': response.meta.get('smartproxy_pool'),
            'status': response.status,
            'origin': response.text.strip(),
        }

Step 7 — Add fallback retries between pools

A 403 or 429 from the target site often means the current IP is blocked. Add this method inside the SmartproxyRotationMiddleware class so a failed request retries once with the other Smartproxy pool.

    def process_response(self, request, response, spider):
        retryable = {403, 429}
        already_switched = request.meta.get('smartproxy_pool_retry', False)

        if response.status in retryable and not already_switched:
            current = request.meta.get('proxy_pool', 'residential')
            alternate = 'datacenter' if current == 'residential' else 'residential'

            new_meta = request.meta.copy()
            new_meta['proxy_pool'] = alternate
            new_meta['smartproxy_pool_retry'] = True

            spider.logger.warning(
                'Retrying %s with %s pool after HTTP %s',
                request.url,
                alternate,
                response.status,
            )
            return request.replace(meta=new_meta, dont_filter=True)

        return response

If you use this fallback, keep RETRY_TIMES low so blocked requests do not loop indefinitely.

Step 8 — Run the spider and save results

scrapy crawl smartproxy_demo -O smartproxy_results.json

Step 9 — Verify the proxy exit IP with cURL

Use the same Smartproxy HTTP endpoint outside Scrapy to confirm credentials and connectivity.

curl --proxy 'http://USERNAME:PASSWORD@HOST:PORT' https://httpbin.org/ip

Replace USERNAME, PASSWORD, HOST, and PORT with the values for the pool you are testing.

Troubleshooting

407 Proxy Authentication Required: Check the username and password, make sure they are URL-encoded, and confirm you are using the correct host and port for the selected pool.
Tunnel connection failed or TLS errors: Confirm the Smartproxy endpoint supports HTTPS targets through HTTP CONNECT. Test it first with the cURL command in Step 9.
Scrapy ignores the proxy: Confirm SmartproxyRotationMiddleware is set below 750 in DOWNLOADER_MIDDLEWARES, and that HTTPPROXY_ENABLED is True.
Frequent 403 or 429: Lower CONCURRENT_REQUESTS_PER_DOMAIN, add DOWNLOAD_DELAY, switch to residential proxies, and rotate user agents only where the target permits it.
Slow responses: Datacenter proxies are usually better for raw throughput; residential proxies are better for geo-targeted requests and sites that block datacenter IP ranges.
Credentials contain @, :, or /: Keep the urllib.parse.quote calls in the middleware. Quoting prevents malformed proxy URLs.

Summary

You now have a Scrapy project that routes requests through Smartproxy residential and datacenter HTTP proxies on Linux. The middleware selects a pool per request, keeps credentials in environment variables, and can fall back to the other pool when a target blocks an IP. Start with low concurrency, verify your exit IPs with cURL, and increase CONCURRENT_REQUESTS_PER_DOMAIN only after the crawl behaves reliably.