Web Scraping with Python: Complete Guide
Build a polite, resilient Python scraper: check for an API first, respect robots.txt, retry with backoff, parse defensively, and use Playwright only when JavaScript demands it.

My first serious scraper was a price monitor for a side project. It worked for two days, then a Saturday-morning run returned nothing: the site had renamed one CSS class, and my script had quietly written empty rows into the database. A week earlier the same script had fired requests in a tight loop and got my IP blocked for an afternoon. Both failures had the same root cause. I had written a script that assumed the site would never change and never mind being hit as fast as my laptop could go.
This guide is the version I wish I had started with: a scraper that is polite to the server, fails loudly when the page changes, and only reaches for a headless browser when it has to.
Quick takeaways
- Look for an API or embedded JSON before you parse HTML. It is faster to write and breaks less often.
- Identify yourself with a real
User-Agent, readrobots.txt, and rate-limit every request. Do not disguise the scraper. - Use a
requests.Sessionwith timeouts and automatic retries with backoff; never callrequests.getwithout a timeout. - Parse defensively and fail loudly: an empty result should raise an error, not write a blank row.
- Reach for Playwright only for pages that genuinely render their data with JavaScript.
Do I need to scrape HTML at all?
Open the browser's developer tools, switch to the Network tab, filter by Fetch/XHR and reload the page. Many "dynamic" pages load their data from a JSON endpoint, and calling that endpoint directly gives you structured data with no HTML parsing. Also search the page source for application/ld+json or __NEXT_DATA__; product and article pages often embed their data there.
Before writing any code, check three things: whether the site offers an official API, what its terms of service say about automated access, and what robots.txt allows. If the data is personal, remember that privacy law applies to scraped data too (KVKK in Turkey, GDPR in the EU), and never try to get around logins or paywalls.
How do I write a polite, resilient fetcher?
Put all network behaviour in one place so every request gets the same protections:
import time
from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
USER_AGENT = "price-watch/1.0 (contact: you@example.com)"
class PoliteFetcher:
def __init__(self, min_delay: float = 2.0):
self.min_delay = min_delay
self._last_request = 0.0
self._robots: dict[str, RobotFileParser] = {}
self.session = requests.Session()
self.session.headers["User-Agent"] = USER_AGENT
retry = Retry(
total=4,
backoff_factor=1.0,
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=("GET",),
respect_retry_after_header=True,
)
self.session.mount("https://", HTTPAdapter(max_retries=retry))
def _allowed(self, url: str) -> bool:
parts = urlparse(url)
base = f"{parts.scheme}://{parts.netloc}"
if base not in self._robots:
parser = RobotFileParser(f"{base}/robots.txt")
parser.read()
self._robots[base] = parser
return self._robots[base].can_fetch(USER_AGENT, url)
def get(self, url: str) -> str:
if not self._allowed(url):
raise PermissionError(f"robots.txt disallows {url}")
wait = self.min_delay - (time.monotonic() - self._last_request)
if wait > 0:
time.sleep(wait)
response = self.session.get(url, timeout=(5, 20))
self._last_request = time.monotonic()
response.raise_for_status()
return response.text
The details matter. The Retry object backs off exponentially on 429 and 5xx responses and honours the server's Retry-After header. The timeout is a (connect, read) pair, so a hung server cannot freeze your job. The delay is enforced between requests, not slept blindly, and robots.txt is fetched once per host. If a site publishes a Crawl-delay, set min_delay to at least that value.
How do I parse a page without breaking on every redesign?
BeautifulSoup is enough for static pages. The habit that saves you is to treat missing elements as errors you can see, instead of letting None flow into your data:
from dataclasses import dataclass
from urllib.parse import urljoin
from bs4 import BeautifulSoup
@dataclass
class Product:
name: str
price: float
url: str
def parse_products(html: str, base_url: str) -> list[Product]:
soup = BeautifulSoup(html, "html.parser")
cards = soup.select("article.product")
if not cards:
raise ValueError("no product cards found: the page layout probably changed")
products = []
for card in cards:
name = card.select_one("h2")
price = card.select_one("[data-price]")
link = card.select_one("a[href]")
if not (name and price and link):
continue # skip incomplete cards, but the empty-page check above still protects us
products.append(
Product(
name=name.get_text(strip=True),
price=float(price["data-price"]),
url=urljoin(base_url, link["href"]),
)
)
return products
Prefer attributes (data-price, itemprop, aria-label) over styling classes; they change less often. Run a sanity check after each scrape, for example "at least 10 items and none with a price of zero", and send an alert when it fails. A scraper that returns nothing is a bug, not a result.
When do I need Playwright or Selenium?
Only when the data does not exist in the HTML or in any JSON endpoint until JavaScript runs. A real browser is slower and heavier, so use it for the pages that need it. Playwright is my default today; its auto-waiting removes most of the sleep-and-hope code that Selenium scripts accumulate:
pip install playwright
playwright install chromium
from playwright.sync_api import sync_playwright
def render(url: str) -> str:
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(user_agent="price-watch/1.0 (contact: you@example.com)")
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("article.product")
html = page.content()
browser.close()
return html
Pass the returned HTML to the same parse_products function, so only fetching changes while parsing stays testable on saved files. Scrapy is the right tool once you crawl thousands of pages and need queues, pipelines and throttling built in; for a few endpoints, the fetcher above is simpler to read.
How do I keep a scraper running for months?
Cache raw HTML on disk while developing, so you tune selectors without hitting the site again. Save timestamps with every record. Run it on a schedule with cron or a workflow runner, and treat failures like any other production job: log the URL and status, alert on repeated errors, and make the job idempotent. The same discipline appears in Cron Isn't Dead and in Error Handling Patterns. For loading the results into a database or warehouse without silent corruption, see Automating Data Pipelines. If you want more small scripts in this style, Python Automation Scripts is a good next stop.
What should I do this week?
Pick one page you scrape or want to scrape. Spend ten minutes in the Network tab looking for a JSON endpoint, read the site's robots.txt and terms, then put the fetcher above in front of your existing code. Add one check that raises when the result set is empty. Those three changes remove most of the weekend failures I have had.
Uğur Kaval
AI/ML Engineer & Full Stack Developer specializing in building innovative solutions with modern technologies. Passionate about automation, machine learning, and web development.


