Freedom Sale
Independence Day Special — Unlock the AI Path 70% off our most popular AI course · Limited time offer
--Days
--Hrs
--Min
--Sec
Claim Your Discount
✦ Beginner ⏱ 30 min

🕷️ Build a Web Scraper with BeautifulSoup

🎯 What You'll Build

A Python script that scrapes quotes, authors, and tags from a practice website — then saves everything to a CSV file you can open in Excel.

📋 What You'll Need

1

Install the libraries

You only need two packages — requests to download web pages, and BeautifulSoup to parse them.

pip install requests beautifulsoup4
2

Fetch a webpage

Download the HTML of a page using requests. We use quotes.toscrape.com — a site built specifically for scraping practice.

import requests
from bs4 import BeautifulSoup

url = 'https://quotes.toscrape.com'
response = requests.get(url)

print(response.status_code)   # 200 = success
print(response.text[:500])    # first 500 chars of HTML
200
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Quotes to Scrape</title>
  ...
3

Parse and extract quotes

Pass the HTML to BeautifulSoup, then find every quote block and pull out the text, author, and tags.

import requests
from bs4 import BeautifulSoup

url = 'https://quotes.toscrape.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

quotes = []
for block in soup.select('div.quote'):
    text   = block.select_one('span.text').get_text(strip=True)
    author = block.select_one('small.author').get_text(strip=True)
    tags   = [t.get_text() for t in block.select('a.tag')]
    quotes.append({'text': text, 'author': author, 'tags': tags})

for q in quotes[:3]:
    print(q['author'], '—', q['text'][:60])
Albert Einstein — "The world as we have created it is a process of our th...
J.K. Rowling — "It is our choices, Harry, that show what we truly are, f...
Albert Einstein — "There are only two ways to live your life. One is as t...
4

Scrape multiple pages

The site has 10 pages. Loop through them by following the "Next" button link.

import requests
from bs4 import BeautifulSoup

BASE_URL = 'https://quotes.toscrape.com'
url = BASE_URL
all_quotes = []

while url:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    for block in soup.select('div.quote'):
        text   = block.select_one('span.text').get_text(strip=True)
        author = block.select_one('small.author').get_text(strip=True)
        tags   = ', '.join(t.get_text() for t in block.select('a.tag'))
        all_quotes.append({'text': text, 'author': author, 'tags': tags})

    next_btn = soup.select_one('li.next a')
    url = BASE_URL + next_btn['href'] if next_btn else None

print(f'Scraped {len(all_quotes)} quotes')
Scraped 100 quotes
5

Save to CSV

Export everything to a CSV file you can open in Excel or Google Sheets.

import csv

with open('quotes.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.DictWriter(f, fieldnames=['text', 'author', 'tags'])
    writer.writeheader()
    writer.writerows(all_quotes)

print('Saved to quotes.csv')
Saved to quotes.csv
💡 Tip: Always check a site's robots.txt before scraping (e.g. quotes.toscrape.com/robots.txt). Add time.sleep(1) between requests to avoid overloading a server.

🎉 You Did It!

You scraped 100 quotes across 10 pages and saved them to CSV. Try changing the selector to scrape a different site — every site has a different HTML structure, but the approach is always the same: inspect → select → extract.

Found something wrong?

Spotted a bug, broken code, or something that doesn't look right? Tell us what's off and we'll fix it.