Beautiful Soup
Forgiving HTML parser for pulling data out of pages that were never meant to be parsed.
What it is
Beautiful Soup is a Python library for parsing HTML and XML documents. It creates a parse tree for parsed pages that can be used to extract data from HTML, XML, and other markup languages, making web scraping easier and more reliable.
Beautiful Soup provides Pythonic methods and attributes to navigate, search, and modify a parse tree. You can easily extract tags, attributes, text, or nested elements from HTML/XML documents.
- Watch for
- Install lxml as the parser — it is considerably faster than the built-in one
- Licence
- MIT
When to use it
The question documentation cannot answer for you — because it cannot recommend something else.
Reach for it when
- Extracting data from a handful of pages, in a script or notebook
- The HTML is malformed and a strict parser would refuse it
Look elsewhere when
- Crawling thousands of pages with concurrency, retries and politeness rules — that is Scrapy
- The content is rendered by JavaScript — you need a real browser via Playwright or Selenium
Installation
pip install beautifulsoup4Getting started
The smallest useful thing you can do with it, and what each part means.
from bs4 import BeautifulSoup
html = '<html><head><title>Test</title></head><body><h1>Hello</h1></body></html>'
soup = BeautifulSoup(html, 'html.parser')
print(soup.title.text)from bs4 import BeautifulSoup
html = '<a href="https://example.com">Example</a>'
soup = BeautifulSoup(html, 'html.parser')
links = [a['href'] for a in soup.find_all('a')]
print(links)Advanced usage
Where the library earns its place over a simpler alternative.
from bs4 import BeautifulSoup
html = '<div><p>Paragraph 1</p><p>Paragraph 2</p></div>'
soup = BeautifulSoup(html, 'html.parser')
div = soup.div
for p in div.find_all('p'):
print(p.text)from bs4 import BeautifulSoup
html = '<ul><li>One</li><li>Two</li></ul>'
soup = BeautifulSoup(html, 'html.parser')
items = soup.select('ul li')
for item in items:
print(item.text)from bs4 import BeautifulSoup
html = '<p>Old Text</p>'
soup = BeautifulSoup(html, 'html.parser')
soup.p.string = 'New Text'
print(soup.p)Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- AttributeError: 'NoneType' object has no attribute 'text'
- Check if the element exists before accessing its attributes or text.
- FeatureNotFound: Couldn't find a tree builder with the features you requested
- Install the appropriate parser library like `lxml` or `html5lib`.
Best practices
- Always specify a parser: 'html.parser', 'lxml', or 'html5lib'.
- Use `.find()` or `.find_all()` for reliable element searches.
- Use `.select()` for CSS selector queries when appropriate.
- Handle exceptions when elements might not exist to avoid errors.
- Combine with `requests` for fetching web pages efficiently.
Alternatives
Comparable options, and the reason you would pick one over the other.
Background
Why it exists, and what it was reacting to.
Beautiful Soup was created by Leonard Richardson in 2004. It was designed to handle poorly-formed HTML and XML documents gracefully, making it ideal for web scraping tasks where the markup is inconsistent. It has become a widely used tool in data extraction, web scraping, and automated web interactions.