How to scrape text from web pages using Python?

1 year ago
Sophia Anderson
1 minute

In Python, we can use the third-party library BeautifulSoup to extract text from web pages. Here are the basic steps for extracting text from a webpage:

  1. Install the BeautifulSoup library: use the pip command to install the BeautifulSoup library, the command is pip install beautifulsoup4.
  2. Importing Libraries: Import the BeautifulSoup library and the requests library in Python code.
from bs4 import BeautifulSoup
import requests
  1. Send an HTTP request and retrieve webpage content: Use the requests library to send an HTTP request and get the HTML content of the webpage.
url = "https://example.com"  # 替换成待抓取的网页链接
response = requests.get(url)
html_content = response.content
  1. Parse HTML content: Utilize the BeautifulSoup library to parse HTML content and create a BeautifulSoup object.
soup = BeautifulSoup(html_content, 'html.parser')
  1. Extract webpage text: Utilize the methods provided by BeautifulSoup to extract the desired webpage text.
# 提取所有文本内容
text = soup.get_text()

# 提取指定标签的文本内容
text = soup.find('div').text  # 提取第一个<div>标签的文本内容

# 提取具有指定CSS类的标签的文本内容
text = soup.find(class_='class-name').text  # 提取具有class属性为class-name的标签的文本内容

# 提取具有指定id的标签的文本内容
text = soup.find(id='id-name').text  # 提取具有id属性为id-name的标签的文本内容

The above is the basic steps for extracting text from web pages, the specific extraction method can be adjusted according to the actual situation.

bannerAds