How to extract website information using Python?

To crawl web information using Python, you can follow these steps:

1. Import the necessary libraries, including requests and BeautifulSoup.

import requests
from bs4 import BeautifulSoup

1. Send an HTTP request using the requests library and retrieve the webpage content.

url = "https://example.com"
response = requests.get(url)
content = response.text

1. Use BeautifulSoup to parse the web page content in order to extract the needed information.

soup = BeautifulSoup(content, "html.parser")

1. Use the methods provided by BeautifulSoup to search for and extract specific elements from a webpage.

# 以提取所有<a>标签的链接为例
links = soup.find_all("a")
for link in links:
    print(link.get("href"))

1. If you need to crawl multiple web pages, you can place the above code in a loop and modify the URL as needed.

It is important to follow the rules and laws of the website when scraping web information, avoiding excessive requests or privacy violations. Some websites may have anti-scraping mechanisms that require using other techniques to bypass.