Problems related to the change of web address after python crawler turns the page

recently, I encountered some problems when I helped a friend crawl the content on a biological website.
the url of the website is as follows:

https://www.ncbi.nlm.nih.gov/.

clipboard.png

20

clipboard.png

500

clipboard.png

:

clipboard.png

The code for the

crawler is as follows:
import requests
from bs4 import BeautifulSoup

base_url =" https://www.ncbi.nlm.nih.gov/gds"
wb_data = requests.get (base_url)
soup = BeautifulSoup (wb_data.text,"lxml")

for link in soup.select (" https://www.ncbi.nlm.nih.gov/."):

url = link.get("href")
print(url)

Why does the link change like this?
this code can only crawl the first 20 pieces of information on a web page. If I want to get all the information on the page, what should the code do to improve it?

places

Mar.28,2021

first of all, I can't open this page. So you can't see the actual situation.
but from the picture, it's likely to be the ajax asynchronously loaded
suggest that you grab the packet and take a look at the address of the ajax request


it should be Referer,. Take a look at requests headers

.
Menu