How does the node crawler get the queried data on the page?

problem description

has written about simple crawlers. According to the web page opened by url , the content in the web page is the information to be crawled, which is easier to do.
now the page defaults to no data or not the data you want, so you need to click a query to re-query the desired data.
but click the query button, the url of the page is unchanged, so you cannot get the desired data according to the url of the page.

in this case, how to get the queried data through the crawler?


then you directly simulate sending the request url sent by clicking the "query" button, and then look at the returned value, instead of directly sending the url of the page

.
Menu