topic description
as shown in the following figure, the number of pages like this is incomplete and there is no link address for the next page. How can I get the next page?
related codes
start_urls = ["https://www.bilibili.com/v/kichiku/course"]
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse_list, args={"wait": "0.5"})
def parse_list(self, response):
-sharp
site = Selector(response)
-sharp
url_li = site.xpath("//*[@id="videolist_box"]/div[2]/ul/li/div/div[2]/a/@href").extract()
for v_url in url_li:
r_url = "https:" + v_url
yield Request(r_url, callback=self.parse_product)
-sharp
next_list = site.xpath("//*[@id="videolist_box"]/div[2]/div[2]/ul/li[@class="page-item next"]").extract()
print next_list
if next_list:
-sharp i += 1
url = "https://www.bilibili.com/v/ad/ad/-sharp/all/default/0/{}".format(-sharpi)
Request(url)
yield Request(url, callback=self.parse_list)
I have thought of passing an I parameter to parse_list, so that I can judge that there is a tag on the next page, which is + 1, but parse_list does not seem to be able to pass parameters in this way. I also want to get the maximum number of pages first, and then construct the URL, of each page and put all the video links of each page into a list, but if all the links are millions, it doesn"t feel good to put them in the list.
is there any good way to get the next page in this case?