How does the Python crawler automatically turn pages and store data in csv?

ask for God"s guidance. I have just come into contact with python crawler. I have some questions. Thank you very much.
I want to crawl some English news headlines and store them in a csv file
my code is as follows

 import csv, requests, re
 from bs4 import BeautifulSoup

 urls = ["https://www.defense.gov/News/Archive/?Page={}".format(str(i)) for i in range(1,10)]

def get_titles(urls,data = None):
    html = requests.get(urls).text
    soup = BeautifulSoup(html, "html.parser")
    articles = []
    for article in soup.find_all(class_="info"):
        Label = "Archive"
        News = article.find(class_="title").get_text()
        articles.append([Label,News])
        with open(r"1.csv","a", newline="") as f:
             writer = csv.writer(f)
             writer.writerow(["Label","News"])
             for row in articles:
                 writer.writerow(row)

for titles in urls:
get_titles(titles)

I want to crawl 1-9 pages of news headlines like this, but the final result is this

clipboard.png
each addition of a news title repeats the previous title to the csv.

ask for God"s guidance!

Mar.09,2021

the reason is that the previous articles list is not empty, so the previous data is output every time. You just need to empty the list variable
articles = []. I hope it can help everyone, and then write writer.writerow (['Label','News']) out of the loop so that you don't have to have Label and News every time

.
  • Python modifies the header of csv files in batches

    how Python3 batch modifies the header of csv files the novice crawler crawled the data into the csv file, but later, if you want to update header header, everything under the header is appended mode. Only header does not append , but if you update th...

    Apr.30,2021
  • Python handles csv file problems

    you need to use python to process csv files, but there are very few materials on the web to process csv data into object arry,. Is it not handled this way, or are there other methods ...

    Jun.03,2021
Menu