Nodjes read csv file insert mysql operation

problem description

the foreground uploads the file, and the background uses the fast-csv module to read the file and insert it. I hope it will be returned to the foreground after the insertion is completed, so I adopted promise, but it didn"t seem to work. Thousands of rows of data will not be inserted into the database, and the execution of the large file will be stopped halfway. I hope you can give me more advice. Thank you

the environmental background of the problems and what methods you have tried

nodejs"s asynchronous execution mechanism, using await/async,promise, is not very effective.

related codes

/ / Please paste the code text below (do not replace the code with pictures)

//
query2 = (sql, params = []) => {
    return new Promise((resolve, reject) => {
      pool.getConnection((err, connection) => {
        if (err) {
          return reject(err);
        }
        connection.query(sql, params, (err, result) => {
          connection.release();
          if (err) {
            return reject(err);
          }
          resolve(result);
        });
      });
    });
  };
  
  var    sql = "INSERT INTO tf_m_section_number VALUES ?";
  var datas=new Array();
  var pList=new Array();
    fs.createReadStream(fPath)
  .pipe(csv())
  .on("data", function(data){
      countsPP;
    if(count<700)
        datas[countPP]=data;
    // if(count==3)
    //     console.log(datas);
     if(count==700) {
         console.log(count);
         pList.push(query(sql,[datas]));
        //console.log(result);
        count=0;
        datas.length = 0;
    }
   //  rows[countPP]=row;
 })
  .on("end",function(){    
    console.log(counts);
      console.log("done");
    if(count>0){
        pList.push(query(sql,[datas]));
    //    console.log(result);
        //return res.send("ok");
            
    }
    Promise.all(pList).then(function(result) {
        // you code...
        return res.send("ok");
    });
    //
  }).on("error",(err)=>{console.log("",err);});

  

what result do you expect? What is the error message actually seen?

all the data can be read into the database, in fact, some data will be lost

Apr.02,2021

to give an idea, the first time to insert a copy of the table, and then empty the original table, and then import again, see before and after the import of the two tables lost data:
if the lost data are the same, then a separate csv contains only the missing part of the data, then try to import this csv, to see what is wrong;
if the lost data is random, then you need to take a specific look.

Menu