How can the collected data be inserted into the database to avoid duplicate data?

A large number of data fields are collected. Before inserting into the mysql database, it is necessary to determine whether there is duplicate data in the database. Multiple fields can be duplicated together, that is, some individual fields are allowed to be duplicated. How to avoid inserting duplicate data?
my previous practice is to splice the data to be compared and then use sha1 to insert this value into the database when inserting the database. Later, before inserting the data, we can determine whether the SHA1 value in the database is duplicated, and skip this piece of data if there is any repetition. But this will be repeated individually, I don"t know why. Is there a better way?

Mar.21,2021

sort the values of multiple fields in a dictionary sort () and then sha1, so that the comparison will not be repeated


set the unique key


after the collection link SHA1 is encrypted, set the unique primary key


, and then the Filter method is called when the time stamp

is encrypted.
Menu