If the mysql data exists, it is updated, and if it does not exist, it is inserted. How to improve the efficiency?

now we need to first judge whether a row of data is in the data table, update if it exists, insert if it doesn"t exist,
but when there are more than 2000 rows of data that need to perform such business logic, I find that the efficiency is very low, which is relatively fast at the beginning, and will gradually slow down after more than 1000 rows. The current solution is:
first query whether it exists according to conditions, insert if it does not exist, and update if it exists. Such a piece of data requires two sql, and cannot be inserted in batches. The execution of such articles is extremely inefficient.
my optimization plan is,

  1. use ON DUPLICATE KEY to let mysql solve the logic on its own.
  2. use replace to solve.

but these two methods are the characteristics of mysql itself. Do you have any better solutions to improve the efficiency of such execution?

Mar.15,2021

the efficiency problem needs to be solved by the unique key method, which not only improves the efficiency, but also avoids repetition. Other methods, such as non-indexing, also have efficiency problems when there is a large amount of data.


thousands of pieces of data, the efficiency will not be too bad, pay attention to the comparison of whether the existence of the field to add the index on the line.

if you do have a large amount of data (at least hundreds of thousands of levels), you can consider bulk loading to insert into the temporary table first, and then use the sql statement to do subsequent updates or inserts.

Menu