Performance problems in batch Import of MongoDB with Java

I am using Java to implement a function of bulk importing data into MongoDB.

my idea is to read the target content, convert it to a Document object, put it into List, and then import it into the library through the insertMany method of MongoDB Driver. The code is as follows:

but now a performance problem has been found:

:26K+30MB+10

I locate the problem by myself and find that the time-consuming code is a block of circular statements, but I don"t know any other ways to implement it. I also ask you Daniel to give me some advice.

Mongodb java

Jan.23,2022

list is wrapped with concurrency-safe vector or Collections.synchronizedList (List list), and then added in chunks and multithreaded. This kind of task is easy to divide, such as handing over to one thread every 2k.

Previous: The web side obtains the physical serial number of the USB disk to log in.

Next: Excuse me, how does loopedSlides automatically get the total number of slide in Swiper?

Koa2 uses mongodb to write api interface to get data, how to get data elegantly (I am adding data to a new object, it's too low)
const MongoClient = require( mongodb ).MongoClient; const url = mongodb: localhost:27017 ; Database Name const dbName = youyue ; var obj = {}; module.exports = async(ctx,next)=>{ const num = ctx.query.num, page = ctx.query.pag...

Node.js mongodb javascript koa2

Mar.03,2021
How to efficiently query `findAll () `with spring-data-mongodb?
question In business, all tables in the database need to be queried and statistically analyzed and shown in the echats figure. The previous data is fine when the data is small, and the cache can barely support the column . now there is nearly 1 mi...

Mongodb java spring-data-mongo

Mar.06,2021
The efficiency of mongodb concurrent query is very low.
personal blog A personal blog written by has a concurrency problem: using Aliyun s pts to test concurrency, it is found that when the number of concurrency reaches 8 (which is negligible), the database query is particularly slow (there is very littl...

Node.js mongodb javascript

Mar.18,2021
The Java MongoTemplate query returns a specified field and a specified amount of data
suppose I have an entity class Article @Data @Document(collection = "articles") public final class Article { @Id private String id; private String name; private String url; private String author; private List<Commen...

Mongodb java

Mar.29,2021
MongoRepository update field
when using MongoRepository, I found that I want to update a field, but MongoRepository only provides the save method, but the save method requires a primary key. I directly use the default _ id as the primary key, and use the save method to insert a piec...

Springboot mongodb java

Sep.17,2021
Mongoose data query problem?
requirement description: mongodb contains the main information of two collection,theme storage topics, as well as an array of product id. Now if you want to find out the details of the products in products through the detailed id of the product, how sh...

Mongoose mongodb javascript

Dec.01,2021
About the design of cartoon server
I want to write a comic platform server, comics will be divided into paid and free, how should I deal with these picture resources. If I put it on oss, how can I tell whether the requesting user has paid or not? under the guidance of which god, I would b...

Mongodb java golang cPP python

Jan.23,2022
How does node.js use mongodb to make high-performance multi-concatenated table queries?
< H2 > for example, there are two tables < H2 > student Table id-> user id name-> user name classId-> Class id class Table id-> Class id name-> Class name if I want to query the information of all the students and the class information they b...

Mongoose mongodb javascript node.js

Jan.28,2022
What are the problems with mongodb highly aggregated queries?
problem description aggregates data in a specified format into a specified format related codes { "_id" : "10.182.135.21_CPU_20180627200000", "parameter" : { "CPUUtil" : [ { ...

Mongodb java

Feb.11,2022