The io of the host occupied by mongodb is very high, and the overall processing of the host is slow.

problem description

Server environment: system: ubunt16 configuration: 16core 64GB memory disk information: logical volume 9.3T ordinary disk
mongodb version: 3.6.1
mongodb data volume: 2.4T

< H2 > mongodb configuration information: < / H2 >

fork=true
auth = true
quiet = true
port = 27017
bind_ip = 0.0.0.0
dbpath = / data/mongodb/data
logpath = / app/mongodb/logs/server.log
logappend = true
nojournal = true
smallfiles = true
storageEngine = wiredTiger
replSet=cluster0

< H2 > keyFile=/app/mongodb/conf/keyfile < / H2 >

the environmental background of the problems and what methods you have tried

when deploying
View disk io:

< H2 > [cat / proc/diskstats] < / H2 >

root@mongodb-master:/log/mongodb-sharp cat / proc/diskstats | grep sd
80 sda 89216340 721299 19543253752 1198749100 135233114 18424358 16912073556 3601130864 0 806097224 504709580
8 1 sda1 698 13692 7092 82 20 144 4216 7236
82 sda2 100 20 360 00 36
85 sda5 89215468 721299 1954323168 1198739024 135233106 18424356 16912073536 3601130720 0 806095420 504674104
8 16 sdb 15143226 31 4556290432 217882804 133682523 150413 13015451088 2788505648 0 616255212
8 17 sdb1 15142998 31 15142998 2788505636 0 616254740 3006203100
8 48 sdd 2259963 0 1056467744 5453640 14378 2363980184 1235405640 0 12897720 1240845948
8 49 sdd1 2259681 0 1056458448 5452560 4640371 14378 2363980176 1235405628 0 12896664 1240844852
8 64 sde 282 07440 776 00 0736 776
8 80 sdf 2820 7440 816 00 00 0800 816
8 32 sdc 16152552 35 7334393944 90275500 13080257 37351 4956688512 0 4219663108

< H2 > 8 33 sdc1 16152283 35 7334384752 90274620 13080256 37351 4956688504 4219663100 0 70932716 14932592 < / H2 > < H2 > [iotop] < / H2 >

Total DISK READ: 0.00Bbe/4 root
Actual DISK READ: 0.00Bbe/4 root
TID PRIO USER DISK READ DISK WRITE SWAPIN IO > COMMAND
151089 be/4 root 0.00BUnip 0.00% 97.46%. / mongod-f.. / conf/mongo.conf [WTOplog.lThread]
18364 be/ 4 root 0.00 BUnip s 0.00 BUnip s 0.00% 0.10% [kworker/u384:0]

< H2 > 4096 be/4 fy 0.00 BUnix s 0.00 BUnip s 0.00% 0.00% sshd: fy@pts/0 < / H2 > < H2 > [free-g] < / H2 >
          total        used        free      shared  buff/cache   available

Mem: 62 36 00 25 25

< H2 > Swap: 63 063 < / H2 >

mongodb log information:
slow sql statement:
2018-10-28T12:00:22.757+0800 I COMMAND [conn5729454] command svpn_log_porterman.user_operation_logs command: insert {insert: "user_operation_logs", $db: "svpn_log_porterman", ordered: true, $clusterTime: {clusterTime: Timestamp (1540699220, 16), signature: {hash: BinData (0, 476E3988CC42E310EAB16EF5E2E558C3CDE2AEC2), keyId: 6577146130629394433}}, Lsid: {id: UUID ("a45ceeae-eb95-4634-9d5a-ab9e876b8fec")} ninserted:1 keysInserted:4 numYields:0 reslen:214 locks: {Global: {acquireCount: {r: 2, w: 2}}, Database: {acquireCount: {w: 2}}, Collection: {acquireCount: {w: 1}}, Oplog: {acquireCount: {w: 1} protocol:op_msg 2009ms
Synchronize slow sql statement:
2018-10-28T10:30:04.456+0800 I COMMAND [conn6239997] command local.oplog.rs command: find {find: "oplog.rs", limit: 1, sort: {$natural: 1}, projection: {ts: 1, t: 1}, $readPreference: {mode: "secondaryPreferred"}, $clusterTime: {clusterTime: Timestamp (1540693684, 5), signature: {hash: BinData (0, B1D7AE2F4DF9DFB12BCCDCEBFA0D82A6FBFC8434), keyId: 6577146130629394433}, $db: "local"} planSummary: COLLSCAN keysExamined:0 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:243 locks: {Global: {acquireCount: {r: 4}, Database: {acquireCount: {r: 2}}, oplog: {acquireCount: {r: 2} protocol:op_msg 115950ms

add: the disk is not damaged, and the database will report that the cpu wait is greater than 32% at night (there are tens of billions of data in a table)

what result do you expect? What is the error message actually seen?

problem description

there is less delay in inserting data, and the host is not stuck. Now the connection is very stuck, and the server is not stuck for a while.
Note: I am a mongodb rookie. I hope God will guide me patiently

.

what result do you expect? What is the error message actually seen?

Sep.23,2021
Menu