Receive a large amount of data from hardware devices in real time, and what are the recommended storage methods for large data?

problem description

there are hundreds of devices, and each device produces some real-time data every second. These data will be analyzed and used later. What is the best way to store these data?

the environmental background of the problems and what methods you have tried

previously stored in mysql, Aliyun"s mysql storage is also limited or upgraded. The key is that even if the upgrade data is large, the speed is slow. Is there any better way to store this data, which is convenient for storage and analysis later? If saving to a file

Jul.05,2022

MongoDB can or can consider


if data loss is allowed, consider using ELK collection and storage.


mysql stores relational data, such as device ID, time, etc.
specific data is recommended to be stored in Aliyun's OSS, which ensures that the query is convenient, and the MYSQL load is not in the order of magnitude of


million per second. It is recommended to use hbase for storage
since you do not require real-time analysis, you can upload batch data with local cache in the device
if you need to use message queue. kafka

is recommended
Menu