try to write a script to download the target url using the request module ( https://www.npmjs.com/package.). request is based on the http module API encapsulation ( https://nodejs.org/api/http.h.), so there should be no extra overhead.
but it is found that the download speed is much slower than using the wget that comes with the system, and it is getting slower and slower.
what is the reason behind this?
wget uses the c language, while nodejs uses the javascript language.
The c language has to be compiled into machine instructions before it runs, while javascript converts scripts into machine instructions through an interpreter at run time.
this causes the c language to outperform javascript in most cases.
in addition, when downloading large files (GB and above), you cannot use code similar to the following
var request = require('request');
request('http://xxx', function (error, response, body) {
// ...
});this is likely to cause memory exhaustion, so stream processing, such as
, should be used.request('http://xxx').pipe(fs.createWriteStream('xxx'))