Details
I'm attempting to adjust the buffer size (highWaterMark) for the stdout from a child_process.spawn() call, and I'm having a hard time doing this successfully. Below is some code that illustrates what I'm looking for. I'm trying to perform some benchmarking with various stdout buffer sizes, but until I can actually adjust the buffer size, I won't be able to do so. Any advice would be appreciated, thanks!
(Note, you'll need a file called "testfile_32MB" in place to run this example code, and it should be at least a few megabytes in size.)
const child_process = require('child_process');
const stream = require('stream');
const fs = require('fs');
const childProc = child_process.spawn("cat", ["testfile_32MB"], {highWaterMark: 1024*1024}); // <-- child_process.spawn does not have a highWaterMark option, so including it has no effect, but I'm including it just as an illustration of what I want.
childProc.stdout.writableLength = 1024*1024; // <-- This does not help.
childProc.stdout._writableState.writableLength = 1024*1024; // <-- This does not help.
childProc.stdout._writableState.writableHighWaterMark = 1024*1024; // <-- This does not help.
childProc.stdout.writableHighWaterMark = 1024*1024; // <-- This does not help.
childProc.stdout._readableState.highWaterMark = 1024*1024; // <-- This does not help.
childProc.stdout._writableState.highWaterMark = 1024*1024; // <-- This does not help.
childProc.stdout.on("data", (data) => {
console.log("data length [from spawn()]:", data.length); // <-- data.length will always be <= 64KB! How can we get it higher?
});
// Just to clarify what I want:
const readStream = fs.createReadStream("testfile_32MB", {highWaterMark: 1024*1024});
readStream.on("data", (data) => {
console.log("data length [from createReadStream()]:", data.length); // <-- This prints values <= 1MB, which is what I wish I had for child_process.spawn/stdout...
});
Node.js version
12.21.0
Operating system
Ubuntu linux
Scope
code
Module and version
Not applicable.
Details
I'm attempting to adjust the buffer size (highWaterMark) for the stdout from a child_process.spawn() call, and I'm having a hard time doing this successfully. Below is some code that illustrates what I'm looking for. I'm trying to perform some benchmarking with various stdout buffer sizes, but until I can actually adjust the buffer size, I won't be able to do so. Any advice would be appreciated, thanks!
(Note, you'll need a file called "testfile_32MB" in place to run this example code, and it should be at least a few megabytes in size.)
Node.js version
12.21.0
Operating system
Ubuntu linux
Scope
code
Module and version
Not applicable.