Nodemon doesn't wait for `SIGUSR2` graceful shutdown?

[rocketeerbkw]

• nodemon -v: 2.0.9
• node -v: v16.1.0
• Operating system/terminal environment: macOS Catalina, iTerm 2
• Using Docker? What image: Yes, but I have a non-docker breaking example
• Command you ran: yarn run nodemon watch.js

I'm trying to write a shutdown handler to gracefully close my mariadb sql client pool, but it never works because the script is killed too soon. I copied the example from the README and changed the signal to SIGTERM but it never lets the shutdown function finish.

If I manually send a kill -s SIGUSR2 <pid> then it will wait for the shutdown script, but nodemon doesn't wait.

My shutdown function uses async/await, but I was able to reproduce with out it. All the below info applies to async/await too (meaning it works with manual kill but not nodemon).

Expected behaviour

The script doesn't get killed by nodemon before it can gracefully shut itself down

Actual behaviour

Nodemon kills the script before my SIGUSR2 script can finish

Steps to reproduce

create a file watch.js:

process.stdin.read();

process.once('SIGUSR2', (signal) => {
  for (let i = 1; i <= 10000; i++) {
    console.log(i);
  }

  process.kill(process.pid, signal);
});

Save watch.js to trigger a restart and notice that the logs never reach 10,000 (for me it never goes past 4k). Send a manual kill -s SIGUSR2 <pid> and notice that the logs go to 10k, the script ends and then nodemon restarts it.

---

If applicable, please append the --dump flag on your command and include the output here ensuring to remove any sensitive/personal details or tokens.

[rocketeerbkw]

I can reproduce on a linux system as well

• nodemon -v: 2.0.10
• node -v: v12.21.0
• Operating system/terminal environment: Ubuntu 21.04 on a raspberry pi 4, bash
• Using Docker? What image: no
• Command you ran: ./node_modules/.bin/nodemon watch.js

[remy]

That's correct. nodemon sends the SIGUSR2 signal to the subprocess and if it doesn't exit within a predefined time frame, it'll force quit the sub process. Otherwise your sub process could be stuck hanging and nodemon would never be able to restart it.

[rocketeerbkw]

Can the "predefined time frame" be changed? I can't even close a database pool to a local db before nodemon kills it and crashes the database. This is a "guess I can't use nodemon" issue for me.

[remy]

🤔 thinking back, I originally wrote it so that if you trapped the signal from nodemon then your script would be responsible for stopping.

Logic has changed over the 11 years I've been developing nodemon so maybe it's been lost along the way. Though quickly looking this evening, it looks as I'd expect:

	  subprocess.kill(signal);

      pids.sort().forEach(pid => exec(`kill -${sig} ${pid}`, noop));

      waitForSubProcesses(child.pid, () => {
        // finally kill the main user process
        exec(`kill -${sig} ${subprocess.pid}`, callback);
      });

[rocketeerbkw]

if you trapped the signal from nodemon then your script would be responsible for stopping

That's what I assumed would happen based on the README:

If you need to clean up on shutdown inside your script you can capture the kill signal and handle it yourself.

Note that the process.kill is only called once your shutdown jobs are complete.

I did some timings locally and nodemon kills it after ~50ms for me. That seems really short.

process.stdin.read();

process.once('SIGUSR2', async(signal) => {
  while(true) {
    console.log(Date.now());
  }
});

[rocketeerbkw]

I'm looking into this more, I think it's a bug. The code for waitForSubProcesses has a debug message "still waiting for x sub-process to finish..." and combined with the README comments I think that nodemon should wait for the process to exit itself.

I added some debug statements and it looks like the psTree call in waitForSubprocesses returns an empty array for pids which means the function immediately calls the callback which kills the process.

rs
  nodemon bus emit: restart +3s
  nodemon:run psTree of 53461: [] +3s
  nodemon:run sending kill signal to [] +0ms
  nodemon:run checking ps tree for pids of 53461 +0ms
watch.js: SIGUSR2 start
  nodemon:run psTree of 53461: [] +41ms
  nodemon:run bus.emit(exit) via SIGUSR2 +8ms
  nodemon bus emit: exit +92ms
[nodemon] starting `node watch.js`
  nodemon:run fork sh -c node watch.js +3ms
  nodemon bus new listener: exit (1) +3ms
  nodemon:run start watch on: [ '*.*', re: /.*\..*/ ] +0ms
  nodemon:watch early exit on watch, still watching (1) +4s

I'm assuming that the "subprocess" here means the script that nodemon is running.

[rocketeerbkw]

I used git bisect and determined that this is the breaking commit: 47dfb8b.

The biggest change I see in that commit is the addition of child.kill(signal); before it calls waitForSubProcesses. If I commend out that line on master then nodemon waits for my SIGUSR2 function to complete as I expect.

[remy]

And with it comments out and without the sub process handling the exit itself does the spawned sub process still restart?

In theory, the kill signal is still SIGUSR2 which doesn't force a shutdown if it's being handled by the sub process...

[rocketeerbkw]

without the sub process handling the exit itself

meaning don't call process.exit? When using this example (and commenting out child.kill) the loop completes and nodemon restarts:

process.once('SIGUSR2', (signal) => {
  for (let i = 1; i <= 10000; i++) {
    console.log(i);
  }
});

  nodemon watch is complete +131ms
rs
  nodemon bus emit: restart +2s
1
2
3
...
9998
9999
10000
  nodemon bus.emit(exit) via SIGUSR2 +3s
  nodemon bus emit: exit +292ms
[nodemon] starting `node watch.js`
  nodemon fork sh -c node watch.js +3ms
  nodemon bus new listener: exit (1) +4ms
  nodemon start watch on: [ '*.*', re: /.*\..*/ ] +1ms
  nodemon:watch early exit on watch, still watching (1) +2s

I also tried with a async/await based delay wait 1 second and it also works without child.kill:

const delay = ms => new Promise(resolve => setTimeout(resolve, ms))

process.once('SIGUSR2', async (signal) => {
  console.log(`watch.js: ${signal} start`);
  await delay(1000);
  console.log(`watch.js: ${signal} stop`);
  process.kill(process.pid, signal);
});

  nodemon watch is complete +125ms
rs
  nodemon bus emit: restart +1s
watch.js: SIGUSR2 start
watch.js: SIGUSR2 stop
  nodemon bus.emit(exit) via SIGTERM +3s
  nodemon bus emit: exit +1s
[nodemon] starting `node watch.js`
  nodemon fork sh -c node watch.js +4ms
  nodemon bus new listener: exit (1) +4ms
  nodemon start watch on: [ '*.*', re: /.*\..*/ ] +0ms
  nodemon:watch early exit on watch, still watching (1) +2s

If I use an infinite loop, it looks like nodemon does not kill it, it just waits forever:

process.once('SIGUSR2', (signal) => {
  while(true) {
    console.log(Date.now());
  }
});

[remy]

Sorry, I wasn't clear at all (!). If you remove the hook in your subprocess that listens for SIGUSR2 does the process still restart?

[rocketeerbkw]

Without a SIGUSR2 listener and without child.kill it does get killed:

console.log('watch.js: process started');
process.stdin.read();

  nodemon watch is complete +39ms
watch.js: process started
rs
  nodemon bus emit: restart +7s
  nodemon:run sending kill signal to  +7s
  nodemon:run checking ps tree for pids of 12767 +1ms
  nodemon:run bus.emit(exit) via SIGUSR2 +3ms
  nodemon bus emit: exit +47ms
[nodemon] starting `node watch.js`
  nodemon:run fork sh -c node watch.js +3ms
  nodemon bus new listener: exit (1) +4ms
  nodemon:run start watch on: [ '*.*', re: /.*\..*/ ] +1ms
  nodemon:watch early exit on watch, still watching (1) +7s
watch.js: process started

[remy]

Right, well first of all: excellent job in following up and digging.

That's definitely a regression and I'll see if I can find time this week to sort it (or happy to take a pr).