This is legitimately the best usage of this meme I’ve seen in years. Termination signals hnnngg
That’s why you launch them through systemd.
But systemd is the devil and makes nothing better, right?
Right?
Really? I’ve never had issue with it
It’s a joke about the criticism systemd gets
openrc exists
Openrc is bloat, you should manually pair electrons
The constant nagging by you systemd people worked. I’ve written a unit that does what I need it to do. That was more annoying than I think it needed to be, but well… my solution didn’t work at all.
AFAIK kernel itself doesn’t send any signals to processes on shutdown/reboot, it just stops executing them. This is a job service manager (e.g. systemd) that terminates processes using SIGTERM before asking kernel to shutdown.
You are never guaranteed to be able to do anything during a crash. You are better off handling these kinds of edge cases in a recovery phase during the start of your app.
It’s not a crash. It’s a graceful shutdown. I expected that to also shutdown my app gracefully.
I’m actually trying to store the program state that hasn’t been persisted yet to disk. Good luck doing that after the next boot.
Persist everything to disk in real time. When the signal hits exit immediately.
Persist everything to disk in real time.
That’s the thing I’m trying to avoid.
Easier to do than to get never-exercised edge-case code to work flawlessly. Are you sure you can’t just throw sqlite at the problem? It’s often overkill but, hey, it’s there on the shelf, might as well use it and I’ve seen it out-perform hand-rolled data structures. Non-persistent ones, written by very confident C coders. And remember crashes are unavoidable, if nothing else then someone can trip over the power cord.
You can’t know that from my issue description, but throwing a database at that problem really is ridiculous overkill.
Still thanks for the suggestion
It’s mostly about throwing ACID at the problem, sqlite just happens to be battle-tested to a ludicrous degree, it’s light enough to not be unconscionable overhead in simple situations (unless you’re on embedded), and performant enough to also deal with nastier situations so I prefer it over some random K/V store with the same guarantees. It’s also a widely-used and stable data format which might come in handy.
That said, if you want to go lightweight do consider good, ole, POSIX filesystem guarantees, in particular that mv is atomic (as long as you stay on the same filesystem but that’s easy to ensure by mv’ing within a directory). That’s not durable on its own, you’ll need to fsync for that, and consistency and integrity is up to your code.
Admittedly it’s very difficult if you want to maintain consistency but the benefits are enormous!
Crash-only software. To be resilient you need some kind of ACID anyway which means that you can let go of your shutdown procedure and just send yourself SIGKILL instead.
My computer has a problem where occasionally it will become completely unresponsive. (Mouse cursor doesn’t move. Keys have no apparently effect. Whatever app is running freezes. I think its a hardware problem with the graphics card, but I don’t know what. Logs at the time it freezes say “the GPU has fallen off the bus”.)
Anyway… I recently learnt about Magic SysRq. And I’ve been able to shutdown the computer from this unresponsive state with
SysRq, R E I S U O. Where as I understand it, the “E” tells processes the end nicely if they can; and then the “I” just ends them by force.(At this point, I’m realising that the
Eis SIGTERM, not SIGINT - so that screws up the relevance of my story; but I figure I’ll keep going anyway.)The point is, I’ve been using key combo with a nice pause between each key, thinking there was some chance that processes might be ending gracefully. But when I tried it while the computer wasn’t frozen, the computer was able to inform me that the E and I commands were disabled. (I don’t know why.) So even though I wanted to give a nice “please end” signal, in the end that just wasn’t happening.
Have you tried other TTYs? When I press Ctrl+Alt+F3, for example, I switch to a ‘real’ terminal that (I assume) doesn’t require the GPU to work. From there you might be able to recover.
On Linux Mint the default, graphical userface is on TTY7 (CTRL+Alt+F7).
Yes, I have tried that. But it has no apparent effect when the computer is in that frozen state. (i.e. pressing ctrl+alt+F3 or whatever doesn’t change what is on the screen.
One time I tried imagining that the terminal came up, and so I ‘signed in’ and tried to shut down without seeing anything on the screen. But it didn’t work. … but since I couldn’t see anything, it’s quite possible I just mistyped something. So I guess that’s inconclusive.
You could try enabling systemd-oomd. It’s a userspace OOM killer and seems to be aggressive enough to mostly stop that from happening.
I’m not sure what you mean. Stop my computer from hanging like that? Or make it so that SIGTERM and SIGKILL work with those magic keys? I don’t know what an OOM killer is.
I’d definitely be interested in something that stopped the computer from crashing. But I kind of doubt that’s going to happen; because as I said, I’m pretty sure it’s a hardware thing. (But I suppose appropriate software might allow it to successful recover from the problem without having to restart.) It’s pretty rare by the way. I use the computer almost every day, and I haven’t had this problem happen for a few weeks. I’ve basically given up trying to fix it.
Yeah the hanging is being caused by your RAM being filled. Systemd-oomd will kill things more aggressively than the kernel out-of-memory handler that should stop it from locking up.
Alright, then that sounds like its definitely worth trying. Thanks for the suggestion. I guess it will be hard for me to tell whether or not it is working, because in the ideal case: nothing happens. But I’ll definitely try it out and hope for the best.
I realise I’m late to the party and you’ve alreadu gone the systemd unit way, but had your script trapped sigterm to begin with?
When I run the script myself and kill it, it gets the signal and acts correctly. Only when I poweroff the system, this doesn’t work.
SIGINT is sent when you press Ctrl+C. SIGTERM is sent in just about every other situation - basically when the system wants the program to end. For instance when systemd wants to stop the service or the default signal with programs like
killpkillhtopetc. You should catch both of these signals.I did try to catch all of these signals:
| "SIGABRT" | "SIGALRM" | "SIGBUS" | "SIGCHLD" | "SIGCONT" | "SIGFPE" | "SIGHUP" | "SIGILL" | "SIGINT" | "SIGIO" | "SIGIOT" | "SIGKILL" | "SIGPIPE" | "SIGPOLL" | "SIGPROF" | "SIGPWR" | "SIGQUIT" | "SIGSEGV" | "SIGSTKFLT" | "SIGSTOP" | "SIGSYS" | "SIGTERM" | "SIGTRAP" | "SIGTSTP" | "SIGTTIN" | "SIGTTOU" | "SIGUNUSED" | "SIGURG" | "SIGUSR1" | "SIGUSR2" | "SIGVTALRM" | "SIGWINCH" | "SIGXCPU" | "SIGXFSZ" | "SIGBREAK" | "SIGLOST" | "SIGINFO";Good luck trapping the SIGKILL signal
Yep, in newer versions of node even attempting to do so will terminate the process.
How are you running your script?
(I have no idea how to solve your issue I’m just asking questions to sound smart and helpful)
It’s a node process invoked by a
run.sh, which gets executed via a.desktopfile in the~/.config/autostartdirectory.I went with a systemd unit now and it works.
I can’t remember off the top of my head, but your shell script might not be relaying the SIGTERM. Make sure you start your node process with the “exec” statement. This will replace the script’s process with node instead of having node be a subprocess of your script.
When I run the script myself and kill it, it gets the signal and acts correctly. Only when I poweroff the system, this doesn’t work.
I also tried prepending
exec, but no dice.
“SyStEmD iS bLoAt…”
Your DE may be the one not relaying the sigterm, or it may be losing the PID because of the double launching.
Does the LSB have something to call on termination? Or you may want to call an executable there instead of a script.
I didn’t know why a person would go to these lengths to deal with a misbehaving computer, as compute devices are generally for work, and need to work in order to do work, and any kind of crash is going to get my entire focus until it is banished to Hades…
…but then, learned something along the way I probably otherwise would not have, because of @[email protected]’s tenacity.
Can’t SIGTERM be observed to react to a poweroff?
That’s what I thought, but my program never receives the signal.
That might be an issue with who invokes your program.
Interesting. Is this top answer accurate then?
https://stackoverflow.com/questions/31731980/handling-a-linux-system-shutdown-operation-gracefully
It turns out I’m getting SIGCHLD. It might be related to how my script is started – it is a bash script that starts a node process and is itself run by Cinnamon’s (?) startup applications feature.Wrong; still investigating
Missed opportunity to make the last panel have her eyes closed (or just be fully black).
SIGHUP or SIGPWR, maybe?
nope, SIGCHLD.Wrong.But thanks.
