How to check if pimatic had a proper shutdown?

---

Hi and hello to all of you,

i have a question about the behaviour of pimatic after a “hard” system crash caused by a power failure and how to manage it.

As i figured out, pimatic starts after a power loss of my raspberry pi, with the same state of buttons and switches as the last stored status in the database. By default these values are stored by the “diskSyncInterval” every 4h i think.

So with this i run into a problem. I have an off grid solar powered fridge supplied by batteries - controlled and managed by pimatic So i have e.g. a rule to check the voltage and the daily generated power.

If the fridge is running (controlled by a pimatic GPO-switch witch dependencies of voltage and generated power) and the power cuts off caused by the solar controller detecting under-voltage (bad batterie) and cutting everything out also my raspberry with the relais and pimatic gets cut off very ungraceful

Lets do a short run through:

Good sunny day --> enough generated power --> high battery voltage --> turn switch/fridge on --> voltage drops in normal ranges but bufferd by the sun and beer gets cold --> pimatic stores stats to database --> fridge is still on --> battery collaps in the night --> everything is cut off --> without the fridge running the volatge/battery gets better --> system gets power back --> pimatic starts with last known stats --> fridge is turned on --> system crashes …

As i figured out, i can manage it with a rule and the condition “when pimatic is starting switch everything off” …

But i need to restart the raspberry 4 times by cron a day as a prevention of connection loss caused by bad celluar network and bad UMTS/LTE USB Sticks.

So i need a way to check if the shutdown was planned or graceful by cron and “system reboot” or if it happens by a power failure. In case of the failure all sould be switched off - in case of a normal shutdown we can continue as before.

Yes, it is a very special topic! I am appreciating every kind of tips or help! If you need further information just text me here - there are a lot of rules for controlling and managing the system!

Thanks!

you can use the command line program tuptime.
it indicates the last restart reason.

> tuptime
...
System shutdowns:	1 ok   ->   1 bad
...
> tuptime -t -c -r | awk -F, 'NR==3 {print $5}'
"BAD"

@kosta Excellent proposal.

However, I my setup the all previous shutdowns are flagged “BAD” no matter what I do (pulling the cable, sudo reboot, or sudo shutdown -r 0.
Any ideas? I am running on Raspbian Buster.

...
pi@pi4:~ $ sudo shutdown -r 0
Shutdown scheduled for Fri 2019-07-19 14:27:14 CEST, use 'shutdown -c' to cancel.
...
pi@pi4:~ $ sudo tuptime -t -c -r
"No.","Startup Date","Uptime","Shutdown Date","End","Downtime"
"9","14:27:24 19.07.2019","54 seconds","","",""
"8","14:23:54 19.07.2019","7 seconds","14:24:01 19.07.2019","BAD","3 minutes and 23 seconds"
"7","14:22:07 19.07.2019","7 seconds","14:22:14 19.07.2019","BAD","1 minute and 40 seconds"
"6","14:20:00 19.07.2019","51 seconds","14:20:51 19.07.2019","BAD","1 minute and 16 seconds"
"5","14:19:11 19.07.2019","7 seconds","14:19:18 19.07.2019","BAD","42 seconds"
"4","14:02:07 19.07.2019","12 minutes and 54 seconds","14:15:01 19.07.2019","BAD","4 minutes and 10 seconds"
"3","13:56:43 19.07.2019","7 seconds","13:56:50 19.07.2019","BAD","5 minutes and 17 seconds"
"2","12:57:56 19.07.2019","7 seconds","12:58:03 19.07.2019","BAD","58 minutes and 40 seconds"
"1","19:18:43 04.07.2019","8 seconds","19:18:51 04.07.2019","BAD","14 days, 17 hours, 39 minutes and 5 seconds"
pi@pi4:~ $ tuptime -t -c -r | awk -F, 'NR==3 {print $5}'
"BAD"
...
pi@pi4:~ $ sudo shutdown -r 0
Shutdown scheduled for Fri 2019-07-19 14:36:47 CEST, use 'shutdown -c' to cancel.
...
pi@pi4:~ $ sudo tuptime -t -c -r
"No.","Startup Date","Uptime","Shutdown Date","End","Downtime"
"10","14:36:46 19.07.2019","30 seconds","","",""
"9","14:27:24 19.07.2019","7 seconds","14:27:31 19.07.2019","BAD","9 minutes and 15 seconds"
"8","14:23:54 19.07.2019","7 seconds","14:24:01 19.07.2019","BAD","3 minutes and 23 seconds"
"7","14:22:07 19.07.2019","7 seconds","14:22:14 19.07.2019","BAD","1 minute and 40 seconds"
"6","14:20:00 19.07.2019","51 seconds","14:20:51 19.07.2019","BAD","1 minute and 16 seconds"
"5","14:19:11 19.07.2019","7 seconds","14:19:18 19.07.2019","BAD","42 seconds"
"4","14:02:07 19.07.2019","12 minutes and 54 seconds","14:15:01 19.07.2019","BAD","4 minutes and 10 seconds"
"3","13:56:43 19.07.2019","7 seconds","13:56:50 19.07.2019","BAD","5 minutes and 17 seconds"
"2","12:57:56 19.07.2019","7 seconds","12:58:03 19.07.2019","BAD","58 minutes and 40 seconds"
"1","19:18:43 04.07.2019","8 seconds","19:18:51 04.07.2019","BAD","14 days, 17 hours, 39 minutes and 5 seconds"
pi@pi4:~ $ tuptime -t -c -r | awk -F, 'NR==3 {print $5}'
"BAD"
...
pulling cable at 14:38h
...
pi@pi4:~ $ sudo tuptime -t -c -r
"No.","Startup Date","Uptime","Shutdown Date","End","Downtime"
"12","14:38:50 19.07.2019","38 seconds","","",""
"11","14:37:59 19.07.2019","7 seconds","14:38:06 19.07.2019","BAD","44 seconds"
"10","14:36:46 19.07.2019","30 seconds","14:37:16 19.07.2019","BAD","43 seconds"
"9","14:27:24 19.07.2019","7 seconds","14:27:31 19.07.2019","BAD","9 minutes and 15 seconds"
"8","14:23:54 19.07.2019","7 seconds","14:24:01 19.07.2019","BAD","3 minutes and 23 seconds"
"7","14:22:07 19.07.2019","7 seconds","14:22:14 19.07.2019","BAD","1 minute and 40 seconds"
"6","14:20:00 19.07.2019","51 seconds","14:20:51 19.07.2019","BAD","1 minute and 16 seconds"
"5","14:19:11 19.07.2019","7 seconds","14:19:18 19.07.2019","BAD","42 seconds"
"4","14:02:07 19.07.2019","12 minutes and 54 seconds","14:15:01 19.07.2019","BAD","4 minutes and 10 seconds"
"3","13:56:43 19.07.2019","7 seconds","13:56:50 19.07.2019","BAD","5 minutes and 17 seconds"
"2","12:57:56 19.07.2019","7 seconds","12:58:03 19.07.2019","BAD","58 minutes and 40 seconds"
"1","19:18:43 04.07.2019","8 seconds","19:18:51 04.07.2019","BAD","14 days, 17 hours, 39 minutes and 5 seconds"

That’s strange. In my environment I have no problem.
even on different systems.

there is a tuptime option
tuptime -g will mark the next reboot as OK.
Normally, the init script executes exactly the same command during a shutdown.

@kosta said in How to check if pimatic had a proper shutdown?:

tuptime -g

Strange, even calling tuptime -g did not yield the expected result. I’ll try on another board with Raspbian Stretch.

pi@pi4:~ $ sudo tuptime -g
System startups:        12   since   19:18:43 04.07.2019
System shutdowns:       0 ok   ->   11 bad
System uptime:          6.83 %   -   1 day, 2 hours, 1 minute and 38 seconds
System downtime:        93.17 %   -   14 days, 19 hours, 4 minutes and 55 seconds
System life:            15 days, 21 hours, 6 minutes and 33 seconds

Largest uptime:         1 day, 1 hour, 46 minutes and 27 seconds   from   14:38:50 19.07.2019
Shortest uptime:        7 seconds   from   14:22:07 19.07.2019
Average uptime:         2 hours, 10 minutes and 8 seconds

Largest downtime:       14 days, 17 hours, 39 minutes and 5 seconds   from   19:18:51 04.07.2019
Shortest downtime:      42 seconds   from   14:19:18 19.07.2019
Average downtime:       1 day, 8 hours, 16 minutes and 49 seconds

Current uptime:         1 day, 1 hour, 46 minutes and 27 seconds   since   14:38:50 19.07.2019
pi@pi4:~ $ man tuptime
pi@pi4:~ $ sudo shutdown -r 0
Shutdown scheduled for Sat 2019-07-20 16:25:49 CEST, use 'shutdown -c' to cancel.
...
pi@pi4:~ $ sudo tuptime -t -c -r
"No.","Startup Date","Uptime","Shutdown Date","End","Downtime"
"13","16:25:47 20.07.2019","28 seconds","","",""
"12","14:38:50 19.07.2019","7 seconds","14:38:57 19.07.2019","BAD","1 day, 1 hour, 46 minutes and 50 seconds"
"11","14:37:59 19.07.2019","7 seconds","14:38:06 19.07.2019","BAD","44 seconds"
"10","14:36:46 19.07.2019","30 seconds","14:37:16 19.07.2019","BAD","43 seconds"
"9","14:27:24 19.07.2019","7 seconds","14:27:31 19.07.2019","BAD","9 minutes and 15 seconds"
"8","14:23:54 19.07.2019","7 seconds","14:24:01 19.07.2019","BAD","3 minutes and 23 seconds"
"7","14:22:07 19.07.2019","7 seconds","14:22:14 19.07.2019","BAD","1 minute and 40 seconds"
"6","14:20:00 19.07.2019","51 seconds","14:20:51 19.07.2019","BAD","1 minute and 16 seconds"
"5","14:19:11 19.07.2019","7 seconds","14:19:18 19.07.2019","BAD","42 seconds"
"4","14:02:07 19.07.2019","12 minutes and 54 seconds","14:15:01 19.07.2019","BAD","4 minutes and 10 seconds"
"3","13:56:43 19.07.2019","7 seconds","13:56:50 19.07.2019","BAD","5 minutes and 17 seconds"
"2","12:57:56 19.07.2019","7 seconds","12:58:03 19.07.2019","BAD","58 minutes and 40 seconds"
"1","19:18:43 04.07.2019","8 seconds","19:18:51 04.07.2019","BAD","14 days, 17 hours, 39 minutes and 5 seconds"

Nice! Thank you @kosta! The tool sounds perfect - i will check it out and report, if it work!

@kosta said in How to check if pimatic had a proper shutdown?:

That’s strange. In my environment I have no problem.
even on different systems.

I am trying this now Raspbian Stretch where I am getting the following error message:

pi@raspberrypi:~ $  tuptime -t -c -r
Usage: tuptime [options]

tuptime: error: used operator must be combined with [-k|--kernel]

When I use the -k option I am getting promising results even though the output format and ordering looks somewhat different to what I had on Raspbian Buster. I’ll try different scenarios and report back later.

pi@raspberrypi:~ $ tuptime -k -t -c -r
No.        Startup Date                                       Uptime       Shutdown Date   End     Downtime                                     Kernel

1     15:56:11 03/07/19   17 days, 21 hours, 2 minutes and 9 seconds   12:58:20 21/07/19    OK   38 seconds   Linux-4.19.42-v7+-armv7l-with-debian-9.9
2     12:58:58 21/07/19                     5 minutes and 22 seconds

Btw, on my Raspbian Stretch test system I got tuptime version 3.3.1 installed while on Raspbian Buster it is version 3.5.0

@mwittig i got it running on Raspbian Stretch from the git repo.

pi@raspberrypi:~ $ tuptime -t -c -r

worked here but i am not sure if tuptime detects the reboots/shutdowns right.

pi@pi:~ $ tuptime -t -c -r 
"No.","Startup Date","Uptime","Shutdown Date","End","Downtime"
"6","18:48:57 21.07.2019","13 minutes and 11 seconds","","",""
"5","18:21:19 21.07.2019","25 seconds","18:21:44 21.07.2019","BAD","27 minutes and 13 seconds"
"4","18:11:35 21.07.2019","26 seconds","18:12:01 21.07.2019","BAD","9 minutes and 18 seconds"
"3","18:00:24 21.07.2019","28 seconds","18:00:52 21.07.2019","BAD","10 minutes and 43 seconds"
"2","17:35:11 21.07.2019","24 seconds","17:35:35 21.07.2019","BAD","24 minutes and 49 seconds"
"1","12:00:19 21.07.2019","5 hours, 34 minutes and 47 seconds","17:35:06 21.07.2019","OK","5 seconds"

Number 3 and 6 were normal shutdowns ( sudo reboot ) so the status here should be “OK”. I will look into that.

@fritzbox360 Seems like it does not work reliably.

What do you mean by “i got it running on Raspbian Stretch from the git repo”? What did you install from which git repo?

@mwittig i did not use the raspian repos - i downloaded it hier direct from github. --> https://github.com/rfrail3/tuptime

I am running on 3.5.0 - so i will try the normal strech repos with 3.3.1

i’m running tuptime version 3.3.3

I have tested 3.5 and i get the same results.

pi@raspberrypi:~ $ tuptime -V
tuptime version 3.5.0
pi@raspberrypi:~ $ tuptime 
System startups:	3   since   07:42:08 22.07.2019
System shutdowns:	0 ok   ->   2 bad
System uptime: 		19.51 %   -   3 minutes and 48 seconds
System downtime: 	80.49 %   -   15 minutes and 40 seconds
System life: 		19 minutes and 28 seconds

Largest uptime: 	2 minutes and 49 seconds   from   07:58:46 22.07.2019
Shortest uptime:	29 seconds   from   07:42:08 22.07.2019
Average uptime: 	1 minute and 16 seconds

Largest downtime:	9 minutes and 58 seconds   from   07:42:37 22.07.2019
Shortest downtime:	5 minutes and 42 seconds   from   07:53:04 22.07.2019
Average downtime: 	7 minutes and 50 seconds

Current uptime: 	2 minutes and 49 seconds   since   07:58:46 22.07.2019

@fritzbox360 said in How to check if pimatic had a proper shutdown?:

i did not use the raspian repos - i downloaded it hier direct from github. --> https://github.com/rfrail3/tuptime

OK, thanks for clarifying this. v3.5.0 is the version I got installed on Raspbian Buster, btw.

The question now is, whether or not, tuptime produces the expected results, i.e. failure cases versus graceful shutdowns properly detected - which is not the case for with an up-to-date Raspbian Stretch running tuptime v3.3.1. I’ll also try to installed v3.5.0 from the repo.

So by now i switched to version 3.3.1 - lets check if there is a difference in a few hours.

Be in mind to delete the tuptime user before switching the version. I run into some problems here.

Will report back later!

After a few tests i have the same problem as @mwittig with the -k kernel-flag.

This is my list by now:

pi@pi:~ $ tuptime -t -k -c -r 
No.          Startup Date                               Uptime         Shutdown Date   End                             Downtime                                 Kernel
                                                                                                                                                                      
1     12:00:19 21.07.2019   5 hours, 34 minutes and 47 seconds   17:35:06 21.07.2019    OK                            5 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
2     17:35:11 21.07.2019                           24 seconds   17:35:35 21.07.2019   BAD            24 minutes and 49 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
3     18:00:24 21.07.2019                           28 seconds   18:00:52 21.07.2019   BAD            10 minutes and 43 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
4     18:11:35 21.07.2019                           26 seconds   18:12:01 21.07.2019   BAD             9 minutes and 18 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
5     18:21:19 21.07.2019                           25 seconds   18:21:44 21.07.2019   BAD            27 minutes and 13 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
6     18:48:57 21.07.2019                           25 seconds   18:49:22 21.07.2019   BAD              32 minutes and 1 second   Linux-4.9.59+-armv6l-with-debian-9.3
7     19:21:23 21.07.2019                           26 seconds   19:21:49 21.07.2019   BAD    2 hours, 5 minutes and 40 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
8     21:27:29 21.07.2019                           26 seconds   21:27:55 21.07.2019   BAD   2 hours, 32 minutes and 32 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
9     00:00:27 22.07.2019                           25 seconds   00:00:52 22.07.2019   BAD   5 hours, 59 minutes and 37 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
10    06:00:29 22.07.2019                           25 seconds   06:00:54 22.07.2019   BAD   5 hours, 59 minutes and 34 seconds   Linux-4.9.59+-armv6l-with-debian-9.3
11    12:00:28 22.07.2019   5 hours, 59 minutes and 43 seconds   18:00:11 22.07.2019    OK                             1 second   Linux-4.9.59+-armv6l-with-debian-9.3
12    18:00:12 22.07.2019    1 hour, 13 minutes and 59 seconds                                                                                                        

Looks nice but the awk is not working right as i expect the status OK/BAD here.

pi@pi:~ $ tuptime -t -k -c -r | awk -F, 'NR==3 {print $5}'
***empty here*** 
pi@pi:~ $ 

Sorry but i hate awk and its syntax so i came up with this quick an dirty script.

#!/bin/bash

if tuptime -t -r | awk -F, 'NR==4' | grep -q OK; then
    echo OK
else
    echo BAD

fi

Works for me to get the status into pimatic - i will have a look if the reporting is right!

Any ideas here for a better script are welcome But a big thank you so far to both of you!

After a few days of testing i still got stuck with the BAD shutdowns.

Confusing is the wrong downtime, but i think it could be a timing problem. If the power cuts out the Pi dont have the right time after booting agin. It takes a few minutes for the UMTS Stick to get ready and another few minutes for the time-synchronisation. Maybe that is creating a wrong entry in the database and the shutdown is marked as BAD.

pi@hidd:~ $ tuptime -t -r
No.          Startup Date                               Uptime         Shutdown Date   End                                                  Downtime
                                                                                                                                                    
27    18:00:15 24.07.2019            47 minutes and 42 seconds                                                                                      
26    12:00:17 24.07.2019      6 hours, 0 minutes and 1 second   18:00:18 24.07.2019   BAD   -1 years, 364 days, 23 hours, 59 minutes and 57 seconds
25    06:00:12 24.07.2019      6 hours, 0 minutes and 1 second   12:00:13 24.07.2019   BAD                                                 4 seconds
24    00:00:10 24.07.2019     6 hours, 0 minutes and 2 seconds   06:00:12 24.07.2019   BAD                                                 0 seconds
23    20:01:59 23.07.2019   3 hours, 58 minutes and 12 seconds   00:00:11 24.07.2019    OK   -1 years, 364 days, 23 hours, 59 minutes and 59 seconds
22    19:54:49 23.07.2019             7 minutes and 43 seconds   20:02:32 23.07.2019   BAD   -1 years, 364 days, 23 hours, 59 minutes and 27 seconds
21    19:42:10 23.07.2019                           22 seconds   19:42:32 23.07.2019   BAD                                 12 minutes and 17 seconds
20    18:00:12 23.07.2019    1 hour, 54 minutes and 22 seconds   19:54:34 23.07.2019   BAD   -1 years, 364 days, 23 hours, 47 minutes and 36 seconds
19    12:00:11 23.07.2019      6 hours, 0 minutes and 1 second   18:00:12 23.07.2019   BAD                                                 0 seconds
18    09:07:39 23.07.2019   2 hours, 52 minutes and 33 seconds   12:00:12 23.07.2019   BAD   -1 years, 364 days, 23 hours, 59 minutes and 59 seconds
17    06:00:11 23.07.2019     3 hours, 8 minutes and 7 seconds   09:08:18 23.07.2019   BAD   -1 years, 364 days, 23 hours, 59 minutes and 21 seconds
16    00:00:12 23.07.2019   5 hours, 59 minutes and 59 seconds   06:00:11 23.07.2019   BAD                                                 0 seconds
15    20:30:05 22.07.2019    3 hours, 30 minutes and 7 seconds   00:00:12 23.07.2019   BAD                                                 0 seconds
14    20:26:42 22.07.2019                           22 seconds   20:27:04 22.07.2019   BAD                                    3 minutes and 1 second
13    19:35:14 22.07.2019            51 minutes and 28 seconds   20:26:42 22.07.2019   BAD                                                 0 seconds
12    18:00:12 22.07.2019     1 hour, 35 minutes and 2 seconds   19:35:14 22.07.2019   BAD                                                 0 seconds
11    12:00:28 22.07.2019   5 hours, 59 minutes and 43 seconds   18:00:11 22.07.2019    OK                                                  1 second
10    06:00:29 22.07.2019                           25 seconds   06:00:54 22.07.2019   BAD                        5 hours, 59 minutes and 34 seconds
9     00:00:27 22.07.2019                           25 seconds   00:00:52 22.07.2019   BAD                        5 hours, 59 minutes and 37 seconds
8     21:27:29 21.07.2019                           26 seconds   21:27:55 21.07.2019   BAD                        2 hours, 32 minutes and 32 seconds
7     19:21:23 21.07.2019                           26 seconds   19:21:49 21.07.2019   BAD                         2 hours, 5 minutes and 40 seconds
6     18:48:57 21.07.2019                           25 seconds   18:49:22 21.07.2019   BAD                                   32 minutes and 1 second
5     18:21:19 21.07.2019                           25 seconds   18:21:44 21.07.2019   BAD                                 27 minutes and 13 seconds
4     18:11:35 21.07.2019                           26 seconds   18:12:01 21.07.2019   BAD                                  9 minutes and 18 seconds
3     18:00:24 21.07.2019                           28 seconds   18:00:52 21.07.2019   BAD                                 10 minutes and 43 seconds
2     17:35:11 21.07.2019                           24 seconds   17:35:35 21.07.2019   BAD                                 24 minutes and 49 seconds
1     12:00:19 21.07.2019   5 hours, 34 minutes and 47 seconds   17:35:06 21.07.2019    OK                                                 5 seconds

As tuptime ist not working properly, i got another idea. Why isnt there a way to create a rule for “when pimatic is shutting down”? If the pimatic service stops properly there is a “shutdown action” like write database to disk etc. if the power cuts out there is nothing like that (e.g. for missing database values).

A usable rule would be perfect but i dont know how to implement that.

Another option would be to create a system script with runs with planned shutdown/reboots in cron to set/touch a flag on disk and remove it x minutes after the reboot.

What do you think?