Author Topic: Are Scheduled Auto Reboots A Good Idea  (Read 1530 times)

Offline MtView

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +1/-0
Are Scheduled Auto Reboots A Good Idea
« on: February 09, 2018, 02:25:11 pm »
Today a geofence scene didn't run. I checked and noticed that my controller was offline for 30 seconds which of course was when the scene needed to be triggered. It was also offline for 8 minutes around 520a EST.  I don't know why it went offline but it got me thinking about scheduled auto reboots.

Are there any downsides to running a reboot every other morning at 3am so long as I have an internet connection? I think Vera has been having server issues so that could be why Vera went offline.

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 256
  • Karma: +8/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #1 on: February 09, 2018, 03:15:18 pm »
Today a geofence scene didn't run. I checked and noticed that my controller was offline for 30 seconds which of course was when the scene needed to be triggered. It was also offline for 8 minutes around 520a EST.  I don't know why it went offline but it got me thinking about scheduled auto reboots.

Are there any downsides to running a reboot every other morning at 3am so long as I have an internet connection? I think Vera has been having server issues so that could be why Vera went offline.

I run a scheduled reboot in the middle of the night every day. I haven't seen any downsides. Obviously if a trigger occurs during the reboot, that action won't run. According to Vera, it needs an internet connection during a reboot to contact the NTP server to set the time. The NTP servers are hosted by OpenWRT. If there is no internet, Vera will spend a lot of time retrying.  Recall that Vera has no RTC hardware---no NTP, no system clock.  I don't use geofencing, so I can't comment on the controller online/offline issue. Search the forum---there is example LUA code for a reboot scene that first checks internet connectivity. No internet, no reboot.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +27/-12
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #2 on: February 09, 2018, 04:15:22 pm »
Are there any downsides to running a reboot every other morning at 3am so long as I have an internet connection?

Maybe this is only an "emotional" opinion, but in general I'm not a fan of frequent reboots, as I considered them as "hard-way" solution.
Any tasks which were done in the background by the system are interrupted and I'm not sure if this can affect your system (i.e. what happens if the reboot interrupt saving data to memory?) or not.

But the real question is if scheduled reboots will solve any of your issues?
As an example: quite recently I've had problem with lua reloading every 10-15 minutes, which made my system rally unresponsive (like: you switch on a light, your luup reloads, aaaaand... after 2-3 minutes your light is on).
But I tried rebooting and it simply didn't solve that issue. the reloads continued with same frequency, and the issue had to be solved by Customer Care.
From the other hand, I had also the issue of rapidly growing amount of cached memory, and for this issue reboot helped a lot (memory went back to normal). But even here CC proposed running a script which clears cached memory without need to reboot and I prefer to use this as a first choice solution.

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 256
  • Karma: +8/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #3 on: February 09, 2018, 05:14:07 pm »
Are there any downsides to running a reboot every other morning at 3am so long as I have an internet connection?

Maybe this is only an "emotional" opinion, but in general I'm not a fan of frequent reboots, as I considered them as "hard-way" solution.
Any tasks which were done in the background by the system are interrupted and I'm not sure if this can affect your system (i.e. what happens if the reboot interrupt saving data to memory?) or not.

But the real question is if scheduled reboots will solve any of your issues?
As an example: quite recently I've had problem with lua reloading every 10-15 minutes, which made my system rally unresponsive (like: you switch on a light, your luup reloads, aaaaand... after 2-3 minutes your light is on).
But I tried rebooting and it simply didn't solve that issue. the reloads continued with same frequency, and the issue had to be solved by Customer Care.
From the other hand, I had also the issue of rapidly growing amount of cached memory, and for this issue reboot helped a lot (memory went back to normal). But even here CC proposed running a script which clears cached memory without need to reboot and I prefer to use this as a first choice solution.
Unix/Linux reboots (soft reboots) do a graceful shutdown before restarting, so you shouldn't lose anything. This is (of course) not the case with a power cycle hard boot.  The issue is whether a periodic reboot solves problems, as opposed to preventing them from occurring. My system is stable, but experience has shown me that after running for a long time (many days or weeks), it gradually becomes less stable.

Why is this? I don't know, but a periodic reboot prevents this undesirable behavior. For systems that are actively having problems, a reboot, as you quite properly point out, may not solve the issue. If there were non-reboot solutions, I'd be happy to try them, but so far I haven't seen them.

Offline kartcon

  • Full Member
  • ***
  • Posts: 147
  • Karma: +7/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #4 on: February 09, 2018, 05:21:18 pm »
kwieto, Have you seen or tried the Cache Clearing script? Just wondering if you tried it, what the effect was and how you might have implemented it (ie as a scene, with LUA or PLEG).

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 256
  • Karma: +8/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #5 on: February 09, 2018, 05:26:07 pm »
kwieto, Have you seen or tried the Cache Clearing script? Just wondering if you tried it, what the effect was and how you might have implemented it (ie as a scene, with LUA or PLEG).
Same here---very interested.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +27/-12
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #6 on: February 09, 2018, 05:35:58 pm »
Customer Care suggested me to run a scene on a 30 minutes schedule with following code:

Code: [Select]
os.execute("echo 3 > /proc/sys/vm/drop_caches")
As my system clears cache properly in general, I incorporated it to a scene which runs only if amount of free memory drops below certain limit.
If this won't help, then there is next limit at which reboot is executed.
I've found that if free memory (don't mistake with "available memory" drops to less than 10Mb, then the controller gets unstable.
So my limits for Plus are 50Mb for cache cleaning and 25Mb for reboot.
For Edge these values should be lower, I think of 18Mb and 12Mb, as my free memory is a little more than 20Mb in normal state.

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 256
  • Karma: +8/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #7 on: February 09, 2018, 06:13:25 pm »
My VP currently shows around 140Mb of free memory, nowhere near your 50Mb cache limit. I'll keep and eye on it. Please let us know how this works over the next few weeks, and thank you for posting.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +27/-12
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #8 on: February 10, 2018, 10:53:54 am »
If everything works, Plus clears some cache by itself more or less every hour (for Edge it seems to be every two hours).
The problem starts when this process doesn't work our is not sufficient.
In such case memory is consistently consumed by increasing cache. In Plus case it took about 6 hours to go from 135-140Mb to 5-8Mb. This points to another risk with scheduled reboot - your memory may be drained far before reboot is scheduled and as result there is high risk that your reboot won't be performed or (worse) controller will go into permanent rebooting loop due to lack of enough memory and inability to clear the cache (that's more our less how one of my Edge' got bricked).
« Last Edit: February 10, 2018, 04:25:44 pm by kwieto »

Offline kartcon

  • Full Member
  • ***
  • Posts: 147
  • Karma: +7/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #9 on: February 10, 2018, 07:40:20 pm »
+Bump and Thank you for sharing!

Offline MtView

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +1/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #10 on: February 10, 2018, 08:36:16 pm »
Customer Care suggested me to run a scene on a 30 minutes schedule with following code:

Code: [Select]
os.execute("echo 3 > /proc/sys/vm/drop_caches")
As my system clears cache properly in general, I incorporated it to a scene which runs only if amount of free memory drops below certain limit.
If this won't help, then there is next limit at which reboot is executed.
I've found that if free memory (don't mistake with "available memory" drops to less than 10Mb, then the controller gets unstable.
So my limits for Plus are 50Mb for cache cleaning and 25Mb for reboot.
For Edge these values should be lower, I think of 18Mb and 12Mb, as my free memory is a little more than 20Mb in normal state.

This is really interesting kwieto.  Can you post the script for triggering cache clearing at the 50Mb limit?

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +27/-12
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #11 on: February 11, 2018, 05:51:14 pm »
The code is simple, but as prerequisite you need to install System Monitor plugin (and here I use Ping sensor plugin as well).

Then put following lua in the scene:

Code: [Select]
local Memory = luup.variable_get("urn:cd-jackson-com:serviceId:SystemMonitor", "memoryFree", 209)
local Ping = luup.variable_get("urn:micasaverde-com:serviceId:SecuritySensor1", "ArmedTripped", 56)

if tonumber(Memory) < 50000 then
  os.execute("echo 3 > /proc/sys/vm/drop_caches")
end

if ((tonumber(Memory) < 25000) and (tonumber(Ping) == 0)) then
  os.execute("reboot")
elseif (tonumber(Memory) < 13000) then
  os.execute("reboot")
end

return (tonumber(Memory) < 50000)

As an explanation:
- Device ID 209 is ID of my system memory plugin device (of course you should change it according to your setup)
- Device ID 56 is ID of my Ping Sensor plugin device (set to trip when internet is down). If you don't use such plugin, you can use lua code instead - here on the forum you can find some examples how to write such code and incorporate it into a rebooting scene
- I decided to add additional condition that if memory drop below 13Mb, then reboot is done regardless of internet connection (from my experience, amount around 5Mb or less brings risk of bricking controller during reboot)
- you can write the code without "tonumber()" section, but then all values should be in quotation marks (so [tonumber(Memory) < 50000)] is more or less equal to [Memory < "50000"])
- you can omit part [return (tonumber(Memory) < 50000)]. I added it because  this way if memory is above 50Mb, scene won't report as "ran" even if triggered by schedule or trigger. This way it is easier to track when was the last effective run of the scene. Unfortunately if you set notifications, they will be sent every time scene is run, regardless of this condition (my first idea was to have notification if scene was effectively performed, so clearing or reboot really took place).
- you can run this scene on schedule (like every n hours), but I added it to the watch table which I have in startup lua anyway, made according to this recipe: http://forum.micasaverde.com/index.php/topic,18679.msg217315.html#msg217315
This way every time amount of free memory chages the scene is triggered. If memory is above 50Mb, then it simply does nothing (and according to the "return" condition at the end it does not even update the date of last run).
« Last Edit: February 14, 2018, 09:12:14 am by kwieto »

Offline MtView

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +1/-0
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #12 on: February 11, 2018, 07:35:47 pm »
Kwieto - Thank you for this excellent information and the very clear instructions!

Offline Don Phillips

  • Hero Member
  • *****
  • Posts: 1291
  • Karma: +33/-32
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #13 on: February 11, 2018, 08:29:25 pm »
Kwieto - have you thought of using PLEG? A scene would not have to run, the memory level would trigger an action and you could have an alert sent to you that it executed plus have time stamps when actions executed. 

Here is a sample that clears the cache. When I ran the action, I got a message the cache was cleared and my available memory went from about 71K to about 93k. I'll watch it the next few days and report back.
« Last Edit: February 11, 2018, 08:57:32 pm by Don Phillips »
Vera 3, 1.7.1030, CT101 t-stat, Everspring motion detector, GE/Jasco switch, Leviton outlet, AeonLabs sensor, NuTone garage door, Blue Iris, Sricam SP011, iPhone locator, APCUPSD, VeraMate, VeraAlerts, PLEG, House Modes, Countdown Timer, DVR, Virtual/Multi Switch, Weatherunderground, LB60Z-1 bulb

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +27/-12
Re: Are Scheduled Auto Reboots A Good Idea
« Reply #14 on: February 12, 2018, 07:08:44 am »
have you thought of using PLEG?

I made an approach to PLEG some time ago and my experience is very ambivalent.
I know PLEG is powerful tool, have some advantages over "native" scenes/lua support and has it's devoted users, but I personally feel better with lua than with PLEG.
The major thing with PLEG for me is the way of organization. If you write Lua, you can put everything in one place and make it process-oriented, so you can follow it through from start till the end. All pieces (variables, triggers, conditions, activities) are organized in logical order (check this and if it is like that, then to this, if not then...")
PLEG organization fragments the process into separate "drawers" (one for triggers, another for conditions, schedules, actions...) This way makes it harder to see the general picture of a specific "scene", especially when your "inventory" (defined trigers, conditins, etc) will grow.
It is like going from general picture to the details (top-down) vs. going from details to general picture (bottom-up) when designing scenes.

And, frankly speaking, I don't use that much of a code in my scenes to justify installing two additional plugins (PLEG and PLC) at least for the time being.

Edit:
I looked into the file you attached, and frankly speaking, it is really hard for me to get the picture of your action, there are a lot of things completely not related to clearing cache and everything is in various places (this is exactly my problem with PLEG). I found it a "detective" task to understand how your action is organized. However, I assume there are people for whom that way is easier to understand.

BUT: If I understand your conditions ("cLow_Memory_Available: pMemory_Available < 50000" and "pMemory_Available: memoryAvailable[urn:cd-jackson:com:serviceID:SystemMonitor]") you are checking amount of available memory instead of free memory.
That's wrong, because available memory doesn't take into consideration cached memory (cached = freable = available) while increasing cached memory is the main problem here. So checking available memory won't do the job (see red line at the screenshot above and compare it to the blue one (Free) - esp. around 6th Feb).
From the other hand, your amount of memory can be decreased from other reasons than increasing cache, so I decided to track amount of free memory as it cover all components. Of course if amount of free memory is decreasing because of the other reason than the growing cache, the first stage of the scene (clearing cache) won't help. But then you have reboot command at the next stage.

Edit2: And take care about luup reload variable (I see you track it), as it seems that it doean't show all luup reloads. See second screenshot. Purple line is LuupRestart, blue is LuupRestartTime. In the time period on the screenshot luup was reloaded 30 times according to what you have on blue line (and this is the true value), while only three of them are presented on the purple line
« Last Edit: February 12, 2018, 08:34:10 am by kwieto »