Author Topic: Frequent restarts  (Read 2917 times)

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #30 on: January 04, 2018, 05:53:41 am »
Very odd - basically something was corrupt. I decided I needed a reliable Vera more than I wanted historical data so removed datamine.

I didn't do such deep investigation as you, but I have very similar experience - frequent luup restarts are (most probably) connected somehow to dataMine2 plugin, especially where there are more activity on the monitored devices.

I see also difference between Plus and Edge - on Plus, when I do some change on the device (i.e. name, or something in parameters) usually shortly after I have blue strip on top communicating about Luup restart. This doesn't happen on Edge, at least not that often. I'm not sure if this is just difference in the reporting (like Edge is just not showing such information), or there is something different in real behaviour of both.
I've installed system monitor plugin to track restarts, but too recently to have any reliable conclusions.

Offline wilme2

  • Hero Member
  • *****
  • Posts: 1217
  • Karma: +28/-13
Re: Frequent restarts
« Reply #31 on: January 04, 2018, 09:55:00 am »
FWIW To find out why my Vera was restarting, I went into apps/develop apps and added in the following to my startup Lua:
Code: [Select]
local filename = '/etc/cmh/restarts.log'
local outf = io.open(filename, 'a')
outf:write(os.date('%Y-%m-%d %H:%M:%S '..os.time()..' '))
outf:write('\n')
outf:close()

I then ssh'd onto the Vera, and checked "cat /etc/cmh/restarts.log" (note this file will continue to grow so will need to be monitored) on a regular basis. When I identified a restart, I then checked /tmp/log/cmh/LuaUPnP.log (note this file gets wiped when Vera reboots, as opposed to luup restarting) to find out what was happening at the time of the restart. In my case the restarts were happening directly after datamine tried to read data from a device. I removed that device from the list of things to log, and I found it still kept trying to read that same device but from a *different* device, and was still causing a restart. Very odd - basically something was corrupt. I decided I needed a reliable Vera more than I wanted historical data so removed datamine.

Here is another way to do it - you don't even have to ssh to pull up the "crash log"...

http://forum.micasaverde.com/index.php/topic,31292.msg226026.html#msg226026
Long Live UI5!  (OK finally made the switch to UI7 in January 2018, and am pretty happy....)

Offline ninkasi

  • Full Member
  • ***
  • Posts: 210
  • Karma: +12/-4
Re: Frequent restarts
« Reply #32 on: January 04, 2018, 08:50:47 pm »
Nice one - better than my hack... ;-)
Home Automation veteran. Rolled own, then MisterHouse and X10, now Vera - mysensor arduino and ESP8266 network - zwave lights, locks, switches - milights - yeelights - google home(s) - Broadlink RM - raspberry pi(s) with various scripts for power and weather station - Logitech Harmony - Imperihome

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #33 on: January 05, 2018, 05:02:40 am »
For me it is unfortunately more difficult, as I have one unit at remote location, where I'm present once per month, more or less, and don't have much time to sit and track Vera behaviour.

Any Ideas how to access logs in such case?

By the way, I use DataMine to track memory reported by System Monitor plugin. And I've found something which seems to be interesting when I compared Edge and Plus data - amount of free or available memory on Edge have very high jumps up and down compared to Plus, for no apparent reason.
When you see the log it look as on the screenshot below

By comparison, memory on Vera Plus is much more smooth, regardless that it has about three times more devices installed. There are also jumps and downs, but within 5k of memory, while in Edge's case it is about 20k of memory. From the other hand Plus memory goes up and down more frequently (there is jump "up" in free memory c.a. every three hours, while on Edge it happens every 6-8hours).



Offline Don Phillips

  • Hero Member
  • *****
  • Posts: 1326
  • Karma: +35/-32
Re: Frequent restarts
« Reply #34 on: January 05, 2018, 07:49:49 pm »
My Vera 3 reloaded LUUP around 8:30 this morning.  Here is my memory graph:
Vera 3, 1.7.1030, CT101, Everspring motion sensor, GE/Jasco switch, Leviton outlet, AeonLabs sensor, NuTone garage door, Blue Iris, Sricam SP011, iPhone locator, APCUPSD, VeraMate, VeraAlerts, PLEG, House Modes, Countdown Timer, DVR, Virtual/Multi Switch, Weatherunderground, LB60Z-1 bulb, Hue, Alexa

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #35 on: January 06, 2018, 03:30:27 pm »
In my case these jumps are not related to luup reloads. There was no luup reload during all period shown on the Edge's graph which I've posted above.

Offline wilme2

  • Hero Member
  • *****
  • Posts: 1217
  • Karma: +28/-13
Re: Frequent restarts
« Reply #36 on: January 06, 2018, 06:42:18 pm »
In my case these jumps are not related to luup reloads. There was no luup reload during all period shown on the Edge's graph which I've posted above.

There is a housekeeping process which runs every few hours to free up memory...  This is OpenWRT level I think - it was there in previous firmwares (UI5, etc)...
Long Live UI5!  (OK finally made the switch to UI7 in January 2018, and am pretty happy....)

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #37 on: January 09, 2018, 07:00:07 am »
It seems that something is wrong with this process on Edge.
And it seems that it has something to do with Datamine2.
Yesterday my Edge stopped responding and needed manual reboot. After this process all datamine channels were cleared and Datamine is ignoring USB memory attached to the controller (the memory shown on the Datamine Panel is Vera memory, not the USB stick).
And according to System Info plugin, there is much more memory free after that.
Seems that Datamine was buffering data on internal memory instead of saving it to USB stick, and this caused so high jumps in amount of memory available, compared to readings from Plus.

Offline Thorden

  • Full Member
  • ***
  • Posts: 113
  • Karma: +9/-5
Re: Frequent restarts
« Reply #38 on: January 09, 2018, 08:30:46 am »
I uninstalled DataMine after reading several post here pointing to that. For me, that didn't change anything.
My VeraPlus still restarts 10-12 times a day (Luup engine restart). I now log it to Syslog to be able to track it.

The restarts every second hour or so happens regardless of sensors being tripped or any other device activity. It happens when nobody is home too. I have several plugins that are active doing stuff, like Heliotrope, Weather Underground, Squeezebox and DEMII. Will continue to investigate and see if I can find a pattern.



 

Offline wilme2

  • Hero Member
  • *****
  • Posts: 1217
  • Karma: +28/-13
Re: Frequent restarts
« Reply #39 on: January 11, 2018, 09:20:57 pm »
I uninstalled DataMine after reading several post here pointing to that. For me, that didn't change anything.
My VeraPlus still restarts 10-12 times a day (Luup engine restart). I now log it to Syslog to be able to track it.

The restarts every second hour or so happens regardless of sensors being tripped or any other device activity. It happens when nobody is home too. I have several plugins that are active doing stuff, like Heliotrope, Weather Underground, Squeezebox and DEMII. Will continue to investigate and see if I can find a pattern.

Try the code I linked to above that automatically grabs the last ~30 seconds of the luup log before a crash to help ID the problem.  While I am new to UI7, I troubleshot my Vera3/UI5 extensively and found the problem it had with restarts using that code.  You can do it straight from the log, but it is easier if you use something to grab it for you.  Also see ElVira.xls spreadsheet process that you can use to dump the log around a crash (Luup engine restart) direct to Excel...  http://forum.micasaverde.com/index.php?topic=30265.0
Long Live UI5!  (OK finally made the switch to UI7 in January 2018, and am pretty happy....)

Offline Thorden

  • Full Member
  • ***
  • Posts: 113
  • Karma: +9/-5
Re: Frequent restarts
« Reply #40 on: January 12, 2018, 02:32:52 am »
I did copy the startup lua scripts that captures log on restarts. Works like a charm.
Have tried to analyze what is going on from the logs, but no success. Have verbose logging enabled per request from support, so it generates a lot of stuff that's hard to understand. I can't see a pattern, and I don't know what to look for.

Thanks for the tip about ELVira, will look into it!   

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #41 on: January 22, 2018, 03:34:33 am »
I've found the issue why Edge was drained from memory so fast, it was due to enabled function of saving logs to USB (I didn't do it when installing USB stick to Vera, but I set the system restoring backup from other controller, probably it was set there). After unchecking this, memory "jumps" became similar as on Plus.

But now I have strange issue with Plus: For last days it was pretty stable (no luup reloads within couple of days), but since yesterday luup reloads 4-5 times per hour, which is every 12-15 minutes.
I saved some log file, but I don't really know what to look for in it.
Any ideas?


Offline akbooer

  • Master Member
  • *******
  • Posts: 6153
  • Karma: +275/-69
  • "Less is more"
Re: Frequent restarts
« Reply #42 on: January 22, 2018, 04:04:44 am »
USB logging certainly has an impact.  See, for example, this very old post with a memory usage plot before/after...

http://forum.micasaverde.com/index.php/topic,25564.msg180765.html#msg180765
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #43 on: January 22, 2018, 04:25:00 am »
I know, and I thought it was disabled (I didn't check it after restoring backup, but at that time of restoring there was not USB stick in the port, seems that system checked logging as "ON" regardless of that).

For now I am more concerned what is happening to the controller which reloads luup every 12-15 minutes... :(

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #44 on: January 22, 2018, 05:30:19 am »
Here is screenshot from DataMine plugin - there was about 1 luup reload between 16th and 21st Jan, then it is reload after reload (with stabilization for 1h or so after I did vera reboot) since that day.