Author Topic: Frequent restarts  (Read 2907 times)

Offline kartcon

  • Full Member
  • ***
  • Posts: 158
  • Karma: +7/-0
Re: Frequent restarts
« Reply #15 on: December 13, 2017, 05:15:25 pm »
Thanks for the responses. Overall I am VERY happy with my Vera Secure, however yesterdays incident really came out of the blue. I work very hard at making my system as lean as possible while still providing in depth information on exactly how the system is performing, what events it 'sees' and what actions it is taking. Its been 24 hours since the meltdown and all is good, but that does make me wonder if I should do a scheduled FULL RESTART every once in a while. I used to do this with an old router that would drop its connection if left running for days on end. A reboot made it much more reliable. There are several posts that detail the Lua code to do this, but I've never seen a need. Any opinions? - with pro's and con's if possible.

Offline kartcon

  • Full Member
  • ***
  • Posts: 158
  • Karma: +7/-0
Re: Frequent restarts
« Reply #16 on: December 13, 2017, 08:18:50 pm »
Well shortly after my last post where I stated all was well, Major Meltdown #2 occurred. Now even a cold reboot (power cord out, battery disconnected) will not revive my controller. Sent a Tech Support ticket followed by a call to customer service. Waiting on a reply. Vera Secure stuck in a reboot/restart loop. Today in the notification panel I see the following:
Code: [Select]
Luup : Failed to download all plugins. Will retry in 10 minutes.
Panel Manager[19] : Running Lua Startup
No new plug-ins in weeks, so maybe this is an update? Virtual Panel may be the culprit or may be an innocent bystander. Can't tell since the unit never gets fully loaded in order to see what works and what does not. Dead in the water until I hear from Customer Service.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Frequent restarts
« Reply #17 on: December 14, 2017, 04:03:59 am »
Hmm, I probably don't have good news for you. I've lost 2 Edge's during a year for similar reason. Fortunately both were under warranty.

In both cases first I encountered hang-ups where I had to unplug the device to make restart and have it working again. After the reboot I've seen error message "Can't write user data" on the dashboard, time of the alert was the same as when I've lost connection with controller.
After two or three cases like that both controllers hung-up and went into the continuous rebooting loop. Contacted support, they organized a call with me and tried to get controllers working guiding me what to do and trying to get the access logging remotely to my computer. No luck.

In both cases the comment from support was that it seems that it probably can't erase the overloaded data and thus can't finish the reboot.

In the first case, the controller final broke was straight after I asked support for help with this "can't write user data error" - I've got an e-mail with a comment that they removed some temp files, and just after this controller went into rebooting loop.
For second case it happened after some short power outage.
At first I was convinced that the final breakdown was due to the support intervention (like accidentally removing wrong files during the cleaning), after second case I'm not so sure anymore (I have to underline that in both cases support did excellent job to help me).

The first one was also my first Vera, so I can image that it could went less stable due to making the setup partially in a trial-and-error way.
For the second controller I re-created network from the scratch avoiding installation of unnecessary plugins. No major changes in the setup before breakdown, so I can't point any suspected reason.
As I use it in some remote location in the country (say: vacation house), I've installed UPS to secure any possible power outages and plan to put it into a sealed box to prevent any influence from the surroundings (since it is in vacation house, temperatures inside can be very low during a winter and rise relatively quick if heating is turned on - sealed box put into insulated place should minimize influence of such changes).
But I have other electronics working under the same conditions (wireless router, Netatmo weather station and camera) and they work without issues.

I don't know what I'll do if the third controller will broke. Some people here use backup controllers (if first one breaks, you power-on the second and recreate your network from the backup file), but replacing two controllers during a year is not a good ratio, especially when the warranty will end for them. I need a stable solution as I need to keep the home running even if nobody is there for a month or so. I still consider Vera as one of the best options for that purpose, hope my issues are just a "black series".

Offline Thorden

  • Full Member
  • ***
  • Posts: 113
  • Karma: +9/-5
Re: Frequent restarts
« Reply #18 on: December 14, 2017, 08:13:28 am »
What is the trace of a Luup-engine reload in the log? Not sure if I am looking for the right stuff.
/tmp/log/cmh/LuaUPnP.log

Does Verbose logging need to be enabled to see it?

Thanks!

Offline kartcon

  • Full Member
  • ***
  • Posts: 158
  • Karma: +7/-0
Re: Frequent restarts
« Reply #19 on: December 14, 2017, 05:09:18 pm »
Here is my follow up report of the issue:
After emailing and leaving a message w Customer Service I began to poke around in Vera's file structure. I'm no expert by any stretch, but I can certainly see when a particular folder is being filled up with 'junk' files. In this case it was ect/cmh/persist. Every system log event was being written to a one line file in that folder, so in a matter of minutes there were hundreds of 65k files. In examining them, I could correlate each file with one line of log data. I took a BIG RISK and decided to delete the oldest dozen or so files. Almost immediately a dozen more were written. Since it didn't totally crash or stop rebooting I decided to erase the oldest 1000 files. After about 30 seconds of 'thinking' the reboot completed and the system started running normally. Wow! But also I started seeing more new files being written. Within a few minutes the available space was used up and the reboots started again.

Next I deleted ALL the junk files from the folder and once the controller was up and running, I did a full restore to one day prior. The issue continued, so I repeated the process with a restore from two days prior. Same problem. On the 3 try I restored from the first of December (13 days prior). ALL GOOD!! Turns out on Dec 7 something changed on my machine. Not sure what, but something. I was able to restore back to Dec 7 and the system has been stable ever since, and NO files written to ect/cmh/persist in 24 hours!

Tech support did call me back after all this took place. Since the system was now stable, they really couldn't help me diagnose the issue, but did offer to clean up temporary files and folders and remove some unused plug ins. I give them credit for what they did, but I also feel like they really weren't able to get to a root cause due to my actions. To be clear, I have no beefs with Customer service, other than a small language barrier issue. They responded promptly and were willing to help even with all I had done.

Bottom Line: I am still happy with my Vera and the system I have here. This issue clearly was not resolved after my first post even though it seemed to be. Only time will tell if it is now back to normal. I feel incredibly lucky that I did not brick my controller and the only clue I had as to where to look was this one line in the logs:
Code: [Select]
01 12/12/17 18:43:04.293 FileUtils::WriteBufferIntoFile cannot write to /etc/cmh/user_data.json.lzo.new err No space left on device <0x773fb520>

Offline RichardTSchaefer

  • Master Member
  • *******
  • Posts: 10091
  • Karma: +763/-142
Re: Frequent restarts
« Reply #20 on: December 14, 2017, 07:37:59 pm »
@kartcon
Great discovery ...
Did you happen to keep one of the files ... to show to Vera ? Or publish here ?
I have a restart that happens when ever there is activity after some time of idle.
I have thought that it has to do with send an event to the Vera servers, but the connection has timed out, and it reloads to reinitialize the connection.
I plan to find a way to shutdown the event server and see if that help ... I do not look at the Vera Events anyway.

Offline kartcon

  • Full Member
  • ***
  • Posts: 158
  • Karma: +7/-0
Re: Frequent restarts
« Reply #21 on: December 15, 2017, 09:57:05 pm »
Richard,

No, I didn't think to save any of the files before removing them, but they were basically a text version of events typically seen in the Vera logs. Each was named numerically - presumably with some version of epoch time. The good news is in the past 24 hours I have had only one unexplained restart. That's a frequency I can live with. I will consider this problem fixed if we can go a couple more days without another meltdown. Fingers crossed.

Please let me know I you can isolate the cause of the restarts on your system. I would love to get to the point of only having a restart when I schedule one. I suspect there is some common factor, but finding the cause may not be easy especially if detailed accounts can't be provided.

Offline ranneman

  • Full Member
  • ***
  • Posts: 141
  • Karma: +7/-0
Re: Frequent restarts
« Reply #22 on: December 16, 2017, 01:56:23 am »
Happy for you that you "discovered" what caused the issue and fix it. I checked the folder you mentioned to be empty on my VeraPlus. While following this thread I also checked restarts on the VeraPlus. Last restart was December 12th so pretty stable. In general I would say the VeraPlus restarts once every 6 days which is acceptable to me. Today it will restart a lot as I' am going to get Alexa and play around with that. First thing to do rename all my devices to Engish (I' am Dutch)... many restarts...
Vera lite UI7 Europe, Vera Plus, Zwave.me switches/dimmers, Zwave.me plus multi switch, Fibaro switches and multi sensors, Everspring sensors motion, temp., humidity. Honeywell Evohome, Greenwave, Sonos, Brel shades, RFXCOM, MiLights, Alexa.

Offline RLKinMD

  • Sr. Newbie
  • *
  • Posts: 35
  • Karma: +0/-0
Re: Frequent restarts
« Reply #23 on: January 02, 2018, 05:52:10 pm »
My VeraPlus restarts exactly every six minutes. I will open a ticket with Vera to see if they can figure out the problem.
VeraPlus, GE45604 OutdorSwitch, GE45609 switches, Intermtc HA01C receptacles, Evrsprng ST812 FloodDetect & ST814 Temp-HumiditySensor, Aeon SmartSwitch 2ndEd., Foscam FI9821W V2. & FI9805W(2), Schlage BE469WK & NX locks. Apps: PingSensr, Smtp and Push, Nest Tstat & Protcts(3), WebCamDropboxUploader2

Offline kartcon

  • Full Member
  • ***
  • Posts: 158
  • Karma: +7/-0
Re: Frequent restarts
« Reply #24 on: January 02, 2018, 07:34:57 pm »
I would also look at any device that could cycle in a 6 minute period (motion sensors, timers, schedules, scenes or any other device that 'sleeps' for a duration). If this is a new issue, I would go back to a prior backup and see if that solves the problem. If its been ongoing for 24 hours or more, start thinking about your most recent changes (new plug-ins, new scenes, devices or code).

Tech support is good, and in some cases the only way to fix a problem, but I also believe that as end users we are more intimately aware of the state of our system and need to take some responsibility for its condition. I'm not saying "You broke it, you fix it!" but I am saying in an open platform such as Vera, there are lots of ways to break things and we need to understand what progression of events brought us to this point. Sometimes simple logical thinking can shed light on the cause.

Good luck getting your system stable and please follow up once the issue is corrected. I'm sure there are many users who could benefit from your experience.

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 282
  • Karma: +11/-0
Re: Frequent restarts
« Reply #25 on: January 02, 2018, 08:07:31 pm »
I think you are seeing the 6 minute checkpoint user data save. It looks like an error, but isn't, e.g. and will look something like this:

02   01/02/18 16:51:11.199   UserData::TempLogFileSystemFailure start 0 <0x77489520>
02   01/02/18 16:51:11.219   UserData::TempLogFileSystemFailure 4606 res:1
-rw-r--r--    1 root     root            33 Dec 28 14:28 /etc/cmh/HW_Key
-rw-r--r--    1 root     root            32 Jul  3  2017 /etc/cmh/HW_Key2
-rw-r--r--    1 root     root             9 Dec 28 14:28 /etc/cmh/PK_AccessPoint
-rw-r--r--    1 root     root             7 Dec 28 14:28 /etc/cmh/PK_Account
...

 I'd suggest installing the System Monitor plug-in, which will show you actual reboots and LUUP restarts.

Offline Don Phillips

  • Hero Member
  • *****
  • Posts: 1322
  • Karma: +35/-32
Re: Frequent restarts
« Reply #26 on: January 02, 2018, 08:28:44 pm »
My Vera 3 restarts several times a day, last time about 4 hours ago. But as you can see, Vera has not restarted in a while, not since I rebooted.
Vera 3, 1.7.1030, CT101, Everspring motion sensor, GE/Jasco switch, Leviton outlet, AeonLabs sensor, NuTone garage door, Blue Iris, Sricam SP011, iPhone locator, APCUPSD, VeraMate, VeraAlerts, PLEG, House Modes, Countdown Timer, DVR, Virtual/Multi Switch, Weatherunderground, LB60Z-1 bulb, Hue, Alexa

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 282
  • Karma: +11/-0
Re: Frequent restarts
« Reply #27 on: January 02, 2018, 08:53:15 pm »
Don very kindly posted a screenshot of the output of System Monitor, which I should have done.

Offline wilme2

  • Hero Member
  • *****
  • Posts: 1217
  • Karma: +28/-13
Re: Frequent restarts
« Reply #28 on: January 03, 2018, 04:09:56 pm »
#$%@ I thought UI7 was finally stable, and bought a VeraSecure on Black Friday.  I am not in the process of copying over configurations from Vera3/UI5, but tempted to just pull the plug out of the VeraSecure and let it collect dust for a while. 

I have thought that it has to do with send an event to the Vera servers, but the connection has timed out, and it reloads to reinitialize the connection.
I plan to find a way to shutdown the event server and see if that help ... I do not look at the Vera Events anyway.

I assume you have all your devices set to disarmed?  I know in the past we thought that prevented a device triggering from sending the event to the Vera servers...
Long Live UI5!  (OK finally made the switch to UI7 in January 2018, and am pretty happy....)

Offline ninkasi

  • Full Member
  • ***
  • Posts: 210
  • Karma: +12/-4
Re: Frequent restarts
« Reply #29 on: January 04, 2018, 05:10:53 am »
FWIW To find out why my Vera was restarting, I went into apps/develop apps and added in the following to my startup Lua:
Code: [Select]
local filename = '/etc/cmh/restarts.log'
local outf = io.open(filename, 'a')
outf:write(os.date('%Y-%m-%d %H:%M:%S '..os.time()..' '))
outf:write('\n')
outf:close()

I then ssh'd onto the Vera, and checked "cat /etc/cmh/restarts.log" (note this file will continue to grow so will need to be monitored) on a regular basis. When I identified a restart, I then checked /tmp/log/cmh/LuaUPnP.log (note this file gets wiped when Vera reboots, as opposed to luup restarting) to find out what was happening at the time of the restart. In my case the restarts were happening directly after datamine tried to read data from a device. I removed that device from the list of things to log, and I found it still kept trying to read that same device but from a *different* device, and was still causing a restart. Very odd - basically something was corrupt. I decided I needed a reliable Vera more than I wanted historical data so removed datamine.
Home Automation veteran. Rolled own, then MisterHouse and X10, now Vera - mysensor arduino and ESP8266 network - zwave lights, locks, switches - milights - yeelights - google home(s) - Broadlink RM - raspberry pi(s) with various scripts for power and weather station - Logitech Harmony - Imperihome