Securing and stabilizing the Vera by taking it off the grid

[rigpapa]

Thanks Rigpapa,

It's very insightful. The reason why I am pointing to hardware is because... I managed to make the whole unit reboot (not just the luup engine) by SCPing a large amount of small files from the vera to server. I was cloning the entire file system and passing it through the socket.
The Extrooting process which does the same but through USB never caused this so I concluded that the socket was the problem.

That sounds like a dropbear bug, more a core firmware bundle issue than anything LuaUPnP. I was going to suggest using cpio or tar. Unfortunately, it appears that cpio isn't available, which is a shame, since it does a better job with links on restore. But tar is there and usable, but like scp, would come at the expense of restoring links as files rather than links (so the restore consumes more space than that which was backed up).

Code: [Select]

tar -c -z -f - /overlay | ssh patrick@mylocalserverip 'cat >vera-overlay.tgz'

[rafale77]

I did finally manage to clone the file system. Thank you. It just took me to clone folders one at a time and took a little more time doing it. You might be right that it could be a dropbear issue but again I copied back this same files through SCP to another machine (freebsd) running dropbear without any problem. Granted it was going the other direction.

Also, what do you use to monitor the CPU utilization? I did some research on what linux reports and what system monitor, ie top I believe is not the best indicator of utilization spikes.

[rigpapa]

... (freebsd) ...

Ah, so rare to meet another who has the religion...

[rafale77]

hahaha  . For testing, I am running the "luup engine" through QEMU... I worry that because the LuaUPnP doesn't queue commands much, it may either drop them or just freeze up so it isn't so much the CPU load over a period of time that matters, it is the peak. It maybe the reason why you are observing inconsistent behaviors.

[HSD99]

The MediaTek MT7621 SoC's dual-CPU cores are pretty powerful, (1.6DMIPS/Mhz) as is the Gigabit Ethernet interface with hardware acceleration. I'm with @rigpapa---this is a software problem. Unfortunately, that leaves a lot of places to look...

[rafale77]

I had the same opinion and bias as well and indeed the luup engine has quite a bit to look at... my test will validate. I don't think it needs a lot of memory. With only two plugins (altui and system monitor) it ranges from 50-80mb used. But before openLuup, I had it run up to 160-180mb used with all my plugins. The look at the storage situation made me wonder if other things are amiss. My busybox upgrade for example made a big difference...though I had to patch some of the original mios scripts to make system backup or leds work again. Unfortunately without the source code for LuaUPnP, we can?t fix things and I know many of them should be quite easy fixes and should not take a whole lot of work and time. So I am looking at the next thing I can.

Also note that only the vera secure has a dual core MT7621A. The Vera Plus has a single core MT7621S.

[rafale77]

February 1st and got the midnight luup reload. Now I can start my test...
I will see if the hardware is indeed a factor. Hoping to get one month before it reloads. Zwave and Zigbee are all setup.

[mblindsey]

First, thanks to all the contributors on this thread!  There is an amazing amount of good info here. 

I decided to take the plunge and run the script posted by rafale77 to mod my vera (posts #175/176).  Quick notes:

- Set up extroot using an 64gb SSD drive w/usb->sata controller

- Vera running 1.7.4001, and at the time of this post, the only noted tested versions of the mods in this thread I see are: 1.7.26 and 1.7.27 (post #169)

- I did get two package install errors, and installed both manually, without error. specifically:

* opkg_install_pkg: Package libpam md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'.
* opkg_install_pkg: Package libsqlite3 md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'.

- My full install log is attached.

- I installed the backup script in post #19.  I haven't tested it yet out of cron, but I did need to install bash.

- I'm using Home Assistant to talk to Vera and running automations with Node-Red

I'll report back with successes and failures.

Cheers!

--Michael

[rafale77]

- Vera running 1.7.4001, and at the time of this post, the only noted tested versions of the mods in this thread I see are: 1.7.26 and 1.7.27 (post #169)

- I did get two package install errors, and installed both manually, without error. specifically:

* opkg_install_pkg: Package libpam md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'.
* opkg_install_pkg: Package libsqlite3 md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'.

On the vera firmware version numbering: sorry for the confusion. mios uses version and build in their numbers. I meant to say 7.0.26 and 7.0.27 which are UI versions. 1.7.4001 is the released build for the vera plus corresponding to version 7.0.26.

On the opkg errors. these two packages are secondary and probably will not affect you. They were accompanying packages which came with the updated version of lighttpd and the upgrade is optional. The problem appears though as if the download was somehow corrupted so it is a good thing they did not get installed. Can ignore them for now and see how things go.
I have applied these same mod patches to my emulator test unit currently running on a newer linux kernel (3.18.23 vs 3.14.10) and newer version of operating system (openWRT Chaos Calmer 15.05.1 Vs Barrier Breaker 14.07). I have never seen a luup engine crash since so I am optimistic I will get a month without crash out of my test.

Digging further into the zigbee dongle... the vera uses EZSP (Ember Zwave stack) so any USB stick supporting this standard stack will work on the vera...

[rafale77]

Well so much for my test. It failed after 11 days after 2am which I suspect is after a nightly heal which oddly is not reported on ALTUI.
So it doesn't seem like the vera hardware is much of a limitation to the stability. It seems like the zwave network can be a source of luup reloads as well mostly due to the autoconfiguration of the vera. Any sort of changes to the network, example: Aeon HEM sending a signal saying it can support 3 poles while it only has 2. Can cause the vera to automatically add a device, try and fail to configure it and then either force a reload or crash.
The automation around the device configuration is a big problem and in particular the child device creation handling as we all discovered with ghost child devices. Disable auto configuration in the zwave menu is not sufficient. We need to disable it for each device as well to prevent some of it from happening.

[Mike Yeager]

Wow... You have put far more work into trying to make something stable than I ever would have...

[rafale77]

Yes, it's been an agony... Made a lot of progress and the instability I am facing might be benign but I am trying hard to make the vera work... even without vera hardware. I am at a point of being reasonably happy with it.
I have a couple of suspicions about why luup reloaded after the nightly heal yesterday: I moved a couple of my motion sensors and they might have updated neighbor nodes lists or it is what I described below with the handling in particular of aeon devices automatically creating child devices even after they have been configured. Generally though, when this happens, the reload occurs during the zwave backup, not during the heal so it is more likely the former.

Some people go as far as rebooting the vera every day. it isn't acceptable to me because of the unpredictable times I am up in the house and how long the reload takes ( 35 - 45s and, almost 1m for a reboot) which are times all data is lost. Luup reloads should be acceptable only during setup and configurations (user induced).

Summary of Vera instabilities and solutions:

1. Memory leak and shortage:
       Move plugins out of the vera and to openLuup to reduce the amount used
       Upgraded lighttpd (actually available on the mios opkg repo) to address performance and potential memory leak
       Extreme in my case, ran the vera luup engine on non vera hardware
2. NAND flash corruption and flash partition  full
      ExtRoot to an external flash drive, short term alternative is my script to use the storage partition for non-operating system folders.
      Extreme in my case, ran the vera luup engine on non vera hardware
3. CPU and network socket crash/lockups
      Move plugins to openLuup
      which was recently updated to allow some delay in its communications to prevent overloading the vera socket
      Killed the networkmonitor program running in the background.
4. mios server (outages and errors from the server can cause the vera to crash and reboot)
      Killed all tunnels and services connecting the vera to mios servers except the event server which I have not yet been able to disable.
5. zwave network updates and errors
     Disable auto configuration on every device after the device configuration has been completed.
     I wish I could disable nightly heals but I still see some occasional auto heal I believe triggered by the zwave chip itself

[Catman]

Rafale, just a random thought (probably answered elsewhere) but do you have any remote access into your Vera? If so, might I ask how you secure it? From an app, for example?

Cheers

C

[rafale77]

Yes I use Homewave as my phone app and have remote access through a VPN through a dyn DNS. I had explored multiple VPN solutions and settled on using openVPN. It isn't just an access to my vera/openLuup. It gives full access to my network including my security NVR, my NAS etc... It adds the extra step of having to log into the VPN but... I am not relying on anyone else's server and I am not sending them any of my data.

[Catman]

Cheers!

Probably not a high WAF (at least for my W)

C