Author Topic: Securing and stabilizing the Vera by taking it off the grid  (Read 3758 times)

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing the Vera by taking it off the grid
« Reply #15 on: May 22, 2018, 06:42:17 pm »
Ok then to answer your original question, I just removed it from my vera. I will install it only when I need it as I really prefer it to the Vera UI but I am trying to stabilize the vera so to be on the safe side I am dumbing it down as much as possible to see how stable it can be. The recent rash of report of storage related crashes lead me to monitor this.

Remaining issues:
1. I am still seeing random luup reloads like most of us albeit not very frequently (once every 1-2 days). I have gone over 15 days in the past without one with a lot of modifications on the vera back on 7.0.20 but it seems the new firmware introduced a few new quirks. It reminds me a bit of the old Windows XP days when the computer could not run more than a few hours without crashing/rebooting but things are gradually improving. (much better than the 1-2 times a day when the vera had internet access)
2. One event of the lua socket crash out of the luup engine after I changed my firewall settings and I recovered just with a luup reload. I have no idea why. This time it was not caused by my firewall since it can recover even with it on and it seems to be a fluke I cannot reproduce.
3. The biggest issue I have is I am still having random security sensor triggers (rare, once every few days) and frequent miss of untrip signals from motion and door sensors. They always trip perfectly but often do not untrip. I have notifications setup to troubleshoot this. I suspect it is due to the size of my network. I can actually see the LED on my sensors blink when they transmit, the vera just completely misses them. Again it is random, varies a lot in frequency and affects every sensor type and model I have so it is definitely a controller/zwave network issue.
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline Mike Yeager

  • Hero Member
  • *****
  • Posts: 526
  • Karma: +6/-12
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #16 on: May 26, 2018, 08:49:29 pm »
I have been looking at HomeAssistant, but I'm not sure about it. Still learning, but I'm really no looking to spend a ton of time learning to script different things. That's the main reason I got away from ZWay. While it can be fun, I have a life outside of trying to turn a light on when a certain set of conditions are met. PLEG handles this fairly easily, as does Rules Engine to some extent.

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #17 on: May 26, 2018, 09:24:54 pm »
I am with you. I have learned enough lua now to be dangerous so I don't want to go spend the time to learn python or javascript. I have better things to do. The learning on home assistant is only to configure it. Even though I had to fix a bug in python on one of the components, overall I am not spending any more time on it.

The main purpose of this endeavor is to make the entire system more reliable. I have been doing some bug fixes on some plugins and openLuup and have gotten close to ideal reliability... except for the vera which still remains the most unreliable element of the system:

- It still has random luup reloads every few days I can't figure out. It now has no plugins but still runs all the scenes which are triggered by zwave devices. How much further do I need to dumb it down?
- I still experienced a scene running during daytime while it had a luup.is_night conditional. It is random and not reproducible. I am totally convinced that it is a job completion timing issue inside the vera which I have never seen on openLuup. It could very well be also the reason for the luup reloads
- The good news is, over the past week, I have not had any random tripping of devices and have not noticed any failure to untrip so the zwave side seems to be stabilizing.
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline Mike Yeager

  • Hero Member
  • *****
  • Posts: 526
  • Karma: +6/-12
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #18 on: May 28, 2018, 04:42:23 pm »
I'm okay with Python. I have played with a little bit of LUA that I've picked up on here. I don't mind having to do a little bit of programming here and there at all. I've got a network of Arduino devices using XBee radios that communicate with a central control program on an RPi3. It's heavily modified code from Dave's "Desert Home" project, so it's not my original code and I take no credit for his hard work. I have heavily modified it and rewritten some portions of it to do what I wanted it to do. I've also ported it to Python 3 as Dave is still running Python 2. I did it because I'm going to try to figure out how to integrate it with HomeAssistant...

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing the Vera by taking it off the grid
« Reply #19 on: May 29, 2018, 08:18:29 pm »

3. Cloud backup-I'm OK with this, but wish there was an option to backup to a local network drive instead of, or in addition to, the Vera cloud.


I have good news for you... I have done this:

I created a cron job to run a bash script daily and create a local backup. Just replace the necessary input with yours and then edit your cron jobs. I am doing this on the VM which runs openLuup but it can be done on any computer on your network.
Here is the bash script:

Code: [Select]
#!/bin/bash

backup_location=#path to your local backup folder#
max_number_of_backups=30 #I want to keep a month's worth of backup#

# Setup environment
. /etc/profile >/dev/null 2>&1

# Go to Vera backups directory
cd $backup_location

# Create a date-specific backup name
export BACKUP_NAME=vera_`date +%Y%m%d`.tgz

# Backup ZWave dongle to Vera (this is async and takes time to complete)
wget -O /dev/null "http://#veraip#/data_request?id=action&DeviceNum=1&serviceId=urn:micasaverde-com:serviceId:ZWaveNetwork1&action=BackupDongle&Restore="

# Wait for Zwave backup to complete (actual time depends on # of zwave devices -- do the timing and adjust)
sleep 60

# Backup Vera (whcih will now include ZWave backup)
/usr/bin/wget -O "${BACKUP_NAME}" http://#veraip#/cgi-bin/cmh/backup.sh?external=1

function number_of_backups() {
    echo `ls -1 $backup_location | wc -l`
}


function oldest_backup() {
    echo -n `ls -1 $backup_location | head -1`
}


while [ $(number_of_backups) -gt $max_number_of_backups ]
do
    rm -rf "$backup_location/$(oldest_backup)"
done

exit 0

Edit:

To complete the instructions in case people are less familiar with linux:

1. Create a text file above and name it "verabackup.sh" or whathever you want to call it. Remember to edit the ip address of the vera, the number of files you want to keep and the path to the folder where you want the backups to be saved.
2. Copy that file to a folder you like just remember it's path on the server you want your backups to be.
3. Make this sh file executable by sending the command:
Code: [Select]
chmod a+x verabackup.sh from the folder where this script file is
4. Edit your crontab by executing
Code: [Select]
crontab -e and at the very end of the file add
Code: [Select]
0 4 * * * /path_to_script/backupvera.sh Don't forget to replace the path and the name of your script file. This is for the script to run at 4am everyday. You can change this to whenever you want. The first 0 is the minutes and the 4 is the hour. Do not forget the asterisks.
5. Save the file and you are good to go.
The script is returning a 404-Not Found for this:
# Backup ZWave dongle to Vera (this is async and takes time to complete)
wget -O /dev/null "http://#veraip#/data_request?id=action&DeviceNum=1&serviceId=urn:micasaverde-com:serviceId:ZWaveNetwork1&action=BackupDongle&Restore="

Testing the http call in a browser also results in a 404 error.

I checked and dongle is not being dumped. I've looked through the alleged documentation on the MiOs wiki but can't find anything useful. The main backup is working correctly and generates the proper .tzg backup file.

Any insights  would be most appreciated!

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #20 on: May 29, 2018, 09:31:03 pm »
Mine has been running for a few weeks now. I just checked I forgot the port number for the vera for the dongle call.

it should be
Code: [Select]
wget -O /dev/null "http://#veraip#:3480/data_request?id=action&DeviceNum=1&serviceId=urn:micasaverde-com:serviceId:ZWaveNetwork1&action=BackupDongle&Restore="
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #21 on: May 30, 2018, 12:07:31 am »
D'oh!  I should have seen that...still jet lagged.  :-[

Working now. Thanks!

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #22 on: May 31, 2018, 09:52:18 am »
The backup scripts are running for both VPs. I added a test in the cron script for successful completion of each backup. If there is a failure, the script emails me.  An additional benefit of the local Vera backup is that the server has a RAID array, and also writes a nightly backup to an external USB 3.0 drive. Thanks for the tip!

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #23 on: May 31, 2018, 11:10:08 am »
Glad you got it working. Backups and backups of backups are great indeed.

My system has been settling and strangely I have not had any false positive on security sensors and have had fewer undetected sensor untrip in the past few days. I hope it will stay that way. The Vera still does random Luup reloads every 30-80 hours.

I turned by old Vera edge and upgraded it to the latest firmware and ssh into it to find out that it?s storage drive has a number of bad sectors. I had retired it 2 years ago when it suddenly lost all of its zwave devices out of the blue and kept wanting downgrade to a certain firmware when I contacted support. (Spent hours and hours trying to figure it out at the time). Given how the UI7 works and the very frequent update and rewriting of the user-data.json file, if one has a very large system, the file can start getting very big even if it is compressed. Mine was a couple of 100KB. In spite of being SLC NAND flash with very good endurance, the fact that the Veras have very little storage and a portion of it gets rewritten every few seconds, the NAND cell certainly worn off relatively quickly (~1year). There is just not to many NAND cell to do wear leveling on assuming that the OS or the drive FW does it. This was probably the reason for my impression that I had frequent data corruption and... gremlins in my setup.
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #24 on: May 31, 2018, 02:40:05 pm »
Yup. Backups with backups are good.

I've wondered about wear-leveling, with the system checkpoint occurring every 6 minutes, plus who knows what else? My user_data.json.lzo file is around 182KB. I wonder if some of the odd crashes some folks have had are due to FLASH errors? The OpenWRT docs are vague on what wear-leveling mechanism is used. I assume the NAND flash is directly connected to the SOC, so management is coming from the OS or other software.

Both my systems are stable. The production system, running 1.7.3232, just works. I don't pay much attention to LUUP reloads, but they don't seem to occur very often. The test system, using the latest firmware, runs System Monitor---the last LUUP restart was a couple of weeks ago and may have been me playing with a camera.

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #25 on: June 02, 2018, 02:54:29 pm »
Upgraded the production system to 1.7.3831 this morning, after a couple of weeks of testing on the secondary system. The production system had 66% used in the overlay partition before the upgrade. No issues during the upgrade, other than taking a couple of minutes longer than expected, no doubt due to upgrading the Z-Wave firmware on the co-processor. Everything is working, and there were some icon changes that had to be addressed by changing category/subcategory settings. The icons now do a better job of representing the actual device type. so no problem.

The overlay partition is now at 81%. I can see why folks who started with an almost full overlay had upgrade issues. As has been mentioned before, this upgrade comes with all the Z-Wave upgrade files for all regions. See the attached file. There's ~ 1.8MB of unused Z-wave 6.1 .hex files, and at least 720KB of unneeded Z-wave 4.5 files.  A forum member noticed this and deleted the unnecessary  files and replaced them with 0 length files to avoid Vera automatically replacing them. It would obviously be much easier for the install script to delete the superfluous files, freeing up ~ 2.5MB of precious overlay space.

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #26 on: June 02, 2018, 10:02:13 pm »
Well, for me blocking all communications from the vera from the internet has prevented it from getting any of the files back and I have indeed also deleted all of the unused files.
You are lucky to have such long intervals between luup reloads... Mine still reloads every ~36-45h for no apparent reason.
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #27 on: June 03, 2018, 08:53:57 pm »
I don't monitor LUUP reloads on the production system. I had installed System Monitor last year, and it seems that the frequency of LUUP reloads increased, so I removed it. I'll be keeping a close eye on it for the next few days. The upgraded system seems stable and everything is working.

I'll be following you lead on taking Vera off-line, as soon as I'm home for a couple of weeks. I don't want to make major changes and head out of town as the family doesn't appreciate it when Vera goes off the rails. :(

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1119
  • Karma: +52/-21
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #28 on: June 03, 2018, 09:50:07 pm »
I treat these random Luup reloads as if they were system crash and they really are since they can have dreadful consequences... As some other forum members have experienced, it can destroy your house if one is not careful: If the vera is controlling irrigation or HVAC or even security devices. I have experienced flooding of my yard because the vera rebooted while in the middle of a scene. Imagine your locks forgetting to lock etc... These must be taken seriously. I have written code now in my startup lua to prevent many of the catastrophes but one can't use the vera beyond lighting if it is as unreliable as it is. (the worst you would get is coming to the house lighting up like a Christmas tree out of control). I have now removed all the files associated with the rogue vera apps and the unit seems to still be fine.
I am now also suspecting than there is correlation on one of my sensors having false trips and its battery status changing so it seems that using NiMH rechargeable batteries (low voltage) making it report 1% even when full addresses the problem since it never changes battery level.
openLuup (96 devices, 129 scenes, 20 apps) controlling HomeAss + VeraPlus (133 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 1 app) Bridged to Homekit and Alexa

Offline HSD99

  • Full Member
  • ***
  • Posts: 226
  • Karma: +6/-0
Re: Securing and stabilizing the Vera by taking it off the grid
« Reply #29 on: June 03, 2018, 11:45:28 pm »
All true.

I don't use Vera for anything mission-critical--no HVAC, or locks of any kind. The HVAC system is pretty fancy and has a thermostat that's really a terminal for the processor in the system. The only "smart" T-stat that will work with the system is the vendor's own, and it's ridiculously overpriced, is cloud-based, and gets some of the worst reviews imaginable. The stock non-automated T-stat works great, and there's no reason to have it be part of Vera.

Wireless locks scare the heck out of me, and not just because Vera might hiccup and unlock all the doors. There are too many ways they can be compromised, and the latest security flaw in Z-Wave means all these locks could be compromised. I did automate the garage door when I first started with Vera four years ago. It seemed cool to be able to operate the door remotely, until the door opened all by itself for no apparent reason. I removed the control, but still have the door position sensor so I can check it remotely.

There's an electric wall heater and a wall-mount AC unit in the shop outbuilding. These are both on Z-Wave outlets, but since they take care of themselves, there's no danger if they get turned on unintentionally. The remote control is to either heat or cool the shop before use; convenient but not essential. And there is a scene that turns them both off at 11PM just in case we forget.

And as far as irrigation goes, I had a battery-powered irrigation hose timer fail in the "on" position when I was out of town. You can imagine the result, so no automation for irrigation, Vera or otherwise.

I use Vera for lighting, where failures are annoying but not disastrous. I also am careful not to write scenes with delays, for obvious reasons. I run with as few plugins as possible (Countdown Timer, DelayLight, Wunderground and couple more that are solid) and don't want to interface to external cloud systems.

I bought Vera as an experiment, to see if low-cost home automation was ready for the masses. HINT: It's not, but I'm not convinced that any of the other solutions out there would be that much better. I don't think Vera's hardware is that bad---I've built much-higher performance real-time control systems with much less powerful hardware. Vera's problems are on the software and corporate management side, and in fact Z-Wave itself in 2018 is a primitive, low-performance HW/SW solution. That said, my Vera has been very stable since summer of 2017 when I rebuilt it from scratch (I had carried over a config from a VL) and I'm pretty happy with it.

So I'll follow your lead and take Vera off-line as time permits.  If I decide to get a fancier system in the future, I'll probably look at Homeseer as I have a 24/7 Win 10 box that should handle it with ease.  Thanks again for sharing your experiences.