Author Topic: Vera extroot  (Read 3224 times)

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Vera extroot
« on: August 24, 2018, 11:59:39 am »
For those who are running out of storage space on their vera and have an old SSD laying around, you can extroot the vera by moving the entire openwrt file system onto an the external drive.
I found in my case, that it was an essential step to stabilize the vera. Though I strongly recommend to do this with an SSD (which has built in RAM for cache and ECC), one could do this on a USB stick as well.
I do not like using the USB stick because they are slow (yes in spite of the USB2.0 interface limitation, most USB flash stick are even slower) and are failure prone. Indeed, they do not have wear leveling, Garbage collection and are made of lower grade NAND flash. So you are warned. Also I want to give a disclaimer that though this has worked wonders for me, I am not responsible for any loss of data or bricking. My script is quite safe as it should default back to loading from the onboard SLC NAND storage if the USB fails but you never know.

The process involves therefore a USB-SATA2 or SATA3 cable and an SSD.
1. Plug it to the USB port of the vera and
2. go into the vera settings/log section and enable logging to USB. The vera will initiate, partition and format the SSD and reboot. Your logs will be stored in the first partition which will be 512MB. There will be a second unformatted partition on the rest of the drive.
3. Once rebooted, ssh into the vera and run
Code: [Select]
df -hYou should see how much space you have on your drive (the overlay partition)
4. You can upload the script attached here with SCP and run it under SSH into the vera or SSH into it and run the commands from the script one by one.
5. The system should reboot at the end of the script so once it is rebooted (it may double reboot actually) SSH back into the vera and run
Code: [Select]
df -h
This should verify that you succesfully booted from the external SSD. For me it looks like this:
Notice the 55G available for the rootfs partition and 0% utilization. I ran this on a 64GB Crucial C300 drive.

Code: [Select]
Filesystem                Size      Used Available Use% Mounted on
rootfs                   58.1G    102.2M     55.0G   0% /
/dev/root                10.0M     10.0M         0 100% /rom
tmpfs                   124.8M      1.2M    123.6M   1% /tmp
/dev/sda2                58.1G    102.2M     55.0G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1               487.8M     30.4M    427.9M   7% /tmp/log/cmh
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /storage
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /etc/cmh-firmware
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /etc/cmh-backup
/dev/mtdblock9            9.8M      9.8M         0 100% /mios

End note. You can get creative and do all kinds of things with this. For example add a swap file so the vera can have more available RAM etc...

Edit: I have attached the latest version of the script. I will update it here as it evolves.
« Last Edit: January 22, 2019, 10:24:15 pm by rafale77 »
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1046
  • Karma: +167/-1
Re: Vera extroot
« Reply #1 on: August 24, 2018, 12:17:31 pm »
Great job! This is really useful.

One detail: the script should probably include a mkdir -p /mnt/sda2 prior to the mount. If the mount point doesn't exist, the mount will fail and everything after will go down with it.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #2 on: August 24, 2018, 09:59:54 pm »
You are correct of course. I probably already mounted it when I wrote this script. Sorry about that. Attached the corrected one.
« Last Edit: August 24, 2018, 10:09:51 pm by rafale77 »
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Online HSD99

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +17/-0
Re: Vera extroot
« Reply #3 on: August 26, 2018, 12:20:42 pm »
Great idea! Does your SSD run from the USB power supplied by Vera, or does your adapter have an external power supply?

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #4 on: August 27, 2018, 07:37:49 am »
Mine runs from the vera power. I did not need to add an auxiliary power. These SSDs are not very power hungry.
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Online HSD99

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +17/-0
Re: Vera extroot
« Reply #5 on: August 27, 2018, 01:18:47 pm »
Thank you.

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2474
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: Vera extroot
« Reply #6 on: August 28, 2018, 04:11:11 am »
Nice idea, thanks for sharing.

Will this cause any issues with firmware upgrades ?

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #7 on: August 28, 2018, 11:02:40 am »
Nice idea, thanks for sharing.

Will this cause any issues with firmware upgrades ?

This depends a lot on whether the firmware upgrade involves an OS upgrade. Most don?t so it should not have any issues. I upgraded and downgraded between 7.0.27 and 7.0.26 and did not have any issues. The key is whether the fstab file gets changed.
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Online HSD99

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +17/-0
Re: Vera extroot
« Reply #8 on: August 28, 2018, 05:14:15 pm »
Did this on my test VP:
Filesystem                Size      Used Available Use% Mounted on
rootfs                  116.7G    106.2M    110.7G   0% /
/dev/root                10.0M     10.0M         0 100% /rom
tmpfs                   124.8M    344.0K    124.4M   0% /tmp
/dev/sda2               116.7G    106.2M    110.7G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda1               493.6M      1.3M    456.0M   0% /tmp/log/cmh
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /storage
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /etc/cmh-firmware
/dev/mtdblock10          50.0M      1.5M     48.5M   3% /etc/cmh-backup
/dev/mtdblock9           10.0M     10.0M         0 100% /mios

Worked like a charm. Thanks again!

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1046
  • Karma: +167/-1
Re: Vera extroot
« Reply #9 on: August 31, 2018, 10:01:40 am »
Another wrinkle I just discovered is that there's hardcoded path in the test/calculation for the maximum log file size in Vera's script that handles log rotation (see /usr/bin/Rotate_Logs.sh). This causes the max log file size to always be computed based on the /tmp volume/mountpoint, rather than whatever volume is actually configured to store the logs, which is usually mounted below that at /tmp/log/cmh, and is in the script provided earlier in this thread. So, although you may have some large partition made available for logs, it does not appear that an individual log would come close, and the total size of all logs (including preserved rotated copies) would be only 10x that by default. My Plus has a 62M /tmp volume, so, for example, the maximum size of all log files would be 10 * (15% * 62M) = 93M, as no one log file copy would much exceed 9.3M, regardless of the size of my external log partition (default config is use max of 15% of /tmp for logs, keep 10 copies).

A fix for this would be to mount the log partition at /tmp instead of /tmp/log/cmh, a simple change to the fstab in the same way as rafale77's other work in his script. This would also preserve other logs and may be quite useful when things go really pear-shaped, because even with an external USB, those still go to volatile storage (/tmp is a ramfs by default) and are lost on reboots otherwise.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #10 on: August 31, 2018, 10:26:17 am »
Thanks Rigpapa. I had not noticed the tiny size of the logs. With the USB stick I had which was 16GB, I had always wondered why the max utilization of that partition was 10%.

openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Online HSD99

  • Sr. Member
  • ****
  • Posts: 341
  • Karma: +17/-0
Re: Vera extroot
« Reply #11 on: August 31, 2018, 10:27:52 am »
Funny, I was just poking around /usr/bin/Rotate_Logs.sh to see why Vera didn't seem to be taking full advantage of the 512 MB log partition.

Great catch---thanks!

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #12 on: August 31, 2018, 10:55:43 am »
My alternate solution:

go to /usr/bin
and run this

Code: [Select]
cp Rotate_Logs.sh ./

Then edit the the Rotate_Logs.sh file (I installed nano on my vera so I used nano. I think you can use vi)

Code: [Select]
vi Rotate_Logs.sh
scroll down to find this section of the script:


Code: [Select]
# maxLogsDiskUsage is the number of KB.
if [ $maxLogsDiskUsage -eq 0 ]; then
        # If we don't have a value defined in config for maximum size of the logs,
        # we should allow them up to 30% of the /tmp/log/cmh size.
        maxLogsDiskUsage=$(df | awk '{ if ($6 == "/tmp/log/cmh") printf("%d", $2 * 0.3) }') # KB
        # In case something went wrong and we didn't get the 30%, define a static value.
        [ -z "$maxLogsDiskUsage" ] && maxLogsDiskUsage=4096 # KB
fi
and change the folder to /tmp to /tmp/log/cmh like I did. I also increased the max size of the logs to 30% in the formula
« Last Edit: August 31, 2018, 11:00:56 am by rafale77 »
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1046
  • Karma: +167/-1
Re: Vera extroot
« Reply #13 on: August 31, 2018, 10:59:02 am »
My alternate solution:

Also works, of course, but does not preserve a number of other system logs stored higher up in /tmp across reboots.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1690
  • Karma: +93/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Vera extroot
« Reply #14 on: August 31, 2018, 11:41:32 am »
Ohh I see what you are saying. It is the change in the mounting structure. Indeed changing the original sda1 mounting to /tmp might work better.

Will look into that after I am done with my stability test.
openLuup (78 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.