Author Topic: [SOLVED] Vera3 Upgrading failed... not enough space  (Read 1943 times)

Offline johnes

  • Hero Member
  • *****
  • Posts: 591
  • Karma: +6/-7
[SOLVED] Vera3 Upgrading failed... not enough space
« on: August 25, 2016, 05:57:13 pm »
Code: [Select]
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.5M      4.5M         0 100% /rom
tmpfs                    62.2M      6.8M     55.5M  11% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock7           11.0M      6.6M      4.4M  60% /overlay
overlayfs:/overlay       11.0M      6.6M      4.4M  60% /
/dev/mtdblock8            5.5M      5.5M         0 100% /mios

Code: [Select]
2016-08-25_17:49:16 -[21365]- ==fu== Report start firmware upgrade ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== BEGIN MiOS UPGRADE ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== Check if /tmp/newmios.squashfs exists ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== found ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== Check /tmp/newmios.squashfs md5sum ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== md5sum ok ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== Renaming old files created by mios_mount script from: /etc/cmh-firmware/1.7.760/ ==fu==
2016-08-25_17:49:18 -[21365]- ==fu== Remove old squashfs: /etc/cmh-firmware/mios.squashfs ==fu==
2016-08-25_17:49:22 -[21365]- ==fu== Check free space for copy: /tmp/newmios.squashfs to: /etc/cmh-firmware/mios.squashfs ==fu==
2016-08-25_17:49:22 -[21365]- ^[[1;33mWARNING: Waiting 0/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m
2016-08-25_17:49:52 -[21365]- ^[[1;33mWARNING: Waiting 1/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m
2016-08-25_17:50:22 -[21365]- ^[[1;33mWARNING: Waiting 2/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m
2016-08-25_17:50:52 -[21365]- ^[[1;33mWARNING: Waiting 3/3 of 30 sec for the garbage collector to free space on the mini_fo drive^[[1;00m
2016-08-25_17:51:22 -[21365]- ^[[1;31mERROR: Not enough free space, free=4476 KB, required=6136 KB^[[1;00m
2016-08-25_17:51:22 -[21365]- ==fu== Finished MiOS firmware upgrade with failure=1 ==fu==
2016-08-25_17:51:22 -[21365]- ^[[1;33mWARNING: Skipped reporting end firmware upgrade because StartUpgID is NULL^[[1;00m
2016-08-25_17:51:22 -[21365]- ^[[1;31mERROR: Firmware upgrade failed^[[1;00m
2016-08-25_17:51:22 -[21365]- ==fu== Rotating logs... ==fu==
~



Was trying to upgrade to the latest firmware and got a message that the upgade failed.

I looked around the logs and found the above info.

Anyone have any ideas on how to deal with this?
« Last Edit: August 29, 2016, 10:47:04 am by johnes »

Online Isablend

  • Sr. Newbie
  • *
  • Posts: 36
  • Karma: +0/-2
Re: Vera3 Upgrading failed... not enough space
« Reply #1 on: August 29, 2016, 04:36:08 am »
I have exactly the same situation, both /mios and /rom directories reported as being full.   If anyone knows which files can safely be deleted, that would be great.   So far on all the posts I can see its a case of leaving it to support, which is very unsatisfactory.

A simple listing of the files which can be removed would allow this to be resolved.

Offline johnes

  • Hero Member
  • *****
  • Posts: 591
  • Karma: +6/-7
Re: Vera3 Upgrading failed... not enough space
« Reply #2 on: August 29, 2016, 10:35:38 am »
OK - I managed to upgrade.... here's what I did... cleared the vera, (after taking a backup).  Then I did the firmware upgrade, and then restored my backup.

So dumb, but at least I didn't have to wait for literally eternity to get a response from MCV.

Offline JaapvanEkris

  • Sr. Newbie
  • *
  • Posts: 30
  • Karma: +0/-0
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #3 on: August 29, 2016, 11:36:55 am »
Well, my Vera Edge didn't survive the upgrade somehow. It is in a continuous boot-loop. I guess they made a mess of the last firmware???

Jaap

Offline htcheng

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #4 on: August 29, 2016, 06:01:46 pm »
I have had a similar problem, but not quite. I'm getting LUUP startup failure for exit code 16 after "upgrading" to 1.7.855 - will not be doing this again... I can SSH into the Vera, but the UI is on the infinite spinner on the local net and unreachable on the WAN port even though it has an IP and I can ping out to external IPs -- so L2-3 is up and running. Anyone know which directories to clear to free up space? I might resort to reflashing the firmware using the Vera Firmware Flash tool if MCV doesn't respond soon...

I've tried a factory reset and even using the command line to reset the Vera, but it always comes back up in this state.

Hints and suggestions welcomed!

Offline logread

  • Full Member
  • ***
  • Posts: 214
  • Karma: +7/-1
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #5 on: August 30, 2016, 02:45:23 am »
Anyone know which directories to clear to free up space?

/etc/cmh-firmware holds the "huge" (by Vera standards) firmware download file. That file can safely be deleted (somehow the upgrade scripts sometimes fail to do this after an upgrade).

But I do not know if this space issue is the root cause of your problems
Vera Lite UI7, Fibaro FGS-221, FGS-212, FGSS-001, FGK-101, FGWPE/F-101, FGMS-001, Aeon HEM G2, GreenWave PowerNode 6,  Everspring ST-814, SE-812, Swiid SwiidInter.
Raspberry Pi2 Raspbian w/ openLuup. AltUI, SV Thermostat, Virtual Switch, Weather (openWeather), System Monitor (openSysMon), HomeWave.

Offline korttoma

  • Hero Member
  • *****
  • Posts: 654
  • Karma: +21/-5
  • Keep it simple, stupid
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #6 on: August 30, 2016, 05:00:16 am »
@logread
did you delete the mios.squashfs from the location you mentioned on your device? Is it stil working properly?

I have not managed to do one "normal" firmware upgrade since I went to UI7 on my VeraLite.
This last firmware I uploaded to the Veralite via WinSCP and started the upgrade procedure using SSH (got the instructions from Vera support).

I will try deleting the file you mentioned next time there is a new firmware.
- Tomas

Offline logread

  • Full Member
  • ***
  • Posts: 214
  • Karma: +7/-1
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #7 on: August 30, 2016, 04:09:03 pm »
Yes. But again not sure this is the root cause
Vera Lite UI7, Fibaro FGS-221, FGS-212, FGSS-001, FGK-101, FGWPE/F-101, FGMS-001, Aeon HEM G2, GreenWave PowerNode 6,  Everspring ST-814, SE-812, Swiid SwiidInter.
Raspberry Pi2 Raspbian w/ openLuup. AltUI, SV Thermostat, Virtual Switch, Weather (openWeather), System Monitor (openSysMon), HomeWave.

Offline pzamac

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #8 on: August 30, 2016, 05:48:32 pm »
For the record (who knows if it will help anyone) I have an old Vera Lite on which I haven't yet applied the upgrade and this also shows /mios and /rom as 100% full. It is on UI7, but only on 1.7.758.

The UI is still responding and offering to upgrade to the latest UI7. (Ha! I don't think so.)
My Vera Edge is just reporting the Lua engine startup problem since the upgrade. Great update guys. I presume this is why I still haven't heard back from support.

Anyhow, 100% full looks like situation normal on /mios and /rom.

Just my $0.02.


Online Isablend

  • Sr. Newbie
  • *
  • Posts: 36
  • Karma: +0/-2
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #9 on: September 01, 2016, 01:17:22 pm »
I'm getting the error code 16 error also, with the additional info that it can't find file libjpeg.so.61, not sure why that would no longer be present unless the firmware has removed it.   Any thoughts?

Online Isablend

  • Sr. Newbie
  • *
  • Posts: 36
  • Karma: +0/-2
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #10 on: September 03, 2016, 11:40:10 am »
Vera Support (John M, many thanks) has been able to login to my Edge and restore to Factory blank.   As part of that he has fixed the error and all is well in the world again.

I had noticed that in one of the log files (/tmp/log.Init-LuaUPnP) an error message was showing that a library file ?libjpeg.so.62? as reported an issue. which was stopping Lua from initiating.   Space was also cleared up which seems to have been at the root of the problems.

Offline a-lurker

  • Hero Member
  • *****
  • Posts: 687
  • Karma: +35/-7
Re: [SOLVED] Vera3 Upgrading failed... not enough space
« Reply #11 on: March 10, 2017, 09:43:16 pm »
Re: update fails with a message along the lines of "Critical error. Update date failed. Call support". These comments refer to a Vera 3 and firmware 1.7.902:

The firmware updater does an initial check for space and may indicate there is not enough space or Datamine needs to be backed up and removed. If it finds it has enough space it will allow the update to proceed. After the user starts the update, the updater makes a back up file that is stored in /etc/cmh-backup/backup.tgz for its own purposes. It's deleted at some point after the flash occurs. Refer to the updater script /usr/bin/cmh-upgrade.sh where backup-store.sh is executed on line 263. If you regularly refresh the directory /usr/bin/cmh-upgrade.sh you will see the backup pop up during a firmware upgrade.

The backup file created can take up a lot of space, especially if the user has large plugins installed such as AltUI or PLEG. So for example AltUI will take up a total of 1,728 KB (actual measured value) - that's the original files plus the files in the tarred zipped backup. So deleting all the AltUI files in /etc/cmh-ludl/ makes "twice" the difference. Note that on rebooting, the firmware will download any missing plugin files that where originally downloaded from the store (ie not those manually installed - they need to be backed up by you). Do deletions at your own risk.

The  "Critical error. Update date failed. Call support" occurs because the available space is rechecked again after the back up is created. Refer to the updater script /usr/bin/cmh-upgrade.sh line 512. Looks like MCV thinks it's a garbage collection problem when it's actually huge backups causing problems.

It would be useful to check the backup size before the user initiates the firmware update by creating it earlier.

Another problem occurs in that the upgrade log file is destroyed immediately after the upgrade fails - see /tmp/log.upgrade, because the logs are rotated. Refer to /usr/bin/cmh-upgrade.sh line 695. So trying to work out what went wrong is not easy.

Here is a list of all the files (and folders) found in my backup file:
Code: [Select]
..\Backup contents\etc\cmh
..\Backup contents\etc\cmh-ludl
..\Backup contents\etc\cmh-ra
..\Backup contents\etc\config
..\Backup contents\etc\crontabs
..\Backup contents\etc\dropbear
..\Backup contents\etc\ethers
..\Backup contents\etc\filelist.txt
..\Backup contents\etc\lighttpd.users
..\Backup contents\etc\mios_backup.info
..\Backup contents\etc\openwrt_release
..\Backup contents\etc\passwd
..\Backup contents\etc\TZ
..\Backup contents\etc\TZ-full
..\Backup contents\etc\cmh\alerts.json
..\Backup contents\etc\cmh\cmh.conf
..\Backup contents\etc\cmh\datetime
..\Backup contents\etc\cmh\devices
..\Backup contents\etc\cmh\dongle.3.20.dump.0
..\Backup contents\etc\cmh\dongle.3.20.dump.1
..\Backup contents\etc\cmh\dongle.dump
..\Backup contents\etc\cmh\ergy_key
..\Backup contents\etc\cmh\first_boot
..\Backup contents\etc\cmh\HW_Key
..\Backup contents\etc\cmh\HW_Key2
..\Backup contents\etc\cmh\keys
..\Backup contents\etc\cmh\last_report
..\Backup contents\etc\cmh\lighttpd_failures
..\Backup contents\etc\cmh\lock_log_levels
..\Backup contents\etc\cmh\PK_AccessPoint
..\Backup contents\etc\cmh\servers.conf.default
..\Backup contents\etc\cmh\srv_timestamp
..\Backup contents\etc\cmh\sync_kit
..\Backup contents\etc\cmh\sync_rediscover
..\Backup contents\etc\cmh\user_data.json.luup.lzo
..\Backup contents\etc\cmh\user_data.json.lzo
..\Backup contents\etc\cmh\user_data.json.lzo.1
..\Backup contents\etc\cmh\user_data.json.lzo.2
..\Backup contents\etc\cmh\user_data.json.lzo.3
..\Backup contents\etc\cmh\user_data.json.lzo.4
..\Backup contents\etc\cmh\user_data.json.lzo.5
..\Backup contents\etc\cmh\vera_model
..\Backup contents\etc\cmh\wan_failover
..\Backup contents\etc\cmh\zwave_locale
..\Backup contents\etc\cmh\wan_failover\check_internet.hosts

-- all the files in \etc\cmh-ludl\, which are the plugin files listed in the UI under "App--->Develop apps-->Lupp files",
-- are include in the back up. AltUI plugin is very large and PLEG is pretty big as well. They may cause problems.

..\Backup contents\etc\cmh-ra\cmh-ra.conf
..\Backup contents\etc\cmh-ra\keys
..\Backup contents\etc\cmh-ra\keys\cmh-ra-key.priv
..\Backup contents\etc\cmh-ra\keys\cmh-ra-key.pub
..\Backup contents\etc\config\ddns
..\Backup contents\etc\config\dhcp
..\Backup contents\etc\config\dropbear
..\Backup contents\etc\config\firewall
..\Backup contents\etc\config\luci
..\Backup contents\etc\config\luci_devinfo
..\Backup contents\etc\config\multiwan
..\Backup contents\etc\config\network
..\Backup contents\etc\config\ntpclient
..\Backup contents\etc\config\system
..\Backup contents\etc\config\timeserver
..\Backup contents\etc\config\ucitrack
..\Backup contents\etc\config\wireless
..\Backup contents\etc\config\wol
..\Backup contents\etc\crontabs\root
..\Backup contents\etc\dropbear\dropbear_dss_host_key
..\Backup contents\etc\dropbear\dropbear_rsa_host_key

Here's the upgrade log that almost succeeded. Note that the log reckons there is 8 KB to play with. It wasn't enough:

Code: [Select]
...
initial log stuff
...

2017-03-08_12:38:23 -[4498]- ==fu== BEGIN MiOS UPGRADE ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== Check if /tmp/newmios.squashfs exists ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== found ==fu==
2017-03-08_12:38:23 -[4498]- ==fu== Check /tmp/newmios.squashfs md5sum ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== md5sum ok ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== Renaming old files created by mios_mount script from: /etc/cmh-firmware/1.7.902/ ==fu==
2017-03-08_12:38:24 -[4498]- ==fu== Remove old squashfs: /etc/cmh-firmware/mios.squashfs ==fu==
2017-03-08_12:38:28 -[4498]- ==fu== Check free space for copy: /tmp/newmios.squashfs to: /etc/cmh-firmware/mios.squashfs ==fu==


...Note: we have 8 KB free!!!

2017-03-08_12:38:28 -[4498]- ==fu== We have 6120 KB jffs free space and required is 6112 KB ==fu==


2017-03-08_12:38:28 -[4498]- ==fu== Copy MiOS Squashfs into jffs2 partition ==fu==
cp: write error: No space left on device
cp: can't preserve times of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve ownership of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve permissions of '/etc/cmh-firmware/mios.squashfs': No space left on device
2017-03-08_12:38:42 -[4498]- ERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 0 / 3. Wait 30 sec
cp: write error: No space left on device
cp: can't preserve times of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve ownership of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve permissions of '/etc/cmh-firmware/mios.squashfs': No space left on device
2017-03-08_12:39:28 -[4498]- ERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 1 / 3. Wait 30 sec
cp: write error: No space left on device
cp: can't preserve times of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve ownership of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve permissions of '/etc/cmh-firmware/mios.squashfs': No space left on device
2017-03-08_12:40:13 -[4498]- ERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 2 / 3. Wait 30 sec
cp: write error: No space left on device
cp: can't preserve times of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve ownership of '/etc/cmh-firmware/mios.squashfs': No space left on device
cp: can't preserve permissions of '/etc/cmh-firmware/mios.squashfs': No space left on device
2017-03-08_12:40:58 -[4498]- ERROR: Failed to copy MiOS Squashfs into jffs2 partition. Try 3 / 3. Wait 30 sec