Author Topic: Lots of "cannot write user data" errors  (Read 372 times)

Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Lots of "cannot write user data" errors
« on: June 13, 2018, 05:32:07 am »
Hi.

I got a lot of "cannot write user data" lately. So I investigated, since most of them are happening around 2 or 3 am.

Here's what I found on my logs

02           06/12/18 2:33:34.911    [33;1mfinished check for exceptions[0m <0x72c8a520> (repeated 10 times, emphasis mine)
02           06/12/18 2:33:34.912     [33;1mOL: (0x12fc444) (>362267) ThreadedClass ../JobHandler/JobHandler.cpp l:110 time: 2:33:34a (0 s) thread: 0x77ae5000 Rel: N Got: Y[0m <0x72c8a520>
02           06/12/18 2:33:34.912     [33;1mfinished check for exceptions[0m <0x72c8a520>
02           06/12/18 2:33:34.913     [33;1mOL: (0x12fc444) (>362268) ThreadedClass ../JobHandler/JobHandler.cpp l:59 time: 2:33:34a (0 s) thread: 0x77ae5000 Rel: Y Got: Y[0m <0x72c8a520>
02           06/12/18 2:33:34.913     [33;1mfinished check for exceptions[0m <0x72c8a520>
02           06/12/18 2:33:34.914     [33;1mOL: (0x14c4b50) (>362269) SendReceiveMutex ../ZWave/ZWJob_GenericSendFrame.cpp l:105 time: 2:33:34a (0 s) thread: 0x77ae5000 Rel: Y Got: Y[0m <0x72c8a520>
02           06/12/18 2:33:34.914     [33;1mfinished check for exceptions[0m <0x72c8a520>
02           06/12/18 2:33:34.914     [33;1mOL: (0x14c4b0c) (>362272) SendMutex ../ZWave/ZWaveSerial.cpp l:510 time: 2:33:34a (0 s) thread: 0x77ae5000 Rel: N Got: Y[0m <0x72c8a520>
02           06/12/18 2:33:34.915     [33;1mfinished check for exceptions[0m <0x72c8a520>
02           06/12/18 2:33:34.915     [33;1mOL: (0x14c4b50) (>362273) SendReceiveMutex ../ZWave/ZWaveSerial.cpp l:511 time: 1:00:00a (1528763614 s) thread: 0x77ae5000 Rel: Y Got: Y[0m <0x72c8a520>
---------------exited-------------
---------------after sleep-------------
---------------after kill -9-------------
---------------exited------------- (repeated more than 200 times, emphasis mine)

And then the engine crashed and I had to perform a reboot (by replugging the power cord again).

Support is insisting that it might be related to some faulty disk sector, but I suspect something to be more related to the Zwave heal. Anyone got similar problems? Thanks.
« Last Edit: June 13, 2018, 02:54:03 pm by therealdb »
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lost of "cannot write user data" errors
« Reply #1 on: June 13, 2018, 06:58:34 am »
Search the forum there is a lot of topics about such error, or issues with accessing the controller via webservers (which in some cases precede "can't write user data" error).

From my experience, the"can't write user data error" is related to lack of memory, RAM or Storage. If either of them is filled too much (in my experience: free RAM is less than 5000K or free Storage is less than c.a. 250k) controller start choking and becomes unresponsive.
Usually you need to do a reboot to clear the stuffed memory enough to make it work again.

You can make a routine to check RAM and clear it if needed (I posted a script doing that job some time ago), you can do the same for Storage.



Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Re: Lost of "cannot write user data" errors
« Reply #2 on: June 13, 2018, 09:18:15 am »
I?m not low on storage or ram. Support told me that it?s probably related to a plug-in or something. We?ll see.
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lost of "cannot write user data" errors
« Reply #3 on: June 13, 2018, 10:28:18 am »
I?m not low on storage or ram.

You're sure?
What was amount of RAM and Storage stright before the error occured?

Support told me that it?s probably related to a plug-in or something. We?ll see.

They usually say that, though I often think that it is only part of the truth.

Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Re: Lost of "cannot write user data" errors
« Reply #4 on: June 13, 2018, 02:51:03 pm »
More than 50000 for ram (I have a scene watching sys Monitor) and I have USB for logging and managed to get 60% for overlay.

So, I suspect it's something more, but support seems to be elusive.
We'll see.
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station

Offline 3rdStng

  • Jr. Member
  • **
  • Posts: 93
  • Karma: +1/-0
Re: Lots of "cannot write user data" errors
« Reply #5 on: June 13, 2018, 03:41:32 pm »
I was getting this error until I cleared up some of my storage space. If I SSH'ed into my controller and ran 'df -h' I had a few partitions that were 100%. I still do, even after I cleaned up some files and support cleaned up some too.

Within the /etc/cmh-zwfw directory, there are a bunch of firmware's for other languages. These files do auto download if you just clear them out. I removed them, actually moved them to /tmp/somewhere. However the /tmp location I put them at didn't survive a reboot. Anyway, so remove the file but then recreate the file by touching it. 'touch zwave_zm5304_fw_4.05_au.hex' for example. You can see in my attached image that the file size is 0, except for my US firmware. After clearing up the space, I no longer was getting the error and other things that appeared broken were also resolved.
« Last Edit: June 13, 2018, 03:44:13 pm by 3rdStng »

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lost of "cannot write user data" errors
« Reply #6 on: June 13, 2018, 03:45:25 pm »
More than 50000 for ram (I have a scene watching sys Monitor) and I have USB for logging and managed to get 60% for overlay.

Then maybe they are right about bad sector.
In my case 100% of such errors were related to memory issues (mysteriously deep drain of RAM or really stuffed storage).

Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Re: Lots of "cannot write user data" errors
« Reply #7 on: June 13, 2018, 03:59:24 pm »
We'll see. I originally planned to buy a new one, but I'm starting to wonder if the amount of time required to rebuild the system could be similar to move to a more stable system.
My Vera seems to know that I'm traveling and to mess things up when I can't reach it...
In the meantime I ordered a tplink switch to reboot it in case of emergency.

What seems strange to me is the kind of errors I see in the logs.
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station

Offline 3rdStng

  • Jr. Member
  • **
  • Posts: 93
  • Karma: +1/-0
Re: Lots of "cannot write user data" errors
« Reply #8 on: June 13, 2018, 04:03:30 pm »
I'd be interested in hearing more about how the tp-link switch will reboot the Vera. Unless your Vera is running over a PoE port, I'm not familiar with how a network switch will restart it. (Unless it is not a network switch you are referring to.)
Me personally, I scheduled a weekly reboot scene running Lua code to get it done.
Code: [Select]
os.execute "reboot"

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lots of "cannot write user data" errors
« Reply #9 on: June 13, 2018, 04:24:35 pm »
We'll see. I originally planned to buy a new one, but I'm starting to wonder if the amount of time required to rebuild the system could be similar to move to a more stable system.

If you talk about moving from one controller to the other, you can save backup from one and restore it on the other, and it works.
I don't know how with older controllers, but migrating between Edge and Plus works both ways without issues.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lots of "cannot write user data" errors
« Reply #10 on: June 13, 2018, 04:32:42 pm »
I'd be interested in hearing more about how the tp-link switch will reboot the Vera.

TP-Link has also WiFi power switches in their offer, similar to WeMo.
I have spare controller so I simply attached two to separate WeMo switches, both with configuration loaded from the same backup file. One is powered off. If everything fails, I can switch one unit off and turn the other on, using that unit till I'll be able to reach them and do proper maintenance.

Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Re: Lots of "cannot write user data" errors
« Reply #11 on: June 13, 2018, 04:34:52 pm »
It's a wifi switch, similar to a sonoff.
I have a raspberry running other things and my own bot/integration, so I'll just write a check and ping the Vera every x minutes.
I already reboot it daily, but it got stuck right after the last reboot.
I managed to get 11 days of 'uptime' this way. I was at only 2 or 3 before.
@kwieto: I know. I'm not sure I will trust a system going nuts after 3 years, two of which doing almost nothing. We'll see.
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station

Offline kwieto

  • Hero Member
  • *****
  • Posts: 543
  • Karma: +21/-11
Re: Lots of "cannot write user data" errors
« Reply #12 on: June 13, 2018, 05:22:57 pm »
@kwieto: I know. I'm not sure I will trust a system going nuts after 3 years, two of which doing almost nothing. We'll see.

3 years is quite a good result, supposing that your system didn't have any major crashes during that time. Not only for Vera, but for any kind of "technology" solution.
I don't defend Vera here, as I'm also considering migration after my last troubles.
So far I was able to restore my controller from breakdowns and make it operational again, after last measures it works without any issues (the question is how long?). Till I have spare one as a backup, I'll probably stay, at least if I won't have troubles making daily use difficult.

Offline therealdb

  • Jr. Member
  • **
  • Posts: 95
  • Karma: +0/-0
  • Automate all the things!
Re: Lots of "cannot write user data" errors
« Reply #13 on: June 18, 2018, 02:42:17 am »
3 years are not bad, but in the first 2 I had my Vera as a basic test environment before moving to my new house, where I fully deployed my solution.

anyway, I got it again, same behavior as in my log trace. support doesn't seem to help. I don't know if I'm going to buy a replace unit, or just switch to a different platform. time to evaluate my choices.
Vera Edge, Fibaro FGRM 222 (12), Fibaro FGS 223 (20), Fibaro FGS 222 (5), Fibaro Universal Binary Sensor (2), Fibaro Plug (3), NeoCoolCam Door Sensor (3), NeoCoolCam PIR (1), Nest (3), Raspberry PI running my own integrations, Harmony Hub, OpenSprinkler, Personal Weather Station