We have moved at community.getvera.com

Author Topic: user_data.json  (Read 2250 times)

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
user_data.json
« on: March 04, 2018, 07:43:36 pm »
I'm not sure if this a feature request or a bug report, so I'll just describe what has occurred and what I see as a possible solution.

I have had several occasions where I boot up my openLuup box and ALTUI has fails to start.  When I look into into the logs, I usually see something like the following:

2018-03-03 20:17:25.366   :: openLuup STARTUP :: /etc/cmh-ludl
2018-03-03 20:17:25.366   openLuup.init::        version 2018.02.25  @akbooer
2018-03-03 20:17:25.426   openLuup.scheduler::   version 2018.01.30  @akbooer
2018-03-03 20:17:25.436   openLuup.wsapi::       version 2017.01.12  @akbooer
2018-03-03 20:17:25.437   openLuup.servlet::     version 2018.02.19  @akbooer
2018-03-03 20:17:25.437   openLuup.server::      version 2018.02.26  @akbooer
2018-03-03 20:17:25.447   openLuup.rooms::       version 2016.06.23  @akbooer
2018-03-03 20:17:25.454   openLuup.scenes::      version 2018.02.19  @akbooer
2018-03-03 20:17:25.460   openLuup.chdev::       version 2017.05.10  @akbooer
2018-03-03 20:17:25.461   openLuup.userdata::    version 2018.03.02  @akbooer
2018-03-03 20:17:25.461   openLuup.requests::    version 2018.02.18  @akbooer
2018-03-03 20:17:25.462   openLuup.gateway::     version 2018.03.02  @akbooer
2018-03-03 20:17:25.464   openLuup.io::          version 2017.04.27  @akbooer
2018-03-03 20:17:25.464   openLuup.luup::        version 2018.02.10  @akbooer
2018-03-03 20:17:25.505   luup.create_device:: [1] D_ZWaveNetwork.xml / I_ZWave.xml /
2018-03-03 20:17:25.517   luup.create_device:: [2] D_openLuup.xml / I_openLuup.xml / D_openLuup.json
2018-03-03 20:17:25.517   openLuup.init:: loading configuration user_data.json
2018-03-03 20:17:25.539   openLuup.userdata:: loading user_data json...
2018-03-03 20:17:26.014   openLuup.userdata:: JSON decode error @[433754 of 433757]: unterminated string at or near 'trig'
2018-03-03 20:17:26.015   openLuup.userdata:: ...user_data loading completed
2018-03-03 20:17:26.015   openLuup.init:: running _openLuup_STARTUP_
2018-03-03 20:17:26.015   luup_log:0: startup code completed
2018-03-03 20:17:26.015   openLuup.init:: init phase completed
2018-03-03 20:17:26.015   :: openLuup LOG ROTATION :: (runtime 0.0 days)


The user_data load bails on the corrupt JSON file and in the next startup attempt, only openLuup loads as the user_data file has been overwritten to the basic file config.  In my case, this might be occurring because there is no way to shut down my box in an orderly way: it's either power on or power off.  Of course on reboot, all plugins and ALTUI fail to start as their reference has been wiped clean.  I haven't found an easy way to manually reinstall altui through the CLI.  So I have had to essentially re-install everything from your install.bat.

Is there a way to auto-store the last known good configuration, preferably in its own folder?  I imagine you would have to rotate configurations based on samplings made during/after a change in program configuration.  Then, in the event of corruption, one could use a terminal and invoke an OS CLI command to rebuild ones configuration through a .sh file.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: user_data.json
« Reply #1 on: March 04, 2018, 10:47:44 pm »
Take a look at the backup information here for openLuup: http://forum.micasaverde.com/index.php/topic,34468.msg280940.html#msg280940 (look up a few posts to see the script itself).

But you really need to figure out a way to shut it down cleanly.  You can't do a request id=exit from a browser before it goes down?  I think openLuup is writing out user_data periodically - are you really getting that unlucky that the power down happens during this write?  Or is something else going on?

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #2 on: March 05, 2018, 03:00:41 am »
The AltUI More > Table Controllers page has a Backup button.  The openLuup console has a Backup menu to list them.  Any backup file can be used to start openLuup.  Backups are, by default, stored in openLuup's backup/ directory.

However, agree with @jswim788, something else must be going on here.  Pulling the plug is not a good shutdown plan!
« Last Edit: March 05, 2018, 03:30:16 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #3 on: March 05, 2018, 07:35:50 pm »
Unfortunately, on my armbian box, the firmware does not currently support proper shutdown: https://www.armbian.com/orange-pi-pc2/  So like the Vera, to shut down the box means disconnecting the power.

What is strange about the "decode error" is that, per the log line notation, the problem always seems to occur at the end of the json file.  I would think that if this was a power cycle issue, that the corruption would occur anywhere.  So maybe there is something else going on.  But without further diagnostic data, I can't tell.

I can't restore from ALTUI because I can't manually load the UI.  So I have to do a complete re-install and then do a restore--unless there's an easy way to load ALTUI from the CLI.

The first couple of times I've had to rebuild were not that big of a deal because I'm still in configuration mode and learning the nooks and crannies of the OS.  But I imagine that if I have a fairly stable system, and the power goes out, and I lose my config, it will be a bit aggravating.  Having said that, openLuup is very cool piece of software and if I have to work within the current program's limits, then that's the way it will be.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #4 on: March 06, 2018, 06:14:29 am »
Unfortunately, on my armbian box, the firmware does not currently support proper shutdown: https://www.armbian.com/orange-pi-pc2/  So like the Vera, to shut down the box means disconnecting the power.

Fair enough.  But before intentionally pulling the power plug, then you should at least do:

Code: [Select]
http://openLuupIP:3480/data_request?id=exit

This should write and close the file properly.

Quote
What is strange about the "decode error" is that, per the log line notation, the problem always seems to occur at the end of the json file.  I would think that if this was a power cycle issue, that the corruption would occur anywhere.  So maybe there is something else going on.  But without further diagnostic data, I can't tell.

Yes, I agree.  A way forward would be for you to send me an example of the questionable JSON file.  I am assuming that normal reloads happen without error, and this is only when you power down?  I've never had an issue like this.  I'm wondering if it's to do with caching of file writes by the OS?

Quote
I can't restore from ALTUI because I can't manually load the UI.  So I have to do a complete re-install and then do a restore--unless there's an easy way to load ALTUI from the CLI.

Don't understand - can you elaborate?  You simply have to run :

Code: [Select]
./openLuup_reload whatever-user_data_backup_file_you_like &

at the command line to restore an old configuration.  Indeed, there are also ways to download a completely new AltUI in a fresh openLuup system - that's what the install script does - but there should be absolutely no need to do this since you have all the files there.

Quote
The first couple of times I've had to rebuild were not that big of a deal because I'm still in configuration mode and learning the nooks and crannies of the OS.  But I imagine that if I have a fairly stable system, and the power goes out, and I lose my config, it will be a bit aggravating. 

As above, this should be completely unnecessary.  There is definitely a root cause to be found somewhere in your system, but in the mean time we can find a better workaround than what you have at the moment.

Quote
Having said that, openLuup is very cool piece of software and if I have to work within the current program's limits, then that's the way it will be.

The limits are always being extended, mostly as a response to helpful input from users like yourself.  So just let me know of any GOOD IDEAS.  There is a Suggestions thread.
« Last Edit: March 06, 2018, 09:16:56 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #5 on: March 08, 2018, 08:48:15 pm »
I backed up openLuup using the controller backup option and saved the file to both the /cmh-ludl/backup folder, as well as to my local hard drive.

I then used your exit command to close down the program in the browser and all seemed to go well.

On reboot today, I again found the problem where user_data.json was erased.

I then tried to restore using the method you outlined in this post. Attached is the failure message I received.

I can't seem to open the file using any conventional compression programs 7zip etc. Below is the log.  The startup log shows the bare bones basic startup.
2018-03-06 22:17:26.243   :: openLuup LOG ROTATION :: (runtime 0.0 days) 
2018-03-06 22:17:26.244   openLuup.init:: init phase completed
2018-03-06 22:17:26.244   openLuup.server:: starting HTTP server on xx.xxx.xx.xx:3480 tcp{server}: 0x2771f548
2018-03-06 22:17:26.244   openLuup.scheduler:: starting
2018-03-06 22:17:26.244   openLuup.scheduler:: [2]     openLuup device startup
2018-03-06 22:17:26.245   openLuup.context_switch::  ERROR: [string "[2] I_openLuup.xml"]:358: bad argument #1 to 'format' (string expected, got nil)
2018-03-06 22:17:26.245   openLuup.scheduler:: job aborted : [string "[2] I_openLuup.xml"]:358: bad argument #1 to 'format' (string expected, got nil)
2018-03-08 16:57:36.527   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x274dd3c8
2018-03-08 16:57:36.528   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x27600358
2018-03-08 16:57:36.528   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x27660518
2018-03-08 16:57:36.528   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x276dca18
2018-03-08 16:57:36.529   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x274fab88
2018-03-08 16:57:36.529   openLuup.server:: new client connection from 10.17.1.60: tcp{client}: 0x27527908
2018-03-08 16:57:36.861   openLuup.server:: GET / HTTP/1.1 tcp{client}: 0x274dd3c8
2018-03-08 16:57:36.864   openLuup.server:: request completed (194 bytes, 1 chunks, 3 ms) tcp{client}: 0x274dd3c8
2018-03-08 16:57:39.729   openLuup.server:: GET /data_request?id=lr_ALTUI_Handler&command=home HTTP/1.1 tcp{client}: 0x274dd3c8
2018-03-08 16:57:39.731   openLuup.servlet:: No handler for data_request?id=lr_ALTUI_Handler
2018-03-08 16:57:39.731   openLuup.server:: request completed (153 bytes, 0 chunks, 1 ms) tcp{client}: 0x274dd3c8

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #6 on: March 09, 2018, 02:39:47 am »
You are using the wrong process to restore.

The backup file should be given as the single parameter to the openLuup_reload script.

Quote from: openLuup User Guide: Backing up the system
This file is a binary compressed version of the user_data.json file, and typically 5 to 6 times smaller. It may be used as a parameter to the openLuup_reload script and the initialisation process will uncompress it automatically.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #7 on: March 09, 2018, 05:37:15 pm »
Isn't that the command that is shown in the jpeg?  The command I'm using there is:

Code: [Select]
openLuup_reload openLuup-88800000-2018-03-06.lzap
« Last Edit: March 09, 2018, 06:01:07 pm by Buxton »

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #8 on: March 09, 2018, 05:54:45 pm »
No, it isn't.

openLuup_reload is a shell script, not a Lua script.

You need:

Code: [Select]
./openLuup_reload backup/openLuup-88800000-2018-03-06.lzap &

given to the shell prompt when you are in the cmh-ludl/ directory.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #9 on: March 09, 2018, 06:01:47 pm »
Smacks head.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #10 on: March 09, 2018, 06:11:52 pm »
OK Everything is back.
Thx

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #11 on: March 09, 2018, 06:15:56 pm »
Good to know!  Of course, the process has not been started at system boot time, although it should keep running across Luup restarts until the next reboot, when, if it shut down properly, it should also restart normally.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #12 on: March 10, 2018, 02:16:00 am »
Well, almost everything came back.  All my Vera devices, plugins and scenes repopulated in the UI.  And all of the baseline plugins that I have installed (verabridge etc.) repopulated.  And the plugins (with their children) that I manually installed from the create device--implementation files repopulated the UI. 

However, the plugins that I installed from the Alt App store did not show up either in my device tab, or in the plugins tab.  I had to re-install several of those.....  This observed behavior is completely uniform across devices.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: user_data.json
« Reply #13 on: March 10, 2018, 02:23:23 am »
That's not right... had you installed them since the backup file was created?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: user_data.json
« Reply #14 on: March 10, 2018, 05:24:08 pm »
No, they were installed and configured for a few hours prior to doing the back-up.  I'm going to guess that this has something to do with the page that enumerates both baseline plugins and downloaded plugins.  Could it be that this page does not get restored, and that this in turn stops downloaded plugins from loading.  The couple of plugins that I loaded from the implementation file restored properly so the restore failure is not seen across all plugins.