Author Topic: DataYours whisper files not updating after LUUP reload on openLuup  (Read 517 times)

Offline bruring

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +2/-0
  • Why not automate everything?
Hi all,

First of all thanks for this great community - without it Vera wouldn't be as advanced and customisable as it is now.

I have successfully configured openLuup with a bridge to my Vera. It is running DataYours to watch variables.
I didn't set it as a data provider for AltUI, instead I use the DataYours interface (lr_dashboard) to configure the variables to watch and use Grafana to see graphs.
This has been working like a charm for the last 20 days or so - such a relief coming from DataMine  :D

Now, when I reload LUUP on this running system or if I reboot / reload openLuup, the .wsp database files don't get updated. The file modification time doesn't change and Grafana doesn't show new graph points. Luckily I'm running in a virtual machine and can restore a running backup, but this is not a long term solution of course, as there will be a day where I will have to reload LUUP... :-)

Somehow the variables still get logged as I can see up-to-date graphs through the DataYours interface. I tried debugging this, and all the log files show successful loads of DataYours and all daemons, no errors whatsoever. Also the DataYours device shows the correct amount of watched variables.

What would be the best way to debug this?
I've gone through the DataYours documentation and source files, and suspect it has something to do with L_DataWhisper.

How can I get more verbose logging to see where it fails?
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6196
  • Karma: +276/-70
  • "Less is more"
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #1 on: August 11, 2018, 06:00:45 am »
I can help you with this, but my recommendation would be to switch to using the native Data Historian feature of the latest openLuup system.  Its database is compatible with DataYours, indeed you can run both together to try it out.  You will need to update to the latest development branch using the Plugins update page.

I realise that this might be another unwelcome change for you, but it actually means that there is pretty much no configuration to do, and all the interesting variables from Vera will be saved, and available to plot.  The DataYours interface is very clunky, and it's so long since I've used it that I have to go back and remind myself.  You also have the capability to produce vastly superior plots and dashboards using Grafana, as you've discovered.

Let me know if you're up for this, or would prefer to fix your current situation... but Data Historian is absolutely the future!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +2/-0
  • Why not automate everything?
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #2 on: August 11, 2018, 08:57:59 am »
Hi akboer,

Thanks for your reply. Indeed data historian will be the future, I updated to the development branch immediately :-). Great work!
Now the historian is logging everything, but the DataYours whisper files still don't get updated.

Would love to migrate to historian!
I see the structure is a bit different from DataYours. Could you point me in the right direction for the following:

- I've set up watched variables and retention resolution and aggregation manually per watched variable. Is there a way to customize this in Historian?

- I made symlinks in my Whisper DB folder to give everything more human readable names in Grafana (i.e. I've linked Humidity.Living.wsp to Vera-88800000.10391.urn^micasaverde-com^serviceId^HumiditySensor1.CurrentLevel.wsp, Temp.Basement.wsp to Vera-88800000.10461.urn^upnp-org^serviceId^TemperatureSensor1.CurrentTemperature.wsp). So in the Grafana dropdown query menu's, I can choose Temperature, then Basement. This looks a lot nicer in the Legend as well. Not sure if this is the best way to do it though.

- Is there a way to migrate the DataYours WSP database files to Historian?

- On a side note: am I correct by saying that variables first get logged in memory, and written to disk every x minutes? So the Grafana charts only update every x minutes?

Let me know if there are any solutions for above, then I can migrate everything. If not, I would prefer to fix my DataYours whisper database, and migrate slowly to Historian. It's definitely the next step forward!
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6196
  • Karma: +276/-70
  • "Less is more"
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #3 on: August 11, 2018, 10:26:05 am »
These questions put you into the "advanced user" league!

Would love to migrate to historian!
I see the structure is a bit different from DataYours. Could you point me in the right direction for the following:

You mean the metric naming hierarchy?  Yes, for good reason.  I feel the need to justify this.  There are, in fact, three naming conventions that you may run into when using Data Historian:
  • Standard Luup naming - full serviceId, variable name, LOCAL device number
  • Graphite API naming (as seen by Grafana) - nodeName, REMOTE device number/name, short serviceID, variable name
  • Whisper archive files - nodeNumber, REMOTE device number, short serviceId, variable name
where:
  • nodeNumber - 0 for openLuup otherwise PK_AccessPoint or remote Vera 
  • nodeName - the name of the associated VeraBridge, or "openLuup"
  • shortSid - final apha-numeric part of the full serviceId
  • shortDevName - device name with all non-alphanumerics removed

Convention (1) is inevitable for Luup access, as in luup.variable_...(), (2) seemed to be the most readable and understandable, (3) is locked to the remote machine ID.

This way, even if the order of the bridges is changed in openLuup, the files are the same.  If you have to swap out a Vera, then its PK_AccessCode will change and you'll have to rename files to continue using them, but the finder metrics names will not change (unless you rename the associated VeraBridge.)

Pattern-matches for schemas, finders, and cacheVariables() ALL use the API paths and wildcards described here: http://graphite-api.readthedocs.io/en/latest/api.html#graphing-metrics

Quote
- I've set up watched variables and retention resolution and aggregation manually per watched variable. Is there a way to customize this in Historian?

Yes, certainly.  There's two levels of definition/indirection, in order to maintain compatibility with DataYours and the original Graphite/Carbon system on which it is based.
  • There are two files which may be put in the historian/ data directory (whatever you do, don't alias or combine this with the DataYours directory): storage-schemas.conf, and storage-aggregation.conf.  In their absence, virtual defaults are used.  You can find them in the openLuup/virtualfilesystem.lua file.  These are the absolutely standard Graphite configuration files which define archive structure depending on the written metric filename.  However, for Data Historian, these rules are used in a different way...
  • ...using the archive_rules found in the openLuup/servertables.lua file.  These provide the name of which schema to use based on the LOCALdeviceNo.shortServiceId.variable name pattern used internally by Data Historian.
These rules are only used to create missing files.  As you seem already to have discovered, if you create a correctly-named file with any archive structure you like, then that file will be used for the matching variable.

I'm sorry this is complicated, but it's logical, and preserves existing conventions.

Quote
- I made symlinks in my Whisper DB folder to give everything more human readable names in Grafana (i.e. I've linked Humidity.Living.wsp to Vera-88800000.10391.urn^micasaverde-com^serviceId^HumiditySensor1.CurrentLevel.wsp, Temp.Basement.wsp to Vera-88800000.10461.urn^upnp-org^serviceId^TemperatureSensor1.CurrentTemperature.wsp). So in the Grafana dropdown query menu's, I can choose Temperature, then Basement. This looks a lot nicer in the Legend as well. Not sure if this is the best way to do it though.

This ugly DataYours naming was something I abandoned, and, in fact, if you were to have used the AltUI Data Providers route (don't do that now) then you could choose arbitrary names according to your fancy.

If you've not defined the openLuup.Historian.DataYours attribute to point to your DataYours Whisper directory, then those metrics will all appear at the top level of the Graphite search tree.  If you do define it, then they will be moved to a sub-tree called DataYours.  (Equally, there is a DataMine attribute which will bring your dataMine database into the tree as well, but perhaps you have abandoned that already.)

The MODERN way to make your metrics legend and labels look nice is to use the additional functions at the trailing end of the Grafana data chooser: alias, aliasByNode, aliasByMetric.  Suppose you have a metric:

Code: [Select]
openLuup.303_OutdoorTemperature.TemperatureSensor1.CurrentTemperature

then following that metric selection with these functions will give:
  • alias ("Foo") - "Foo"
  • aliasByMetric - "CurrentTemperature"
  • aliasByNode (1) - "303_OutdoorTemperature"
  • aliasByNode (0,3) - "openLuup.CurrentTemperature"
Magic!

Quote
Is there a way to migrate the DataYours WSP database files to Historian?

If I've managed to enable you to understand any of the above, then the answer is "yes".  You simply rename the old file to the one which Data Historian expects for that variable and move it into the historian/ directory.  WHATEVER YOU DO, do NOT do this when the system is running.  Whisper file headers are cached and changing the file mid-stream will completely trash it.

Quote
- On a side note: am I correct by saying that variables first get logged in memory, and written to disk every x minutes? So the Grafana charts only update every x minutes?

No, not quite.  Variables are written synchronously to the disk archive - this is one reason why you should really have it on a SSD and separately back up the database elsewhere from time to time.  It's written to be very fast, but obviously is at the mercy of I/O speed.  Grafana can be configured to update at any rate.  The memory cache saves variable update times to sub-millisecond precision.  Once they get written to disk they are quantised according to the retention rules.  If a Grafana graph time interval falls completely within the memory cache then you get times to maximum precision.

Quote
Let me know if there are any solutions for above, then I can migrate everything. If not, I would prefer to fix my DataYours whisper database, and migrate slowly to Historian. It's definitely the next step forward!

Just to curiosity, then, can you tell me what the value of the DataYours CONFIG_DIR is, whether there are actually any files there, and what the link from the DataYours device panel to DataWatcher config shows you?
« Last Edit: August 11, 2018, 03:21:29 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +2/-0
  • Why not automate everything?
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #4 on: August 11, 2018, 12:04:46 pm »
Thanks for the detailed answer, very much appreciated! It answers a lot of questions! Seems I had a weird workaround to label the variables in Grafana... :-)

In the meantime I've been playing around making sure file permissions are correct, going through the logs, etc. but couldn't find anything. I only found that all device .wsp files don't get updated, but the memory/cpu stats etc files do get updated after a Luup reload. Weird.

I must admit I forgot about the configure option from the DataYours dashboard, found it out while trying to answer your last question, thanks! This really helps troubleshooting. With the information you provided before I'm going to migrate to Historian, but now I feel that I'm so close to fixing this problem that I might do that first so at least I have a running system now. I will let you know how the migration went.

To answer your questions:

My DataYours CONFIG_DIR is /www/ and there are no files there.

This is what I found out before and after the LUUP reload:

Before: DataWatcher Daemon has sent 64225 messages through UDP
After: 0 messages

Before: DataWatcher shows a big array of watched devices.
After: array is empty.

Before: DataCache has a big array of watched devices in tally
After: no dives in tally

Before & after: metric list shows all watched devices and storage schemas.

So it seems like DataWatcher does not receive any value changes from openLuup? If unwatch and then re-watch variables, they still don't get updated.

I've atttached the before and after config files.
« Last Edit: August 11, 2018, 12:07:29 pm by bruring »
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline d55m14

  • Sr. Member
  • ****
  • Posts: 444
  • Karma: +8/-1
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #5 on: August 11, 2018, 01:08:00 pm »
Hi akbooer,

I've several remote DY on Vera system sending data to a centralized Openluup/DY on a RPI and a web application to plot,graph,analyze this data.

With the new Openluup can I continue to use Datayours ? Can I migrate smoothly to Historian and can I maintain the storage-schemas and storage-aggregations of the variables archived in whisper databases ? 

Finally can I preserve the actual funtionality of my system ?

tnks

donato

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6196
  • Karma: +276/-70
  • "Less is more"
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #6 on: August 11, 2018, 01:18:08 pm »
In the meantime I've been playing around making sure file permissions are correct

Good plan.  I think this is your problem.

Quote
My DataYours CONFIG_DIR is /www/ and there are no files there.

Actually, THIS is really your problem, but I think it's happened because of permissions.  Your .conf configuration files should be stored here.  It was a dreadful choice for a default location, but this was made in my Vera days before openLuup.  I suggest you change this to somewhere under cmh-ludl (perhaps cmh-ludl/www/) and try again.

As it stands, the reloaded system has no list of variables to watch, so nothing gets updated in the absence of a DataWatcher.conf file.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6196
  • Karma: +276/-70
  • "Less is more"
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #7 on: August 11, 2018, 01:23:42 pm »
I've several remote DY on Vera system sending data to a centralized Openluup/DY on a RPI and a web application to plot,graph,analyze this data.

Yes, I recall.

Quote
With the new Openluup can I continue to use Datayours ?

Yes, you can.

Quote
Can I migrate smoothly to Historian and can I maintain the storage-schemas and storage-aggregations of the variables archived in whisper databases ? 

Yes, see my previous answer to @bruring here.  You could just copy and rename files to the history/ directory.  Before doing that, you could just try Data Historian with its default schemas, and run it in parallel with DataYours.

Quote
Finally can I preserve the actual funtionality of my system ?
Yes, for sure.  It just gets better with time.  I try not to break anything that is already working.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +2/-0
  • Why not automate everything?
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #8 on: August 11, 2018, 03:31:20 pm »
In the meantime I've been playing around making sure file permissions are correct

Good plan.  I think this is your problem.

Quote
My DataYours CONFIG_DIR is /www/ and there are no files there.

Actually, THIS is really your problem, but I think it's happened because of permissions.  Your .conf configuration files should be stored here.  It was a dreadful choice for a default location, but this was made in my Vera days before openLuup.  I suggest you change this to somewhere under cmh-ludl (perhaps cmh-ludl/www/) and try again.

As it stands, the reloaded system has no list of variables to watch, so nothing gets updated in the absence of a DataWatcher.conf file.

That makes sense. My openLuup installation is under /etc/cmh-ludl, so I'll change it to /etc/cmh-ludl/www. Is there a way to save the running configuration to a conf file? Or do I need to reconfigure?
« Last Edit: August 11, 2018, 03:45:25 pm by bruring »
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6196
  • Karma: +276/-70
  • "Less is more"
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #9 on: August 11, 2018, 04:52:22 pm »
That makes sense. My openLuup installation is under /etc/cmh-ludl, so I'll change it to /etc/cmh-ludl/www. Is there a way to save the running configuration to a conf file? Or do I need to reconfigure?

Not totally automatically, but with a bit of editing...

Your DataWatcher diagnostic page should have a variable like this

Code: [Select]
  watch_tally = {
    ["003.urn:upnp-org:serviceId:TemperatureSensor1.DewPoint"] = 224,
    ["004.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature"] = 194,
    ["005.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature"] = 37,
    ["006.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature"] = 51,
    ["007.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel"] = 204,
    ...

Your DataWatcher.conf file would look like this if you had one...

Code: [Select]
[DataWatcher]
#
# configuration file auto update: Tue Feb 14 12:41:19 2017
#
watch=003.urn:upnp-org:serviceId:TemperatureSensor1.DewPoint
watch=004.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature
watch=005.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature
watch=006.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature
watch=007.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel

Simple task to go from (a) to (b) ...?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +2/-0
  • Why not automate everything?
Re: DataYours whisper files not updating after LUUP reload on openLuup
« Reply #10 on: August 12, 2018, 07:52:23 am »
Hi akboer,

That's it!
I've manually created the configuration file, and now watching works after a reload.

Weird thing is that the DataYours plugin defaulted to /www/ instead of www/ for it's configuration files, so if you're running a custom openLuup configuration you would have to make sure to check this. Maybe you can change it in the source?

I'm going to migrate to the historian today or tomorrow, I'll keep you posted how that goes!

Thanks again, Joris
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.