We have moved at community.getvera.com

Author Topic: DataYours - a Data Storage Provider for AltUI  (Read 7760 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
DataYours - a Data Storage Provider for AltUI
« on: January 25, 2016, 11:55:08 am »
DataYours now provides a local, or remote, data storage provider within the AltUI framework, and works on Vera and under openLuup on any machine.  It enables the use of the AltUI interface to archive and plot device variable history.  The source files are available from GitHub: https://github.com/akbooer/DataYours

DataYours will register itself with AltUI, if installed, and be available as a Data Storage Provider under the graphing menu for each variable.  A simple naming convention with a one-letter suffix will, by default, configure a data archive for that variable to be:
  • .d - one minute resolution for one day
  • .w - five minute resolution for one week
  • .m - twenty minute resolution for one month (30 days)
  • .q - one hour resolution for one quarter (90 days)
  • .y - six hour resolution for one year
The database can be configured to be local to Vera (or openLuup) or on a remote machine, through the setting of the LOCAL_DATA_DIR parameter.

Attached is an example of system memory usage plotted using the name "memory.d", although not a full day's data yet populates the graph.

The mapping from name extension, like .m, to the archive structure is done with a standard DataYours configuration file storage_schemas.conf which should be located in LOCAL_DATA_DIR.  In the following code snippet are the schema rules which will implement the above mapping.  They can, of course, be changed to suit.  These rules are only use in the creation of new files, so it is also possible to create a specific archive structure for a particular metric and just use that name without any extension modifier.

Code: [Select]
#
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is read whenever a file create is required.
#
#  [name]  (used in log reporting)
#  pattern = regex 
#  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...

#  2016.01.24  @akbooer
#  basic patterns for AltUI Data Storage Provider

[day]
pattern = \.d$
retentions = 1m:1d

[week]
pattern = \.w$
retentions = 5m:7d

[month]
pattern = \.m$
retentions = 20m:30d

[quarter]
pattern = \.q$
retentions = 1h:90d

[year]
pattern = \.y$
retentions = 6h:1y


Since DataYours is inherently a modular, distributed system, the data may be replicated on local and remote databases.  It is also possible to run without DataYours installed on the local system at all, sending everything to a remote machine, but there needs to be a one-time manual registration of the remote machine as an AltUI Data Storage Provider.  The AltUI interface for saving and plotting the data remains, in either case, the same (and is far more straight-forward than the dashboard interface provided by DataYours.)

Example screenshot attached below.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #1 on: January 26, 2016, 09:57:22 am »
A plot of archived memory usage after one day.  Since the archive is for one day only, and it is a round-robin database, the file size will remain fixed and data older than one day automatically overwritten.  Additional graphic options are available for the plot command, in this case an area fill mode to enhance the plot.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline ronluna

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +1/-3
Re: DataYours - a Data Storage Provider for AltUI
« Reply #2 on: May 18, 2016, 12:25:37 am »
Wondering How to get dataYours listed under the graphing menu when using openluup. It is installed and saving data... but for some reason does not show graphing option in ALTUI.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #3 on: May 18, 2016, 03:01:49 am »
Are you using the latest version?  It registers with AltUI as a data provider and should appear on the list of one of the storage options.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline ronluna

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +1/-3
Re: DataYours - a Data Storage Provider for AltUI
« Reply #4 on: May 18, 2016, 10:16:11 am »
I'm 2 weeks behind the current openluup development branch. Considering the DataYours was updated back in January the current running development version running on the system should not be a problem.

Here are some screenshots of the current DataYours configuration and the ALTUI graphing options listed.

wondering if I could be missing something.

Offline ronluna

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +1/-3
Re: DataYours - a Data Storage Provider for AltUI
« Reply #5 on: May 18, 2016, 01:22:54 pm »
I've copied the .lzo files from the installed version on the vera controller, uncompress and moved them into the opneluup directory...

I realize I was using a DataYour outdated version after searched for "RegisterDataProvider" and there was nowhere to be found.

Got a fresh copy from github and DataYours is now listed under graphing tools but I'm getting the following error...

./L_DataWhisper.lua:622: unable to open Whisper file '/var/lib/graphite/whisper/unknown.wsp'

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #6 on: May 18, 2016, 01:28:59 pm »
Ah, better.

I was just searching through my log.  You should now see at startup, something like:

Code: [Select]
2016-05-17 21:45:34.178   luup_log:7: DataWatcher: registering with AltUI [3] as Data Storage Provider

The file read error is because the new data file has not yet been written to... it needs a data variable update!  You should find that this goes away after a while, or a reload, or you can force it with a manual update, and you'll start seeing your data.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline ronluna

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +1/-3
Re: DataYours - a Data Storage Provider for AltUI
« Reply #7 on: May 18, 2016, 09:54:36 pm »
Everything seems to be working as expected @akbooer... Although I'm trying to see if I can use the regular graphite interface to later see if I can get grafana to plot the data being store by DataYours.

DataYours and AltUI can show the .wsp files  just fine but graphite can't process any data stored by dataYours. something that I noticed is that the .wsp files created by dataYours  are different than the ones created by graphite or by storing data like "echo "test.count 4 `date +%s`" | nc -q0 127.0.0.1 2003" to create new .wsp files

dataYours data is in the clear (plain comma delimited text)
and other wsp files are in binary mode.

Wondering if there is a way for  graphite to read files generated by dataYours?

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #8 on: May 19, 2016, 04:36:59 am »
I'm trying to see if I can use the regular graphite interface to later see if I can get grafana to plot the data being store by DataYours.

I previously made some adjustments for other to try Grafana, http://forum.micasaverde.com/index.php/topic,32701.0.html, although I'm unsure as to what degree of success they had.  I'm very keen to support that.

Quote
dataYours data is in the clear (plain comma delimited text)
and other wsp files are in binary mode.

Yes, that's well documented in the DataYours guide:

Quote from: DataYours User Guide, version 2016.02.04 , Design Notes section, page 15
The Whisper code, originally written in Python, has been translated to Lua, and then re- factored somewhat. The database code is pure Lua and will run anywhere, but it is not binary-compatible with Graphite Whisper files because I have chosen CSV rather than binary packing. This makes them exactly three times larger than real Whisper ones, but space (outside of Vera) is not a problem.

...however, I don't think that it's necessary to have binary compatibility of .wsp files, because  seeing, in the above referenced thread, HTTP requests like:

Code: [Select]
grafana/metrics/find/?query=*

leads me to believe that it accesses via HTTP rather than direct file access, which indeed is the usual Graphite way of doing things.  In fact, there's a great piece of work here: http://graphite-api.readthedocs.io/en/latest/, and here: https://github.com/brutasse/graphite-api, which implements these calls and which I have also already translated from Python to Lua.  This provide generic 'storage finders' which can be customised to access any database and I've already written one for dataMine, as well as DataYours.

The problem with doing this on Vera was that you would have to write a separate server to avoid the usual request syntax.  With openLuup, however, I can easily arrange that the native-syntax graphite_api requests work correctly (on port 3480.)

Quote
Wondering if there is a way for  graphite to read files generated by dataYours?

To reiterate, I don't think that's what you want if your final goal is Grafana.  However, if you do want that, and you like Python, I'd suggest writing your own storage finder using the graphite_api code linked to above.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #9 on: October 01, 2016, 08:05:49 am »
I'm trying to see if I can use the regular graphite interface to later see if I can get grafana to plot the data being store by DataYours.

At last, I've made some progress with this.  A new implementation of the DataYours back-end, utilising graphite_api (translated into Lua) shows promise, as per the attached screen shot.  This would mean we can abandon the terrible DataDash dashboard implementation entirely.  AltUI already provides the hooks in the form of Data Storage Providers to create and manage which Vera/openLuup variables to record.

This example actually shows data coming from both a Whisper database (in yellow, as used by DataYours) and a dataMine database (in green) since I've written storage finders for both.  Other data formats could be handled the same way.

At the moment, I've implemented this as an openLuup CGI module, which makes the Grafana HTTP calls easy to handle.  It's still work in progress as I seem to be having a bit of a learning curve with Grafana templates.



Edit: added screen capture of a complete dashboard.
« Last Edit: October 01, 2016, 04:50:49 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: DataYours - a Data Storage Provider for AltUI
« Reply #10 on: November 18, 2016, 12:10:10 am »
I've bumped into 3 issues with DataYours on AltUI using openLuup:

#1 I used a space in the Metric name in the push as in "Energy Usage".  This doesn't work well.  Part of the code thinks that is the name, but the file it tries to open in the whisper directory is "Energy".  I could not get this changed in the GUI and eventually removed it in user_data.json and then it worked.

#2 I am recording data from a smart meter and somehow it got a bad value.  This was put into the whisper database as "nan" (not a number?).  Unfortunately this causes the graph in AltUI to not be displayed with no error or information on why it is not there.  Could the push to DataYours filter these out?  Or replace them with 0 or the last value?  And could the graphing routing ignore them?  Or at least tell you that you have a bad value?

#3 I cannot add any more push variables in the GUI.  I can click on the graph button on the variable and select DataYours and set the metric name, but it doesn't stick.  Nothing is sent to openLuup and nothing is saved.  I can edit the user_data.json directly and add push variables, but I can't do this through the GUI.  What am I doing wrong?  I tab to the next field, I press return - nothing seems to make it stick.  There is no 'save' button, just a 'close' button.  Any ideas?

Otherwise this works well.  Thanks.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #11 on: November 18, 2016, 06:22:51 am »
#1 I used a space in the Metric name in the push as in "Energy Usage".  This doesn't work well.  Part of the code thinks that is the name, but the file it tries to open in the whisper directory is "Energy".  I could not get this changed in the GUI and eventually removed it in user_data.json and then it worked.

Don't do that.  The field is a metric name, not a plot title or free text, or anything like that.  One of the reasons this breaks is that the (industry standard) Whisper plaintext format used to send data to the cache back end is of the form "metric value timestamp" (space separated) so it ends up being stored in a Whisper file called Energy.wsp with invalid value and timestamp.

Quote
#2 I am recording data from a smart meter and somehow it got a bad value.  This was put into the whisper database as "nan" (not a number?).  Unfortunately this causes the graph in AltUI to not be displayed with no error or information on why it is not there.  Could the push to DataYours filter these out?  Or replace them with 0 or the last value?  And could the graphing routing ignore them?  Or at least tell you that you have a bad value?

Bad values seem to happen fairly frequently to (my) meters.  Whether this is a Z-wave transmission error or a device firmware problem, I don't know.  "nan" is a perfectly valid entry in the Whisper database (as is any non-numeric value) and indeed represents NaN, or in fact null.  This is not a problem at all - my files are riddled with NaNs - it's really a consequence of fixed sample rates in each of the archives.  The graphing routine does ignore them, or can be configured to repeat the last valid value, or can treat it as zero.  All defined by command-line parameters that you can edit in to the plot line in the data watch panel for that variable.  I don't know why you're not seeing anything.  Perhaps there hasn't been a valid data update to the variable yet?

Quote
#3 I cannot add any more push variables in the GUI.  I can click on the graph button on the variable and select DataYours and set the metric name, but it doesn't stick.  Nothing is sent to openLuup and nothing is saved.  I can edit the user_data.json directly and add push variables, but I can't do this through the GUI.  What am I doing wrong?  I tab to the next field, I press return - nothing seems to make it stick.  There is no 'save' button, just a 'close' button.  Any ideas?

You've done it once at least correctly?  Check the AltUI docs on this.  You need to make your entries on the page and then press the (red) "graph" icon button at the top (the same one you used to enter the page.)  Also, the configuration will not be permanently saved until the next user_data.json checkpoint (might be up to 6 minutes away) or the next Luup reload (whichever the sooner.)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: DataYours - a Data Storage Provider for AltUI
« Reply #12 on: November 18, 2016, 12:23:50 pm »
Yes, now that I know spaces aren't good, I won't use them.  It would be nice to get the code to print a warning, or not accept it, or something else, but I'm aware of it so it won't be a problem for me in the future.

I had plenty of data but the graph was just blank with the 'nan'.  I edited the whisper file to correct this single value and then went back to the browser and then the graph was fine.  I'll check out the graph options.  This is the graphing accessed via AltUI - I think you have other methods as well.

Yes, the GUI once before let me put in values to be graphed and saved them.  The graph button does change color, I select DataYours.  But it seems that this is not saved now.  I close the window after setting the metric.  After this, I can go right back into it and see that it was not saved.  And nothing pops up in the openLuup log.  Perhaps I should be more patient?  Does it take some time for the AltUI to send this over to openLuup?

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: DataYours - a Data Storage Provider for AltUI
« Reply #13 on: November 18, 2016, 01:58:41 pm »
Any UI change should immediately be reflected in openLuup device variables.  If you're not seeing any variables change in the log when you do this, then it's not happening.

Sounds like an AltUI-related thing?  Maybe some invalid data somewhere in its device variables?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: DataYours - a Data Storage Provider for AltUI
« Reply #14 on: November 21, 2016, 12:17:55 pm »
A restart of the Chrome browser seemed to clear up my entry issues.

But I still get a blank graph when I have a 'nan' in the data.  Chrome says:

Code: [Select]
data_request?id=lr_render&target=EnergyUsage.e&hideLegend=true&height=250&from=-y:2922 Uncaught ReferenceError: nan is not defined(?)
This is on the openLuup system with AltUI.  Is there an option I can use to tell it to ignore the 'nan's?

Also, I tried out the multi-storage and I can see the data getting populated in the .wsp file appropriately, but I'm not sure how to graph that data.  Any examples?  Or are you viewing this outside of AltUI with grafana?  Sorry, I'm not quite up to speed on the graphing and would appreciate some pointers.  Thanks.