Author Topic: DataYours - schema and aggregation rules  (Read 109 times)

Offline bruring

  • Jr. Member
  • **
  • Posts: 58
  • Karma: +1/-0
  • Why not automate everything?
DataYours - schema and aggregation rules
« on: August 14, 2018, 01:01:02 pm »
Just a minor question: so as I've seen, all watched variables are defined in the DataWatcher.conf file.
Where are the aggregation and storage schema configs saved? In my case it's not in the filenames nor DataWatcher.conf file, so it must be somewhere else?
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6157
  • Karma: +275/-69
  • "Less is more"
Re: DataYours - schema and aggregation rules
« Reply #1 on: August 14, 2018, 01:18:04 pm »
They're used to create the files, so once a file exists, its archive structure is stored in the header.

There are also configuration files which are used in the case that the DataYours UI is not employed to create the file.  These are described in the User Guide

Quote from: DataYours User Guide, p. 17, The storage schema and aggregations rules
Read the Graphite documentation on these files.

http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf
http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf

The implementation is complete.

In particular, note that these are (a subset of) regular expressions used to match incoming data series names.

The rules are scanned in order, with the first matching one being used.
If a match is found, then the Whisper archives are created automatically, according to those rules.
If no match is found then the archive configuration menu is presented to the user for manual selection of archives.

One important departure from the official Graphite documentation is the location of the files storage-schemas.conf and storage-aggregation.conf: they should be placed into your target Whisper database directory.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 58
  • Karma: +1/-0
  • Why not automate everything?
Re: DataYours - schema and aggregation rules
« Reply #2 on: August 14, 2018, 03:45:52 pm »
Ah great - didn't know there was a header!

Let me explain - I'm trying to do the following
I have a so called "smart meter" in my house to meter electricity and gas usage. My energy provider - Greenchoice - reads out these values every 24 hours. I can see these values in my private page on their website so I can see my daily usage. Luckily, someone write a PHP script (heavily relying on CURL) to read out the values. I thought I could use this to feed Grafana somehow :-)

I was thinking about two ways to do it:
1) Write a plugin for openLuup/Vera which reads out the variables every 24hrs (using CURL in Vera) - and let the Historian / DataYours monitor this (which would be great for this community but a bit hard for me - I think I could do it if I modify the DarkSkyWeather plugin which basically does the same but then for weather);
2) Create my own script which sends these values (daily) to the DataCache daemon using netcat. It will probably be some MacGyver code :-)

I tried number 2, and it automatically creates a new Whisper file with the values so that is hopeful. I just wanted to set the aggregation / storage schema somewhere.

Which route would you suggest to follow?
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6157
  • Karma: +275/-69
  • "Less is more"
Re: DataYours - schema and aggregation rules
« Reply #3 on: August 14, 2018, 04:19:57 pm »
I was thinking about two ways to do it:
1) Write a plugin for openLuup/Vera which reads out the variables every 24hrs (using CURL in Vera) - and let the Historian / DataYours monitor this (which would be great for this community but a bit hard for me - I think I could do it if I modify the DarkSkyWeather plugin which basically does the same but then for weather);
2) Create my own script which sends these values (daily) to the DataCache daemon using netcat. It will probably be some MacGyver code :-)

I would add a third: write directly to the Whisper file.  My thoughts on these three, then are:

  • plugin - tedious and unnecessary, involving lots of files, mistakes, and heartache (unless you've done it before... or even if you have)
  • script to DataCache - much easier, although as you point out you have to generate a UDP datagram somehow.
  • write to Whisper file* - again easy, especially if you can code in Lua, but the files have to be on a local disk.

So without knowing your actual skills and system constraints, I'd have to say (2) or (3).  You could write either as scene Lua scheduled to run every day.  It's actually a one-line call to create a Whisper file with the required archive structure, and a further line to actually write a data value.  If you don't manually create the file, you'll have to set up schema and aggregation rules in the configuration file to do that based on the syntax of the written filename, but since you only need to do that once, it seems like overkill.

_______________________

*an example here: http://forum.micasaverde.com/index.php/topic,24669.msg173863.html#msg173863
_______________________

Edit: I've split off these recent posts into a new thread, with perhaps a more descriptive title. (Hope you find it.)
« Last Edit: August 14, 2018, 04:29:32 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6157
  • Karma: +275/-69
  • "Less is more"
Re: DataYours - schema and aggregation rules
« Reply #4 on: August 16, 2018, 07:11:09 am »
Actually, it occurs that the simplest would be to just have a scene write to a device variable.  You don't need a plugin or a special device for this, you could use an existing device or create a dummy one.  Then, DataYours does all the heavy lifting in terms of getting the variable into a Whisper file.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.