We have moved at community.getvera.com

Author Topic: openLuup: Data Historian  (Read 5025 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Data Historian
« Reply #30 on: July 14, 2018, 10:42:06 am »
Found the file in the openLuup dir. Ok to delete this?

No, that's probably the right one.  If you look in that file it should have a much newer version number.

There MUST be another one.  Do you have a cmh-lu/ directory, or perhaps in files/?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: Data Historian
« Reply #31 on: July 15, 2018, 12:41:37 am »
I recently updated my unifi controller and it caused enormous havoc on my network.  What Unifi doesn't communicate well is that the version of the controller that you need on your network is highly contingent on the type of equipment you have.  In my case, my APs are a couple of years old, and the controller that works with that equipment is several revisions back from the latest and greatest. 

You can find out what controller you need by visiting the Ubiquity site and looking up the firmware for your specific devices.  There will be a link in the firmware device folder to the version of the latest controller that is compatible with your gear.  Backup your settings before a new install because if the controller that you need is older than the controller you're using, you will be forced to uninstall the existing controller first, potentially wiping out your settings.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: openLuup: Data Historian
« Reply #32 on: July 15, 2018, 12:44:39 am »
AK, I'm having a difficult time getting Grafana to recognize the Whisper files on my USB drive.  This, -- on the data source page of the Grafana homepage instance.  Any tips on what needs doing as I don't see any info on the web regarding this particular type of datafile.  OpenLuup opens the files fine via historian. 

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #33 on: July 15, 2018, 02:18:12 am »
AK, I'm having a difficult time getting Grafana to recognize the Whisper files on my USB drive. 

I wasn't aware that Grafana could/should be able to access the database files directly? 

Quote
This, -- on the data source page of the Grafana homepage instance. 

...or are you trying to use the Graphite Carbon Metrics dashboard for this?

This relies on database queries of the form:
Code: [Select]
/metrics/find?query=carbon.agents.*.updateOperations
which I have not implemented (but could... most of the relevant metrics are gathered internally.)

Quote
Any tips on what needs doing as I don't see any info on the web regarding this particular type of datafile. 

The datafile format for the original Python implementation is described here:
http://graphite.readthedocs.io/en/latest/whisper.html

However, what you probably haven't looked at is my documentation for DataYours, which describes the Lua translation:

Quote from: DataYours User Guide, version 2016.02.04 , Design Notes section, page 15
The Whisper code, originally written in Python, has been translated to Lua, and then re- factored somewhat. The database code is pure Lua and will run anywhere, but it is not binary-compatible with Graphite Whisper files because I have chosen CSV rather than binary packing. This makes them exactly three times larger than real Whisper ones, but space (outside of Vera) is not a problem.

To make them binary compatible you would need to replace the 'struct' implementation with a binary version.

Quote
OpenLuup opens the files fine via historian.

Good, ... so what it is you are trying to do with Grafana?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Data Historian
« Reply #34 on: July 15, 2018, 12:01:11 pm »
@powisquare

The latest development commit (v18.7.15) should fix your VeraBridge problem, if I have diagnosed things correctly.

I do still believe that this is nothing to do with the Data Historian and it's a side-effect of having multiple L_VeraBridge.lua files, but I have used this opportunity to rectify a long-standing inconsistency with file search paths, which should address such issues.

Hoping this does fix things... (and breaks nothing else!)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: openLuup: Data Historian
« Reply #35 on: July 15, 2018, 06:39:08 pm »
I was just playing around with the software--trying to understand the capabilities based on your quote: 

Quote
You don't need DataYours at all, and can link Grafana directly to openLuup.

I misread openLuup to stand as a proxy for the actual data-files.  I'll go over the documentation again.

Offline powisquare

  • Full Member
  • ***
  • Posts: 146
  • Karma: +1/-0
Re: Data Historian
« Reply #36 on: July 16, 2018, 05:23:26 am »
Sorry guys - have not been receiving the usual email notifiactions from micasaverde (?!) so thanks for the pm. Have searched the whole pi and only found L_VeraBridge.lua in /etc/cmh-ludl/openLuup/. Have updated to latest development release openLuup and will see what happens. Just having trouble logging into Grafana now  'This site can't be reached'. arrgh

Had a new smart meter installed which may have sent the cloud key controller into a tiz. British gas now tell me the meter will provide a reading once a quarter automatically. Not quite what I was hoping for but will get onto energy metering on a another thread soon.

Offline powisquare

  • Full Member
  • ***
  • Posts: 146
  • Karma: +1/-0
Re: Data Historian
« Reply #37 on: July 16, 2018, 05:30:48 am »
Ok grafana was not started for some reason.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #38 on: July 16, 2018, 05:40:51 am »
I was just playing around with the software--trying to understand the capabilities based on your quote: 

Quote
You don't need DataYours at all, and can link Grafana directly to openLuup.

Well, as usual, I have not explained things very well.  It's certainly time for an update to the documentation.

DataYours has its own Whisper database, location defined by the contents of DataYours variable LOCAL_DATA_DIR, usually 'whisper/'.  Data Historian has its data wherever you've set the system attribute openLuup.Historian.Directory, usually 'history/'.

Pointing Grafana at openLuupIP:3480/, should allow its metric menus to find both databases, but I've just tried this, and it seems not to find the DataYours files.  This is perhaps the issue you raised.  I will look into it. [edit: it does work, I just didn't have DataYours running when I tested it!]

However, what DOES work, is to add the following line to Lua Startup

Code: [Select]
luup. attr_set ("openLuup.Historian.DataYours", "whisper/")               -- overriding DY finder

Assuming that your LOCAL_DATA_DIR points to 'whisper/', then after a restart, you should see all your original DataYours file under a metric tree called DataYours...

Code: [Select]
"DataYours.Vera-88800000.008.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.Vera-88800000.294.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.Vera-88800000.386.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.cpu.d",
"DataYours.memory.d",
"DataYours.unknown",
"DataYours.uptime.m",

...along with the other sub-trees generated by the historian (one for openLuup, and one for each bridged Vera.)

HTH, AK
« Last Edit: July 16, 2018, 09:42:00 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 146
  • Karma: +1/-0
Re: Data Historian
« Reply #39 on: July 16, 2018, 10:02:26 am »
Brilliant - the nil files are no longer created and PK_AccessPoint values are persistent. Will leave it running and see how it goes. Cheers.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Data Historian
« Reply #40 on: July 16, 2018, 10:35:32 am »
 :)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 146
  • Karma: +1/-0
Re: openLuup: Data Historian
« Reply #41 on: July 19, 2018, 09:47:37 am »
Quick update - the CurrentTemperature  values are all plotting like a dream. CurrentSetpoint and SwitchpowerStatus only display when I set the timeframe to last  7 days. Part of me was thinking only a change in value is recorded but then SwitchpowerStatus has not changed in value ever so that probably is not it. Console/Files/History DB is still showing 'On-disk archiving not enabled' if that makes a difference.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #42 on: July 19, 2018, 10:24:53 am »
Quick update - the CurrentTemperature  values are all plotting like a dream. CurrentSetpoint and SwitchpowerStatus only display when I set the timeframe to last  7 days.

Has your system been up and running continuously for just over 7 days?

Quote
Part of me was thinking only a change in value is recorded but then SwitchpowerStatus has not changed in value ever so that probably is not it.

Except for startup, when current values are written, only variable changes are stored in cache (this saves memory space.)  However, all variable_set values are written to disk archive.  This helps with the application of retention policies between archives.

Quote
Console/Files/History DB is still showing 'On-disk archiving not enabled' if that makes a difference.

AHA!  A eureka moment.

Am I correct in saying that you have not ever installed DataYours on your openLuup system?  This is fine, you don't need it, but in early implementations I used one of its files and I did have it installed, so I was not getting this message.

I believe that you are only seeing data from the in-memory cache, although it will be archived to disk... you just can't see it!

All fixed (I hope) in development release v18.7.19
« Last Edit: July 19, 2018, 12:04:07 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 146
  • Karma: +1/-0
Re: openLuup: Data Historian
« Reply #43 on: July 19, 2018, 12:33:08 pm »
Bingo - as you deduced there is no Datayours installed. History DB is now populated and all is appearing on Grafana. Huge thanks.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #44 on: July 19, 2018, 12:38:03 pm »
Excellent.  Thanks for your help... several bugs squashed in the process!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.