Author Topic: openLuup: Data Historian  (Read 1589 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
openLuup: Data Historian
« on: July 08, 2018, 11:58:49 am »
For your next stop, developments are in progress to archive openLuup device variables to DataYours completely without any configuration, which will make things easier.

The latest version of openLuup in the development branch now has the capability to archive variables.  You don't need DataYours at all, and can link Grafana directly to openLuup.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Don Phillips

  • Hero Member
  • *****
  • Posts: 1306
  • Karma: +34/-32
Re: openLuup: Data Historian
« Reply #1 on: July 08, 2018, 07:37:47 pm »
For your next stop, developments are in progress to archive openLuup device variables to DataYours completely without any configuration, which will make things easier.

The latest version of openLuup in the development branch now has the capability to archive variables.  You don't need DataYours at all, and can link Grafana directly to openLuup.

+1
Vera 3, 1.7.1030, CT101, Everspring motion sensor, GE/Jasco switch, Leviton outlet, AeonLabs sensor, NuTone garage door, Blue Iris, Sricam SP011, iPhone locator, APCUPSD, VeraMate, VeraAlerts, PLEG, House Modes, Countdown Timer, DVR, Virtual/Multi Switch, Weatherunderground, LB60Z-1 bulb, Hue, Alexa

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Data Historian
« Reply #2 on: July 10, 2018, 12:47:40 pm »
Great to hear the news on Data Historian. Just wanted some clarification on this ..

 The on-disk archive is enabled by the single LuaStartup line luup.attr_set ("openLuup.Historian.Directory","history/"), where 'history/' is the path (including trailing '/') relative to cmh-ludl/ where you want the data to reside.

Where should I put the line?

Cheers

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #3 on: July 10, 2018, 01:22:16 pm »
Great to hear the news on Data Historian. Just wanted some clarification on this ..

Yes, I've been a bit low-key on this, but you do have all the data you need...

Quote
The on-disk archive is enabled by the single LuaStartup line luup.attr_set ("openLuup.Historian.Directory","history/"), where 'history/' is the path (including trailing '/') relative to cmh-ludl/ where you want the data to reside.

...yes, that's right.

Quote
Where should I put the line?

Anywhere you like in Lua Startup, so long as it gets executed.  Just save, and (ignoring the AltUI message that Vera will do so anyway) restart manually.

Be sure to take the latest development release (just now, that's v18.7.10) it also has the capability to graph any of the variables listed on the console historian cache and DB pages, without even any need of using Grafana (uses Google Charts, so must be internet connected for this trick to work.)

You may want to customize things a bit for your own needs, but straight out of the box it should work OK.

_______________

PS: If there's an HTTP/CSS guru out there, I may need some help putting these plots into pop-up frames above the console page, rather than overwriting it.  (Must NOT involve JavaScript.)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #4 on: July 11, 2018, 08:07:18 am »
Brilliant. Went to Altui/Misc/Lua Startup and added luup.attr_set ("openLuup.Historian.Directory","history/")

In More/Plugins typed 'development' into the openLuup update box and hit update.

Graphs can be seen by following the relevant link in the Console - openLuup/Historian.

I like the way it is possible to graph multiple variables against each other in datayours and might see if grafana will do something similar.   

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #5 on: July 11, 2018, 08:58:09 am »
Brilliant.
[...]
Graphs can be seen by following the relevant link in the Console - openLuup/Historian.

Excellent.  My aim was to make it that easy.

Quote
I like the way it is possible to graph multiple variables against each other in datayours and might see if grafana will do something similar.

Not sure what you mean here?  DataYours (and Data Historian console pages) both use Google Charts and can provide a variety of plots with multiple variables.  Grafana can do so much more, so you shouldn't have any trouble producing what you need...

Adding multiple metrics to a plot in Grafana is easy, and you can interactively walk though the machine / device / serviceId / variable menus to find what you need.  openLuup also provides a few special functions to tidy up the metric naming in legends, etc: alias(), aliasByMetric(), aliasByNode().
« Last Edit: July 11, 2018, 09:21:22 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #6 on: July 13, 2018, 04:31:51 am »
Nearly there! Able to plot Current Temp, Setpoint and Boiler switch status against each other (well almost). Curiously the Setpoint data has disappeared from yesterday. The History folder exists but when I look in the Console to Files/History DB I see 'On-disk archiving not enabled'. Would this be an issue?

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #7 on: July 13, 2018, 05:16:07 am »
The magic line in startup needs to be there to run at every startup.  Did you remove it?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #8 on: July 13, 2018, 05:28:36 am »
...and, in fact, CurrentSetpoint is not a default parameter to archive.

I will add it, but you can easily do so by editing the servertables.lua file, line 202,adding that variable to the existing, CurrentTemperature, ... etc.

I am still confused by your 'On-disk archiving not enabled' message.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #9 on: July 13, 2018, 08:29:43 am »
lua startup is good I think


-- You can personalise the installation by changing these attributes,
-- which are persistent and may be removed from the Startup after a reload.
local attr = luup.attr_set

-- Geographical location
attr ("City_description", "Greenwich")
attr ("Country_description", "UNITED KINGDOM")
attr ("Region_description", "England")
attr ("latitude", "51.48")
attr ("longitude", "0.0")

-- other parameters
attr ("TemperatureFormat", "C")
attr ("PK_AccessPoint", "88800000")
attr ("currency", "�")
attr ("date_format", "dd/mm/yy")
attr ("model", "Not a Vera")
attr ("timeFormat", "24hr")

-- Any other startup processing may be inserted here...
luup.attr_set ("openLuup.Historian.Directory","history/")
luup.log "startup code completed"




Added CurrentSetpoint to read   "*.*.{CurrentLevel,CurrentTemperature,CurrentPressure,CurrentSetpoint}",  -- temperature, humidity, generic sensors, ... and reloaded luup engine

There are various CurrentSetpoint whisper files which start with nil but none that are preceeded by controller number ? Have tried to attach a snapshot.


Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #10 on: July 13, 2018, 09:38:03 am »
Those seems to be duplicates. I presume that they are not updating.  Can you check that and also their creation times cf. the real ones? 

Have you added one of the bridged Veras since starting to play with this?

Feel free to delete the nil ones and see what happens after a restart.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #11 on: July 13, 2018, 11:10:21 am »
Just reread your post and seem to have missed the first line, sorry, but fwiw did not add any veras to muck things up.

Using Winscp logged in with sudo -s in Advanced/SCPShell/Shell (couldn't delete files with the normal pi login). Deleted all nil files and restarted. Changed the setpoint and am now seeing the new values in Console. Grafana however has lost the data and is currently looking into empty arrays and ... the nil files have returned. Will see if it populates over time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6114
  • Karma: +273/-69
  • "Less is more"
Re: Data Historian
« Reply #12 on: July 13, 2018, 11:39:45 am »
Sorry this is causing trouble.  A few non-archived variables, I had expected (because I don't have thermostats or other exotic devices.)  But this nil nodename is odd.

Can you confirm the number of Veras you have bridged (looks like at least two) and also show me the Variables page for both of those devices?  A Startup Log might help me pinpoint the issue (also the first part of the main log, until all the plugins are initialised.)   It's not something I've seen on any of my systems, but hey, that's what beta testing is for.  (You DID know that you were a beta tester?)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #13 on: July 13, 2018, 12:31:52 pm »
No trouble at all. Honored if truth be told : )

There are three veras bridged but Grafana is only looking to two at present tracking CurrentTemperature and CurrentSetpoint from two thermostats and CurrentSetpoint SwitchPower1.Status from two relays. Please see attachments - hope it's what you are looking for.

« Last Edit: July 13, 2018, 01:41:55 pm by powisquare »

Offline powisquare

  • Full Member
  • ***
  • Posts: 126
  • Karma: +1/-0
Re: Data Historian
« Reply #14 on: July 13, 2018, 12:33:41 pm »
...