We have moved at community.getvera.com

Author Topic: openLuup: Data Historian  (Read 7411 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #90 on: September 21, 2018, 06:00:55 pm »
I did once change all the device numbers over 10000, but when you add a device the Vera just keeps numbering up.

There's a system attribute called Device_Num_Next.  After backing up, you could try changing this in the user_data.json file, or even on the fly with luup.attr_set().

I don't know what would happen if you ran into an existing device, though, but I imagine that you must have some fairly large gaps in the device numbering!

__________________

Edit: tried luup.attr_set() but didn't work.
« Last Edit: September 22, 2018, 08:40:52 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup: Data Historian
« Reply #91 on: September 24, 2018, 04:29:39 am »
Thanks, I'll give it a shot. All devices could fit in two digits indeed.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup: Data Historian
« Reply #92 on: September 24, 2018, 04:48:50 pm »
Hi akbooer,

It looks simpler (and less risky) if I can change the offset in VeraBridge. Looking at the current code this gets calculated again after each restart. Could it be made the initial offset gets calculated at first install and then taken from the off-set variable? This way you can also remove a VeraBridge without impacting the numbering of any remaining. I must admit I do not understand the role of the Primary Bridge (OFFSET == BLOCKSIZE).

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #93 on: September 25, 2018, 06:17:48 am »
Could it be made the initial offset gets calculated at first install and then taken from the off-set variable? This way you can also remove a VeraBridge without impacting the numbering of any remaining.

That's an interesting thought, and certainly would fix that particular problem.  It may raise other issues - I'll have to look closer at that.

Quote
I must admit I do not understand the role of the Primary Bridge (OFFSET == BLOCKSIZE).

Not sure I understand this?  Are you asking why the first bridge has an offset at all?  Clearly, native plugins and virtual devices start numbering in openLuup using low numbers, so any bridge variables have to be offset.  The only concession made to the first bridge is that the Zwave controller gets mapped to device #1 so that somw exotic plugins, which directly use the Zwave controller, work.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup: Data Historian
« Reply #94 on: September 25, 2018, 06:24:24 am »
I must admit I do not understand the role of the Primary Bridge (OFFSET == BLOCKSIZE).

Not sure I understand this?  Are you asking why the first bridge has an offset at all?  Clearly, native plugins and virtual devices start numbering in openLuup using low numbers, so any bridge variables have to be offset.  The only concession made to the first bridge is that the Zwave controller gets mapped to device #1 so that some exotic plugins, which directly use the Zwave controller, work.
No, I get why even the first needs an offset so it won't clash with the openLuup native devices. It is that bit of special logic I did not get the reason for. But  that Zwave device is why. Learning again  :D

Cheers Rene
« Last Edit: September 25, 2018, 06:26:48 am by reneboer »
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #95 on: September 25, 2018, 06:51:23 am »
An obvious way to skip a VeraBridge block of device numbers is to install another copy of the bridge.  The blocks are currently assigned from lowest to highest VeraBridge plugin id, so you could, for example, use bridges 1 and 3, simply not mapping bridge 2 to a remote Vera at all.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup: Data Historian
« Reply #96 on: September 25, 2018, 09:24:08 am »
Hi akbooer,

I fear it is a bit more tricky than that. Looking at the historian files I see that the devices from the first bridge with numbers over 10000 show up with just the first four digits with the second bridge vera number. I.e. device ID 13523 of vera xxxx1711 show on the History DB page as xxxx0073.3523.EnergyMetering1.Watts where xxxx0073 is the vera number of the second bridge. Maybe getting the device numbers on my Vera down is indeed the best option. Unless you want me to be your guinea pig for this scenario.

Also looking at the History DB page the page shows links like  20m:30d,3h:1y,1d:10y. However, clicking them only produces empty graphs. Also for the openLuup native devices. The same parameters do show a graph on the Historian page. Any idea? Or do I need to install a graphana system or so?

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #97 on: September 25, 2018, 12:48:30 pm »
I fear it is a bit more tricky than that. Looking at the historian files I see that the devices from the first bridge with numbers over 10000 show up with just the first four digits with the second bridge vera number. I.e. device ID 13523 of vera xxxx1711 show on the History DB page as xxxx0073.3523.EnergyMetering1.Watts where xxxx0073 is the vera number of the second bridge.

Yes, I knew that something like that would happen with Historian.  What would be needed here is to identify children of bridge devices by their parent.  I didn't do that initially because it is harder - you have to search up the tree for grandparents, etc. (OK, not too hard.)  However, these name space/number conversions actually take place in both directions between three different naming conventions.  I haven't yet worked out whether this is actually possible to do...

Quote
Also looking at the History DB page the page shows links like  20m:30d,3h:1y,1d:10y. However, clicking them only produces empty graphs. Also for the openLuup native devices. The same parameters do show a graph on the Historian page. Any idea? Or do I need to install a graphana system or so?

That's much more concerning.  Do none of them work?  Do those links show number of points updated?  There should be a counter.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup: Data Historian
« Reply #98 on: September 26, 2018, 07:04:01 am »
Also looking at the History DB page the page shows links like  20m:30d,3h:1y,1d:10y. However, clicking them only produces empty graphs. Also for the openLuup native devices. The same parameters do show a graph on the Historian page. Any idea? Or do I need to install a graphana system or so?

That's much more concerning.  Do none of them work?  Do those links show number of points updated?  There should be a counter.
Go figure, today there is data showing?!? It seemed to have needed an extra luup restart or so as it is recording from the time I updated ALTUI yesterday.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #99 on: September 26, 2018, 07:19:43 am »
Go figure, today there is data showing?!? It seemed to have needed an extra luup restart or so as it is recording from the time I updated ALTUI yesterday.

Well, that's a relief!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #100 on: November 27, 2018, 10:55:39 am »
Data Historian now supports mirroring to external Grafana Graphite and InfluxDB databases.  (as of Development branch v18.11.26)
« Last Edit: December 14, 2018, 12:20:50 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline bruring

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +2/-0
  • Why not automate everything?
Re: openLuup: Data Historian
« Reply #101 on: November 27, 2018, 11:55:31 am »
Great! I'm going to try it on InfluxDB!

Does that also support custom date ranges (not only ... so far ranges) in Grafana do you know?
« Last Edit: November 27, 2018, 11:58:13 am by bruring »
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline bruring

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +2/-0
  • Why not automate everything?
Re: openLuup: Data Historian
« Reply #102 on: November 27, 2018, 12:08:29 pm »
Another question, I've been logging my sensors with Historian in a Whisper database for quite some time now.

If I look at graphs in Grafana and look at larger timespans, I notice that sometimes a sensor logs an invalid value (like a way too high temperature or light measurement) which gives a big peak in the graph, flattening all the other measurements.

My question is twofold:
- is there a way to remove invalid data from the whisper files? I've tried this with whisper-tools, but it gives me errors about not readable metadata.
- is there a way to specify a range for the logged variables, thus ignoring and eliminating the invalid values from getting stored in the DB in the first place?

Thanks,

Joris
Trying to make my house smarter than myself since 2012.
Vera Lite & Vera Edge UI7, 15x Fibaro, 10x Aeotec, 6x Zwave.Me, 3x GreenWave, 3x TKBHome, 1x Y-CAM, 1x Netatmo, Logitech Harmony, Google Home. ESXi with pfSense, openLuup, AltUI, Grafana and Vera Concierge amongst others.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #103 on: November 27, 2018, 12:10:52 pm »
Does that also support custom date ranges (not only ... so far ranges) in Grafana do you know?

Not sure I fully understand.  The date range as applied by the Grafana menu bar works, as do the per-graph duration and timeshift fields.  I haven't tried an explicit time interval in the metric query line on a graph, since I'm not really SQL-savvy.


3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Data Historian
« Reply #104 on: November 27, 2018, 12:40:56 pm »
If I look at graphs in Grafana and look at larger timespans, I notice that sometimes a sensor logs an invalid value (like a way too high temperature or light measurement) which gives a big peak in the graph, flattening all the other measurements.

Yes, this is, IMHO, a ZWave transmission problem.  It's particularly prevalent with some devices, notably, for me, energy meters.  It is not an historian problem, per se. It is, however, a nuisance.

Quote
- is there a way to remove invalid data from the whisper files? I've tried this with whisper-tools, but it gives me errors about not readable metadata.

Yes, there is.   And I use it about once a week to fix such problems.  You can't use the Whisper tools, since the database is not binary compatible.  I have, however, written an openLuup CGI, originally for DataYours, which allows editing of the data.  I've made a version which is tailored towards the Historian database.  You exercise the basic editing functionality through an HTML form - I have a crude webpage which suffices, but really would like to do better.  However, I'm thinking that Graphite/InfluxDB mirroring, means that you could use the standard tools instead.

Quote
- is there a way to specify a range for the logged variables, thus ignoring and eliminating the invalid values from getting stored in the DB in the first place?

I wrote a user-defined processing module for DataYours, which would allow just this, but, in the spirit of recording what's actually received, rather than what you would like to be received, I haven't implemented that for the Historian.  There's a philosophical discussion to be had there (one which has been going on here on this forum for at least five years in the context of the dataMine plugin, see: http://forum.micasaverde.com/index.php/topic,14692.0.html)

I'm open for good suggestions as to how to proceed.

So, to recap, there is a simple (and crude) solution, which I'm happy to share, if that addresses your need.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.