We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287702 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #15 on: September 12, 2011, 05:44:29 pm »
Updated software version.

This fixes a couple of bugs if the config file is not present.
It also improves the graphs for light switches and other status type data by eliminating sub sampling for graphs over shortish time periods. This automatically gets enabled for all "Status" and "Watts" variables, but I will look at better ways of doing this.

If you're upgrading, you should only need to upload the J_DataMine1.js and L_DataMine1.lua files.

Edit: File removed as there's now an updated version posted in this thread, alternatively, visit http://code.mios.com/trac/mios_datamine
« Last Edit: September 30, 2011, 06:29:35 pm by Chris »

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #16 on: September 13, 2011, 06:30:54 am »
There are no data files in /dataMine. EnviR update interval: 6 seconds

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #17 on: September 13, 2011, 08:51:50 am »
dataMine works by logging changes, so it's not quite as simple as "6 seconds". If it doesn't change, then it still won't log anything until the temperature does change.
That said, if you've been running this for a long time, it's unlikelt the temperature hasn't changed, so there's quite likely a problem.

Anyway, a few questions might help me out...
  • Did you try logging a light switch to see what that does? It's simple to ensure we definately get variables changing. b
  • When you select a variable for logging, does the dataMineConfig.json get written into your /dataMine directory?
  • Are there any "can't open file for write" errors in your logfile? This should be generated if dataMine can't write to the file. Any other errors loggedy dataMine?
  • When dataMine starts, it prints a list of all the variables it's monitoring - like this -:
    Quote
    luup_log:30: dataMine: Watching: D[5] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts]
    luup_log:30: dataMine: Watching: D[5] S[urn:upnp-org:serviceId:SwitchPower1] V[Status]
    Do you get anything like this?

I can add more logging to try and work out what's happening, but there should be sufficient logging already (at least most errors are logged). It would great if you can send me any dataMine log events - they should all start with dataMine...

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #18 on: September 13, 2011, 04:29:40 pm »
Quote
dataMine works by logging changes, so it's not quite as simple as "6 seconds".

Yes, I know - but the EnviR temperature sensor is sufficiently flaky: :)
http://forum.micasaverde.com/index.php?topic=6752.msg46748#msg46748

The latest version of your plugin seems to work:

EnviR Temperature - CurrentTemperature [R2175].txt:
Code: [Select]
1315939858,25.7
1315939876,25.5
1315939900,25.4
1315939918,25.5
1315939930,25.6
1315939936,25.5
1315939948,25.6
1315939961,25.4
1315939967,25.6
1315939991,25.7
1315939997,25.5
1315940009,25.6
1315940027,25.8
1315940047,25.7
[...]

Corresponding GDL output:

Code: [Select]
Sep 13 20:50:58 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939858 EnviR_Temperature 25.7
Sep 13 20:51:16 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939876 EnviR_Temperature 25.5
Sep 13 20:51:40 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939900 EnviR_Temperature 25.4
Sep 13 20:51:58 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939918 EnviR_Temperature 25.5
Sep 13 20:52:10 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939930 EnviR_Temperature 25.6
Sep 13 20:52:16 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939936 EnviR_Temperature 25.5
Sep 13 20:52:28 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939948 EnviR_Temperature 25.6
Sep 13 20:52:41 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939961 EnviR_Temperature 25.4
Sep 13 20:52:47 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939967 EnviR_Temperature 25.6
Sep 13 20:53:11 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939991 EnviR_Temperature 25.7
Sep 13 20:53:17 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939997 EnviR_Temperature 25.5
Sep 13 20:53:29 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940009 EnviR_Temperature 25.6
Sep 13 20:53:47 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940027 EnviR_Temperature 25.8
Sep 13 20:54:07 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940047 EnviR_Temperature 25.7

- a perfect match.

Some observations and suggestions:

  • The graph looks strange: Some x values have two y values ...
  • Add support for alpha(numeric) data (e.g., wind direction as provided by Google Weather and GWC, see attachments at http://forum.micasaverde.com/index.php?topic=7333.msg46575#msg46575)
  • Add support for timezones (unfortunately, luup.timezone doesn't work)
  • Add support for multiple graphs below each other, idea:
Code: [Select]
menu line for graph 1
graph1
Button: 'add graph'

menu line for graph 1
graph1
menu line for graph 2

menu line for graph 1
graph1
menu line for graph 2
graph2
Button: 'add graph'

...
« Last Edit: September 13, 2011, 04:31:27 pm by Ap15e »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #19 on: September 13, 2011, 06:09:26 pm »
Hey - that's great - I'm pleased it's working...  :)

The reason there are two values is to make graphs that look "correct". Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct. ie the light was off all night - it wasn't slowly turning on over night  :) So, when I come to display the graph, I add the extra values - it actually more correctly represents the data that Vera was displaying, but it's just not consistant with what's being logged. Due to the float bug in Vera, it's much easier to use the same time rather than subtract a second (or whatever) - I originally tried this, and it caused all sorts of problems.

I'd have to think about non numeric data - that's possibly not so simple.

Timezones are actually done, so you can expect to see local times in the next rev.

Multiple graphs, or multiple plots on the same graph is definately something I'm keen on. The plugin "window" is quite small though and this limits the graph size, so multiple plots seem like the way to go (I think, but I'm open to other thoughts on this as both have their benefits).



Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #20 on: September 13, 2011, 07:09:02 pm »
Quote
Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct.

You might find the GDP source code an interesting read.

You could use different plotting algorithms for binary data (light switch), discrete data (dimmer), continuous data (temperature) and alpha(numeric) data (wind direction).

Question:
Does deactivating the logging require a LuaUPnP restart/reboot?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #21 on: September 14, 2011, 05:30:25 am »
Quote
You could use different plotting algorithms for binary data

The flotr package doesn't have a different algorithm, so I've effectively done this myself. I may however move some of this into the JS to try and reduce the load on Vera, but JS isn't the fastest thing around either - it's just a case of apportioning the functionality to the most appropriate place in the system.  I'll try and take a look at GDP.

Quote
Question:
Does deactivating the logging require a LuaUPnP restart/reboot?
As far as I'm aware, there is no variable_unwatch function, so yes, you need to restart Luup for changes to take effect. If you're aware of another method to stop watching a variable, then I'd be happy to implement it.

Offline mcvflorin

  • Administrator
  • Hero Member
  • *****
  • Posts: 1755
  • Karma: +11/-3
Re: New dataMine graphing and logging plugin
« Reply #22 on: September 14, 2011, 06:50:56 am »
If you're aware of another method to stop watching a variable, then I'd be happy to implement it.

You could create a flag e.g. WATCH_VARIABLE and check it at the beginning of the callback function. If the flag is 0/false, return immediately. It's not the most elegant solution, but it works.

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #23 on: September 14, 2011, 11:49:02 am »
Quote
Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct.

Correct. But why are there "diagonal" lines?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #24 on: September 14, 2011, 01:48:48 pm »
Quote
You could create a flag e.g. WATCH_VARIABLE and check it at the beginning of the callback function. If the flag is 0/false, return immediately. It's not the most elegant solution, but it works.

Actually, this is easy - I already have this flag in my config structure (i.e. logging on/off). The config structure gets scanned in the "watch" function, so I can easily add this. It's not the best solution since it still means Vera is going to the (wasted) effort of calling my plugin, but it will work.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #25 on: September 14, 2011, 02:01:28 pm »
Quote
Correct. But why are there "diagonal" lines?

Good question! I think it's probably to do with the way I sub sample the data, and where I record the last value - these aren't in the same place, so sometimes the extra point I put in is not for the last value that was graphed, but the last value in the file.

To avoid sending too much data from Vera to the UI, I sub sample the file. It's simply ("End Time" - "Start Time") / 500 (500 being roughly the number of points on the graph). For data that's roughly evenly logged, this works fine, but for other data where you get a lot of changes over a short time, this falls down. There's a slightly modified algorithm for switch data to avoid this.

I might play around with this and see if it can be removed (or reduced). I don't know how Vera gets on with concatenating large strings which is mainly why this is implemented this way.

Offline kskenyon

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #26 on: September 20, 2011, 09:39:01 am »
I have dataMine working (mostly) on a Vera2@1245 with a USB drive.

Is there a way to delete an item from being logged without editing the files directly?

One of my variables will not stay green when it is clicked in the Config list.  I assume it is not being saved for some reason.  Other variables are working well.

Is it possible to use the local time zone for graphing?  My brain is not used to working in GMT even though the calculation is simple.

Nice work.  I think this will be a popular plugin when it's ready for prime time.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #27 on: September 20, 2011, 11:30:41 am »
Quote
Is there a way to delete an item from being logged without editing the files directly?
Clicking on the variable so that it goes yellow will disable logging. It won't however delete the data files that have already been generated. I was considering adding this, but decided to simply stop logging. Is this what you mean, or do you want the data files deleted?
--- EDIT--- Actually, in the current public version, you will also need to restart Lua for this to take effect. In the next version, this will be automatic.

Quote
One of my variables will not stay green when it is clicked in the Config list.
Can you describe what happens? You click on the variable, it initially goes green (?), but if you then close and restart dataMine, it's no longer green (?) or something else?
I had an error a few days back where somehow the config file got written with no data in it, and that caused an error that stopped variables being logged (although not just on a single variable), otherwise I've not seen any issues with this.

Quote
Is it possible to use the local time zone for graphing?
Not in the version you've got, but it is implemented in the version I'm working on at the moment. Hopefully I'll get this out in a few days time - I'm looking at ways to add multiple channels to a single graph. If it starts to take too long I'll upload an intermediate version to add this feature.

Quote
Nice work. I think this will be a popular plugin when it's ready for prime time.
Thanks.
« Last Edit: September 20, 2011, 01:29:13 pm by Chris »

Offline kskenyon

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #28 on: September 21, 2011, 12:05:33 am »
One of my variables will not stay green when it is clicked in the Config list.
Can you describe what happens? You click on the variable, it initially goes green (?), but if you then close and restart dataMine, it's no longer green (?) or something else?
I have two TED 5000 Devices that are updating OK in the main UI4.  One seems to update the db in dataMine, the other does not.  That's why I wanted to delete the variable from the graph pull down list and try again.  It took several tries to get the variable to stay green in the Config and now it seems to stay green but does not update the database. Not sure what I did to get it to stay green.  Some combination of opening and closing windows until it stayed green.  The other one is working fine and they are otherwise identical devices. I thought about deleting everything that dataMine has created and starting over but it has been collecting data on other variables for a while.  Can I just delete a .txt file for a variable without messing everything else up?



Thanks for the upcoming fix for the TZ.

Offline kskenyon

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #29 on: September 21, 2011, 10:13:01 am »
Not sure what I did but it seems to be working now.  I'd still like to delete some "mistakes" I made from the Graphing pull down menu.