We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 290498 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #360 on: July 08, 2012, 05:54:25 am »
Great plugin.  I have it working except where it comes to adding/removing channels or changing the graph type.  I can only add channels (for those added, graphing works) and only if I have drop down for the room assignment for the dataMine module open (took me a while to stumble on that).
I'm not sure what you mean by "drop down for the room assignment"? dataMine should depend on any other settings in UI5.

If I go back into the config, channels that are loaded (and showing in the graph) are not highlighted.  Also, if I click on one (turns green) and click on it again, nothing happens (shouldn't it turn yellow per the PDF directions?).
Just to confirm - if you click on a variable, it turns green, clicking again doesn't change anything (??), and if you close the config and open it again, then this isn't green any more?
However, if you go to the GUI, then all the channels that you've selected are showing in the list and are graphing correctly?

The GUI and the config use the same data for showing channels, so if the GUI is showing data, then the config should also be showing things in green, so these two points confict (which is why I'd like to confirm I understand what you see).

It's possible that there's a mixed up install - I would have expected the install from apps.mios.com to have overwritten any files loaded separately though.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #361 on: July 08, 2012, 06:02:57 am »
I have data in an rrd dB (which has been capturing energy usage and room temperatures for quite a while now ) on another server on my home network.

Any ideas how I can maybe make that accessible to datamine?

What do you mean "accessible"? You could probably convert the data over to the dataMine file format reasonably easily (dataMine just uses CSV files), but if you mean is it possible for dataMine to read the rrd data, then that's a different story.

If you want to convert existing data into a format that dataMine can read, I can tell you the file name/format in more detail...

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #362 on: July 08, 2012, 09:25:36 am »
..... if you mean is it possible for dataMine to read the rrd data, then that's a different story.

Does it have a happy ending? The external file could potentially be a csv or XML on this external server (same subnet etc.) ?

If you want to convert existing data into a format that dataMine can read, I can tell you the file name/format in more detail...

There seems to be ways to do rrd to csv so that might help me/us?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #363 on: July 08, 2012, 09:37:50 am »
Does it have a happy ending? The external file could potentially be a csv or XML on this external server (same subnet etc.) ?

dataMine does store it's data as a CSV file - actually, a number of CSV files (one per week). dataMine only records data when it changes - of I remember correctly, rrd records data at a specific interval.  The dataMine GUI doesn't read the CSV files directly - the data gets served up via the plugin which can sub-sample if a large period is asked for, or can serve up processed data.

Chris

Offline NovaIceman

  • Sr. Newbie
  • *
  • Posts: 32
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #364 on: July 08, 2012, 11:58:35 pm »
I am using UI4 and I believe i have followed all the install directions correctly, can you help with this error.

Quote
50   07/08/12 20:52:05.852   luup_log:48: dataMine: Reinitialising configuration structure <0x402>
50   07/08/12 20:52:05.857   luup_log:48: dataMine: Startup complete <0x402>
50   07/08/12 20:52:06.081   luup_log:49: dataMine: Initialising Data Mining System <0x402>
50   07/08/12 20:52:06.083   luup_log:49: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>

thanks

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #365 on: July 09, 2012, 03:27:53 am »
I am using UI4 and I believe i have followed all the install directions correctly, can you help with this error.

Quote
50   07/08/12 20:52:05.852   luup_log:48: dataMine: Reinitialising configuration structure <0x402>
50   07/08/12 20:52:05.857   luup_log:48: dataMine: Startup complete <0x402>
50   07/08/12 20:52:06.081   luup_log:49: dataMine: Initialising Data Mining System <0x402>
50   07/08/12 20:52:06.083   luup_log:49: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>

These entries are from two separate copies of dataMine - the device numbers are different (48 and 49), so you have two copies of the device installed somehow.

Cheers
Chris

Offline JOD

  • Hero Member
  • *****
  • Posts: 1973
  • Karma: +4/-0
Re: New dataMine graphing and logging plugin
« Reply #366 on: July 09, 2012, 10:19:13 am »
@Chris,
Awesome plugin. Kudos to you.

Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.


I'm sorry, my responses are limited. You must ask the right questions.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #367 on: July 09, 2012, 11:18:52 am »
@Chris,
Awesome plugin. Kudos to you.

Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.

Unfortunately this won't work  :(  MCV use a reverse link to connect to theirs server to avoid NAT issues in firewalls and DNS problems.  Remote access isn't impossible to configure, but it requires setting up a bunch of stuff that's not so simple (Dynamic DNS and NAT in your firewall)...

Chris

Chris

Offline JOD

  • Hero Member
  • *****
  • Posts: 1973
  • Karma: +4/-0
Re: New dataMine graphing and logging plugin
« Reply #368 on: July 09, 2012, 03:34:27 pm »
Oh well.....
I can review the .txt file if I really need to check on something, just not as much fun as watching the graph.
I'm sorry, my responses are limited. You must ask the right questions.

Offline huib

  • Full Member
  • ***
  • Posts: 179
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #369 on: July 14, 2012, 04:23:26 am »
@chris,


i have got your plugin working and is helpfull

can you advise how to delete one (or two) itmes to be logged

if i select them the line will tunr green
if i de select they turn yellow

but the line is still visible on the graph

so my question is how delete the line in the graph to clear up the screen

thanks

- veralite UI5 - veralite UI7 - 9 fibaro dimmers - 7 fibaro switches - 2 eminent movement sensors - remote control - opentherm gateway - remote powermeter reading - control of whr930 unit - control of washing machine -

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #370 on: July 14, 2012, 06:23:28 am »
if i select them the line will tunr green
if i de select they turn yellow

but the line is still visible on the graph

so my question is how delete the line in the graph to clear up the screen

Unfortunately, this isn't possible at the moment - I'll add this shortly. The reason I originally didn't do this is I wanted to avoid accidentally deleting the data, but I can work around this and clearly it's a necessary feature.

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #371 on: July 14, 2012, 11:12:37 am »
Chris,
I came home after a 5 day work trip and found all Channels had been wiped from the config. This is what happened after the update. Except, this time, all the settings under 'Advanced' were there.

So, I used the built-in uninstall from the Apps tab. Then I looked at the LUA files and found there were still two dataMine XMLs and the dataMine folder is still under <root>
... it looks like maybe the uninstall is not really removing things?

One thought... I never fully reversed the initial manual install... I just installed the new code from the installer once it was posted. Is there anything that might be conflicting?

I reinstalled (using Vera's Apps tab) and reset the channels... so I'll let you know what happens.

Also, I noticed one channel that seems to have an issue. Under "Device: Front Door Lock" it shows all the normal items plus... "Service: urn:upnp-org:serviceId:HVAC_FanOperatingMode1"


« Last Edit: July 14, 2012, 11:18:05 am by Aaron »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #372 on: July 14, 2012, 02:26:15 pm »
So, I used the built-in uninstall from the Apps tab. Then I looked at the LUA files and found there were still two dataMine XMLs and the dataMine folder is still under <root>
... it looks like maybe the uninstall is not really removing things?
I don't know about the XML files - that's all part of how Vera does things, but the dataMine directory will not be removed. Firslty, when an app is removed, it doesn't run an uninstall function in the apps Lua, so there's no way or doing an app specific uninstall. Also, it's not something I would typically do as the data in the dataMine directory should still be of use even if the app gets uninstalled.

It is however strange that you're loosing your setup. I've never had any problem and I wonder if there's some sort of issue with your USB? The configuration is stored in the dataMineConfig.json file in the /dataMine directory, so it would be interesting to look in there and see what's there. It should also be backed up each time it changes, so you should be able to restore the configuration.


Also, I noticed one channel that seems to have an issue. Under "Device: Front Door Lock" it shows all the normal items plus... "Service: urn:upnp-org:serviceId:HVAC_FanOperatingMode1"
While this could be an issue with dataMine, it may also be that this service is, strangely, under this device. I have seen the occasional "strange" services under devices that you think shouldn't have such a service. For example, I've notices that with UI5, all devices have a "VideoURL" variable, which is a bit weird for devices like a temperature sensor!  You should be able to see this under the Advanced tab.

Cheers
Chris

Offline bfromdruten

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #373 on: July 14, 2012, 02:33:46 pm »
Chris,
Aarons message now makes something clear. Last couple of months I have had the same. Suddenly all channels disappeared and no data. I think I have had this 2 or 3 times. Last couple of days no problems, but currently I'm running with default dataMine folder configuration where as previously I changed the folder location. Maybe this helps.

Thanks,
Bernard

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #374 on: July 14, 2012, 02:42:50 pm »
Hi Bernard,
Thanks - I can only think that there's some issue with mapping of the USB drive. There's nothing in dataMine that deletes files, so the issue must be somewhere else. I think (hopefully) that the new version of dataMine, which does the USB mapping internally, should avoid some of these issue. Have you had any issues siince the new version was installed?

Cheers
Chris