We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287582 times)

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #420 on: August 19, 2012, 10:51:50 am »
Sorry for the slow replies. I've been working abroad for the past couple of weeks
That sounds like fun... but my wife is against me being with other broads. Oh wait, maybe you meant overseas?  :P

Any new updates to test?
Not yet - sorry. I blame my employer... I'm at home for the next week so if the other boss doesn't have too much for me to do at home, I'll try and get a new version out that can work with the USB logging.
Well there's your problem... letting work get in the way!  ;)

Seriously though,
I stopped using the USB for Vera logging and have no had the problem since. So as much as it pains me bag on Linux  ;D the problem is Linux. If you can find a way for dataMine to use the already mounted USB drive, that would be perfect.

Also, a while back you mentioned adding some more features with the graphs?


Offline bfromdruten

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #421 on: August 19, 2012, 11:19:16 am »
Aaron, I have had the same problems. My solution is to create two partitions on the usb stick. One partition for usb logging, one for datamine. I used 'EaseUS partition' to get the stick partitioned. Works fine since.

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #422 on: September 02, 2012, 02:08:22 pm »
Chris,
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

I removed the mount for Vera, weeks before so only dataMine is using the USB. - dataMine worked fine for a while.

Are there any updates coming, I need to wipe the install and retry but wanted to check with you first.

thx

Offline rarev

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #423 on: September 02, 2012, 03:46:20 pm »
Chris,
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

I removed the mount for Vera, weeks before so only dataMine is using the USB. - dataMine worked fine for a while.

Are there any updates coming, I need to wipe the install and retry but wanted to check with you first.

thx

Hi Chris,
Having the same problem as Aaron. A red ! next to my channels. Also I'm having trouble selecting different channels. The channels are reset and I cannot unselect channels.

BTW I have a Veralight and USB logging is off

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #424 on: September 02, 2012, 04:56:04 pm »
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

Hi Aaron,
What do you mean by "triggers" - do you just mean "variables being logged"?

The red ! means that there have been no updates to the variable in the last day (or since Lua restarted if it was soon). For analogue variables (temperatures, humidity, etc), this rarely happens. The yellow ! means that there hasn't been anything between 1 hour and 1 day - this is reasonably common (most of my variables show this).

dataMine uses a callback feature in Vera, so it just gets notified when the variables change.  The most likely reason for variables not being logged therefore is if for some reason the channels have been disabled. Can you take a look at the log - when dataMine starts it logs all the variables that it has asked Vera for a callback. If the variables are being logged, then dataMine really should receive the callbacks unless Vera/Lua is doing something strange (very unlikely).

I have added some code to support the USB logging, but I haven't had the chance to test it out yet (just a bit busy with other things at the moment unfortunately).

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #425 on: September 02, 2012, 05:20:40 pm »
Yes, I meant Channels - not triggers.

They are enabled (green) in the config. The Vera is rebooted each night so everything should be 'fresh' (memory, etc).

I get the Red ! on 4 channels - see attached jpg

All the .log files have dates of 7/30 on them... and the LuaUPnP.log is over 1GB!

The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #426 on: September 02, 2012, 05:29:29 pm »
I'm having trouble selecting different channels. The channels are reset and I cannot unselect channels.

Hi @rarev,
What do you mean by "the channels are reset" - I would think this means they aren't selected, but the next part says you can't unselect channels.

Cheers
Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #427 on: September 02, 2012, 05:53:05 pm »
The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.
Stupid question, but I assume that all these channels are actually updating in Vera? I see one of the channels is a lock status, so I assume this is updating when you change the state?
Can you look at the log to see what channels dataMine says it's logging?

Offline mjcohnmd

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #428 on: September 09, 2012, 08:40:35 pm »
I haven't read all the posts so I apologize if this question has already been addressed.

Great plugin.  I'm using it to monitor my six Wayne-Dalton WDTC-20 thermostats.   The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

Thanks,  --JC

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #429 on: September 09, 2012, 10:28:50 pm »
The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.
Stupid question, but I assume that all these channels are actually updating in Vera? I see one of the channels is a lock status, so I assume this is updating when you change the state?
Can you look at the log to see what channels dataMine says it's logging?

Yes they are showing data in Vera. two channels are from google weather which is definitely getting the correct, and updated info.
I deleted all the logs and reset the channels... now one of the channels that was not graphing is, and all the others are not. Something is going hanky

Offline rarev

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #430 on: September 11, 2012, 10:33:09 am »
Hi Chris,
Sorry for not being clear. What I meant was that the channels seemed to disappear. So I selected 5 channels and the virtual DataMine device showed "5". When I checked a few hours later the counter was 0. A reinstall fixed somehow, so I'm afraid I can't reproduce it.

As for the exclamation marks,  I seem to have the same problem as Aaron.

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #431 on: September 11, 2012, 11:50:43 am »
Sorry for not being clear. What I meant was that the channels seemed to disappear. So I selected 5 channels and the virtual DataMine device showed "5". When I checked a few hours later the counter was 0. A reinstall fixed somehow, so I'm afraid I can't reproduce it.

I had this problem... I think it was that I had mounted the Vera logs to USB... thus it was 'sharing' a mount point with dataMine... about every week or so my channels would go bye-bye. After removing the Vera logs, this stopped and they have not disappeared since. One solution provided was to partition the USB drive and mount to different partitions... but Chris said he would make it so we don't need to us a mount point... I'm waiting for this :)

As for the exclamation marks,  I seem to have the same problem as Aaron.

Still happening for me.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #432 on: September 11, 2012, 12:50:12 pm »
Thanks guys,
Unfortunately, I really need more information. It seems something is awry, but god knows what. Unfortunately we really need to look in the log file. There are two things that we need to look for -:

First is the startup - when dataMine starts, it logs all the variables that it's actually logging
Code: [Select]
50 09/11/12 17:27:13.328 luup_log:87: dataMine: Initialising dataMine System <0x2b635680>
50 09/11/12 17:27:13.373 luup_log:87: dataMine: Mounting dataMine storage (/dev/sda1) <0x2b635680>
50 09/11/12 17:27:15.192 luup_log:87: dataMine: Watching: D[5] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x2b635680>
50 09/11/12 17:27:15.193 luup_log:87: dataMine: Watching: D[11] S[urn:cd-jackson-com:serviceId:Irrigation1] V[WaterButtVolume] <0x2b635680>
50 09/11/12 17:27:15.214 luup_log:87: dataMine: Startup complete <0x2b635680>

Next is if there are any problems that dataMine encounters when it's logging data.  Assuming the startup shows that all the variables you think are being logged, really are beinmg logged, then Vera will call dataMine when they change (this almost has to happen - if it doesn't, it's a Vera problem). When dataMine receives this callback, it tries to correlate the data with the variables being logged, and find the file to write the data into. This code can log a couple of errors.

Lastly, is if there's some sort of other error, the Vera will log it. These are more difficult to find, so I'll ignore them for now....

To get the log file entries that dataMine logs, you need to run the following in WinSCP (or some similar browser) -:
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
The other question is, is there anything "special" about the variables that aren't working? Are they character based (eg a thermostat mode, or wind direction), or do they have funny names with special characters (ie not alphanumeric, but say colons or /[]\ type characters).

I'll see if I can find a simpler way to get the logged data, but in the meantime any information that you can provide would be great... 

Cheers
Chris


Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #433 on: September 11, 2012, 12:54:19 pm »
The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

I'll look at adding two options - one to filter "out of range" data (I'll provide the ability to set these ranges). The other is the ability to export the data as CSV - this is something I've been meaning to add anyway... I'll try and add this in the next few weeks (I'm currently a bit busy with work and getting heating into the new extension before winter sets in).

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #434 on: September 11, 2012, 01:08:27 pm »
The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

I'll look at adding two options - one to filter "out of range" data (I'll provide the ability to set these ranges). The other is the ability to export the data as CSV - this is something I've been meaning to add anyway... I'll try and add this in the next few weeks (I'm currently a bit busy with work and getting heating into the new extension before winter sets in).

Cheers
Chris

Just go to Home Depot and get a portable heater... you have work to do ;-)