We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287582 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #585 on: February 24, 2013, 04:57:03 am »
Thanks Aaron,
unfortunately this doesn't show anything useful...

Did deleting the files help? I suspect not, but there's always hope...  If that doesn't work, then I'd suggest copying the files onto your PC, reformatting the stick and copying the files back again. I don't know why this should be necessary, but it should solve the issue...

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #586 on: February 24, 2013, 10:42:26 am »
At 1:19pm yesterday, exactly 1 hour after I deleted the backup files, the data started logging/graphing in dataMine. I made no other changes to the Vera's config, did not reboot, nada.

So there is something going on.

I'll wait until your next version releases and then if it happens again I'll redo the USB setup.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #587 on: February 24, 2013, 12:46:51 pm »
Interesting. I'm not completely convinced it's the directory entries for 2 reasons...
  • I have around 2500 files in my directory - also with the vfat file system
  • From what I can tell from the zip of your datamine directory, the config file was still able to be written...

However, let's see how things go. I am going to restructure the "database" at some stage to put files for each logged variable into a different directory to avoid these sorts of issues. This will take a bit of thought  - ok, not much thought, but the issue is "seemlessly" moving everyones data around....

The new version will help a bit in that it will keep the number of backup files down - these can really build up, so can be an issue, but if that really is your problem, given you only had something like 125 files in the directory (from memory) the problem will likely reoccur in the near future as DM produces 1 file per week per variable...

Let me know how it goes...

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #588 on: February 24, 2013, 12:59:18 pm »
I'd suggest, at least at first, creating only a few folders to separate out things, like...

\dataMine  (current config lives here)
\dataMine\channels
\dataMine\notifications
\dataMine\history  (what is this for?)
\dataMine\configbak


Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #589 on: February 24, 2013, 01:17:30 pm »
Unfortunately, I only want to do this once as there's potential for hassle when moving all the data around... My current thought is to simply have a separate folder for each channel - there may then be a subfolder under that for history data...

History isn't released yet - it's something I've been playing with for a while, and I think it's kinda working now. It basically generates stats for the channels - things like min/max/total etc over different periods - hourly, daily, weekly... I've not looked at it for a couple of weeks as I've been focusing on other things, but it's something to roll out in the future...

Cheers
Chris

Offline skruffy

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #590 on: March 05, 2013, 03:28:24 pm »
Is there any possible way to track temperature from my thermostat down to a decimal place?  I know Vera works with the temperature as an integer, but is it possible for dataMine to report it down to one decimal place?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #591 on: March 05, 2013, 04:30:02 pm »
Is there any possible way to track temperature from my thermostat down to a decimal place?  I know Vera works with the temperature as an integer, but is it possible for dataMine to report it down to one decimal place?
dataMine will only log the data that Vera shows. So, if your thermostat/thermometer logs data with higher resolution, then dataMine will show it. Vera itself doesn't stop this, but Veras z-wave implementation does! So, I could recommend other thermometers (not thermostat) that can monitor temperature, and log it with dataMine, with less than 1 deg resolution...

Cheers
Chris

Offline skruffy

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #592 on: March 06, 2013, 05:13:45 pm »
Any recommendations you have for more accurate temperature readings would be welcome.  Something that works indoors and out would be nice.

Also, is there a way to get different variables to graph with the same scale?  In the attached image, I'm graphing temperature, but the two variables don't line up.  Sometimes it lines up, other times not.  Some of the documentation I've been reading seems out of date so I'm not sure what I'm missing.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #593 on: March 06, 2013, 05:32:41 pm »
Any recommendations you have for more accurate temperature readings would be welcome.  Something that works indoors and out would be nice.
Personally I'm using a number of 1-wire sensors and the EDS OW-SERVER. http://www.embeddeddatasystems.com/OW-SERVER-1-Wire-to-Ethernet-Server-Revision-2_p_152.html. There's a vera plugin available, and these report at 0.1degC. The server is about $100, and the temperature sensors are a couple of $?? each. The RFXCom with various temperature sensors are also probably ok, but I can't vouch for them as I've not got this...

Also, is there a way to get different variables to graph with the same scale?  In the attached image, I'm graphing temperature, but the two variables don't line up.  Sometimes it lines up, other times not.  Some of the documentation I've been reading seems out of date so I'm not sure what I'm missing.
If you save the graph, then the latest version allows you to specify which axis each channel is on, so you can be sure everything ends up on the same axis (or not if that's what you wanted). The manual is reasonably up to date - there's a couple of pictures that I haven't updated for a while, but the main bits like this are all up to date...

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #594 on: March 10, 2013, 12:10:41 pm »
Chris,
dataMine too a crap again. I attached the folder. all channels gone!

I noticed 2 things right away...

tons of Notifications.txt on 3/6, then not a single one until early morning 3/10 - and only 1 on 3/10
dataMineConfig.json.backup went from 3k to 0k on 3/6 and never changed
dataMineConfig.json is 156b ... I suspect it created a new one when I logged in today.

I have done absolutely nothing to dataMine except look at the graph once or twice over the past week or so.


I ran  cat /var/log/cmh/LuaUPnP.log | grep DataMine ... see _dataMine_log.txt file for output.



Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #595 on: March 10, 2013, 12:25:04 pm »
Hi Aaron,
There's nothing obviously wrong - the log looks fine, but for some reason dataMine seems to have reset the config file. The backups look fine, and your data appears ok, so the best thing to do is to rename the last (most recent) backup file over the dataMineConfig.json.

I don't know why it should have done this - the log hasn't captured the event so there's no way to know. Maybe I should implement an auto recovery to backup in this event rather than reinitialising the configuration...

I wonder if you have a lot of Vera restarts? Something must be different about your setup as I assume most people don't have this problem (he sais, waiting for the deluge of emails to the contrary  ;)). Something that could cause this is if Vera resets and files are open on the USB drive, it could corrupt them. This is especially "likely" with the FAT file system - using a Linux journalled file system should significantly reduce the probability of this. That said, I'm using a FAT formatted USB, and I don't have such issues - my Vera resets about once a week at the moment (and I've not upgraded to the latest Beta on my Vera 3 for this reason!).

Anyway, renaming your backup file should get you going again pretty much where you left off...

Cheers
Chris



Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #596 on: March 10, 2013, 01:05:40 pm »
I have the Vera reboot at 6am every morning... if I don't I find there are issues.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #597 on: March 10, 2013, 01:14:15 pm »
I have the Vera reboot at 6am every morning... if I don't I find there are issues.
Fair enough. I'd be surprised if this is causing dataMine issues, but at the end of the day, it's a numbers game - the more often it reboots the more likely it is that there will be issues with files being left open... I'll look at implementing an "auto resort to backup" function in the next version (not the one that will be released in the next day or so though).

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #598 on: March 10, 2013, 01:25:26 pm »
sounds good, thx for helping on this one.

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: New dataMine graphing and logging plugin
« Reply #599 on: March 10, 2013, 02:48:58 pm »
sounds good, thx for helping on this one.

Aaron, out of curiosity... are you running the "ergy" plugin? This caused my Vera to hang almost every day... now with that uninstalled it's very stable...
Good firmware is more important than good karma. (and this costed me lots of Karma)