We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287580 times)

Offline save

  • Sr. Newbie
  • *
  • Posts: 28
  • Karma: +0/-0
datamine graphic
« Reply #465 on: September 20, 2012, 04:43:15 am »
Hello,

I have installed datamine aplication.  I attache the graphic apperars using a AEON labs power meter. What kind of units are -999999 in the Y axis?
The meter runs Ok and it should appear units in Watt as 200 or similar...



Regards

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #466 on: September 20, 2012, 05:37:43 pm »
Hi @save

One thing I'm learning is that everything to do with the data used by this plugin, pretty much revolves around the data captured and stored in the /dataMine/ directory on your Vera.

Can you use WinSCP or alternative to confirm that there are a) corresponding named text files for each variable you're logging in there and b) if so, do they have data stored within them

I had the very same issue and it was because the files were not here

Offline save

  • Sr. Newbie
  • *
  • Posts: 28
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #467 on: September 21, 2012, 11:12:09 am »
Thanks

How must I use WinSCP, must I connect Vera USB port to the computer?

Regards

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: datamine graphic
« Reply #468 on: September 21, 2012, 01:46:41 pm »
I have installed datamine aplication.  I attache the graphic apperars using a AEON labs power meter. What kind of units are -999999 in the Y axis?
The meter runs Ok and it should appear units in Watt as 200 or similar...
dataMine appears to be only getting data for one channel - the one with the green tick. Note that dataMine only logs changes to channels - I would expect a energy monitor to change quite often, but mine can be static for a while if nothing is changing (and it also depends on how the HEM is configured to send data).

Now that you've had dataMine running for a day or so, has anything changed? Does it still do the same thing? What happens if you only graph the channel(s) that has a green tick?

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #469 on: September 21, 2012, 05:15:15 pm »
Hi Chris,

When you have time, please could you let me know the naming convention I need to use for the .csv files that will store the 8 months worth of Current Cost EnviR energy data I'd like to extract from an RRD table and use with you app.

* How do you think DataMine will cope with such a large amount of data (as you know energy monitors can produce a lot)?
* Should I try to strip out all the duplicate values as i guess theoretically they're worthless?
* Are there any plans to manage/graph data thats well over a month old, so people can see the history more easily?
* Have you thought about adding an option to compare previous captured periods?


Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #470 on: September 21, 2012, 05:33:45 pm »
* How do you think DataMine will cope with such a large amount of data (as you know energy monitors can produce a lot)?
It should be fine. I have been logging my energy monitor for the past 11 months and it works just fine. The energy monitor isn't my highest data producer - some of my other sensors (the light sensor is the highest producer) produce up to 120k files per week.
dataMine does sub-sample data to produce graphs with about 700 entries (from memory) and keep the amount of data transfer down, and the speed up. Longer term data files (sort of history files) are coded, but aren't generated automatically (it's something I need to finish).

* Should I try to strip out all the duplicate values as i guess theoretically they're worthless?
Up to you, but I probably would reduce the data to changes only (which is what Vera/dataMine will produce).

* Are there any plans to manage/graph data thats well over a month old, so people can see the history more easily?
Yes - it's actually coded to produce hourly/daily history files which speed up production of longer term graphs. There are some issues with generating this as Vera has a habbit of resetting itself (that said, my Vera 3 seems a LOT better than my Vera 2). This just makes it a bit more difficult - not impossible though. I'm currently trying to get some issues sorted out with the GUI, then I will work on this. The raw files are the important ones though - the other files can be generated any-time.

* Have you thought about adding an option to compare previous captured periods?
It's not something I've thought of, but should be possible. I'm trying to get the GUI into better shape so that I can add more features - the current js library I'm using has issues and I'm currently looking at another library as the existing supplier isn't fixing the issues quite as fast as I'd like.  :(

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #471 on: September 21, 2012, 05:40:15 pm »
Thanks Chris,

Last one - Do I just paste the old values in the current live file?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #472 on: September 21, 2012, 05:52:05 pm »
Last one - Do I just paste the old values in the current live file?
The files have to be generated with a separate file each "week". That's the [R2228] type bit on the end of the file - it's just the current unit time divided by the number of seconds in a week (604800).
This just makes it quicker for the plugin to search through the data - a single file would take "forever".

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #473 on: September 21, 2012, 06:11:04 pm »
Sorry Chris, I'm lost now (my lack of Linux experience),

If I'm looking to create say 30 weekly files, I should aim to pick the first unix date time entry in each weekly file and then divide that value by 604800, and then just put an 'R' at the beginning? So 1337265267 becomes "Name Of Variable [R2211]"
« Last Edit: September 21, 2012, 06:26:32 pm by parkerc »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #474 on: September 22, 2012, 06:00:38 am »
Sorry Chris, I'm lost now (my lack of Linux experience)
It's not really a linux issue - (so called) unix time is used reasonably universally...

If I'm looking to create say 30 weekly files, I should aim to pick the first unix date time entry in each weekly file and then divide that value by 604800, and then just put an 'R' at the beginning? So 1337265267 becomes "Name Of Variable [R2211]"

Basically, yes.
Somewhere I actually have a C program to do this as I used to store data in a MySQL database in an older system I had. Anyway, assuming your data is in chronological order, then you want to do the following -:
Code: [Select]
for each time/variable pair do {
    if value != lastvalue then {
        lastvalue = value
        week = (int)(time / 608400)
        filename = variable name [Rweek].txt
        open file(filename)
        write time,value<CR>
        close file
    }
}

I hope my psuedo-code makes sense  ;) So long as the data is in the right order, you should end up with a bunch of files with your data that should be compatible with dataMine.

Cheers
Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #475 on: October 10, 2012, 02:36:04 am »
Thanks

How must I use WinSCP, must I connect Vera USB port to the computer?

Regards

@save - Sorry, It doesn't look like your question was answered.

With WinSCP, both the PC running it and your Vera need to be on the network, next you need to create a connection to your Vera using it's IP and the root name and password credentials which you can find here... http://wiki.micasaverde.com/index.php/Logon_Vera_SSH#Can.27t_find_the_root_password.3F

Once inside navigate to the dataMine folder and you can see the logs in there.

Offline slajgaj

  • Full Member
  • ***
  • Posts: 176
  • Karma: +2/-13
Re: New dataMine graphing and logging plugin
« Reply #476 on: October 16, 2012, 05:31:48 am »
Hi Chris!

I disabled any device logging, but the variable will still be listed in the graph page dropdown.
How to clear/delete from the graph page list? I don't like to see, because wrong data variable.

Best R.
SL

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #477 on: October 16, 2012, 05:40:28 am »
Chris has not implemented deleting of values that are not being recorded in the current datamine version. The reason for this was to archive values for historic reasons. The only way to whip these values is to modify the file that these values were recorded in. Most likely to make sure that your other data is not lost is to just ignore the older values.

- Garrett

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #478 on: October 16, 2012, 12:35:29 pm »
I will change the way this works with the next GUI version. The original concept was to leave the old channels listed for historical reasons, but there's probably a better way to handle this... The new GUI allows me to add more functionality, so I'll look at handling historical data differently if it's needed...

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #479 on: October 16, 2012, 01:29:00 pm »
I will change the way this works with the next GUI version. The original concept was to leave the old channels listed for historical reasons, but there's probably a better way to handle this... The new GUI allows me to add more functionality, so I'll look at handling historical data differently if it's needed...

Chris

Chris,
I suggest creating a pane in the left side called 'disabled channels'. Then allow users to disable a channel which is then automatically moved to that pane (which can be collapsed to save space). This way, it is still available when/if needed but does not get in the way when it is not.