We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287569 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #450 on: September 18, 2012, 12:55:03 pm »
To find the drive, I think the following works -:
Using WinSCP, click on the little Console / "Open Terminal" button and type "fdisk" with the USB unplugged. Then plug in the USB stick, and type "fdisk" again. You should (I hope!) see another drive installed.

Quote
It looks like I can add channels and it records them - but nothing seems to come up on the graphs (6 selected, 6 recording) no data being seen = every graph shows the time across the bottom and runs a line from a -9999989.01 value ?
From WinSCP, take a look in the /dataMine directory. This is where the files are stored and it would be worth opening one of them to see if there's any data. Note that dataMine only logs changes in value, so depending on what type of data you're logging, this may not happen too often. It doesn't log data periodically... If you're logging a light switch for example, it will only log something when you actually change the switch state.  Until there's more than 1 entry logged, the graphs don't really work.

Let me know if that helps...

Cheers
Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #451 on: September 18, 2012, 02:43:48 pm »
Thanks again Chris,

I'm sure you're sick of all the praise you've been getting for this plugin by now ;) but boy does this plugin look bloody good !!!

As a newbie, a few hours in, i have some questions for you if that is OK.

1) Out of 7 variables it says it should be recording, looking in the /dataMine/ directory, the names of only 3 are listed. The ones that are missing look to be those that are reporting the 9999989.01 values on the graph, yet 2 of them are showing a green tick to suggest they're being updated?

UPDATE 1 - I think the might have found the cause for 2 of them, as a couple of device name have an '/' in them, e.g. RFX Power ELEC2/56898. Plus I could see in the logs that datamine was trying to write to it but it was unable to open the file (not surprising as it's not there) plus I also couldn't create a version of it manually with the exact same name.

UPDATE 2 - This links to question 7 I think, as I have renamed one of the devices that had '/' in the name, which after a refresh is now visible under its new name on the Config tab and surprisingly still has the required variable highlights in green, However the graph entry has not been updated with the new name and looking in the logs it's still saying that datamine is unable to write to a file with the old nam (RFX Power ELEC2/56898....txt) etc .

2) Under the Advanced tab - LastUpdate is set to zero, should that  change with each write for a recording?

3) Under the Config tab - when selecting a variable there is no clock icon visible/available as far as I can see, only an empty square?

4) there are a lot of .backup files (17 to be exact ) in the /dataMine/ directory, is that to be expected?

UPDATE - Ok, I see now, every change in selection of variables that i want to be recorded or not etc creates a back up of the dataMinejson file

5) There is no device icon appearing for the dataMine device in the UI, is that to be expected too?

UPDATE - Looking in the JSON file that was installed from the app store, I can see the following icon listed, "flashicon":"icons\/Data_Mine.swf"., yet nothing appears

6) Is there a way to delete a graph for a variable I selected by mistake?

UPDATE - It seems the only way to do this is to restore a backup file that did not have it in, which sadly removes everything else added subsequently, are there still plans to have a field in the UI to remove/hide old graphs and variable references? ..

7) What happens if I need to rename a device, will it lose it's connection with its associated  log file in the /dataMine/ directory ?

UPDATE - As mentioned in question 1 above, it seems even if I try to change the name in the dataminjson file it is not reflected. All it looks like I can do is restore an old back up?.

8.) I have two temperature logging files, each with the same start date/time, yet depending on the order I select them to be 'graphed' the info is presented in two very different ways. Why?

I will continue to try various things and add updates , but if anyone can help me, your support would be appreciated.
« Last Edit: September 19, 2012, 05:17:20 am by parkerc »

Offline rarev

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #452 on: September 19, 2012, 05:24:23 am »

6) Is there a way to delete a graph for a variable I selected by mistake?


Couldn't find this either. Guess it's not there (yet). Would like to see this option. But deleting a graph can be done by unselecting the channel. I'd like to see the possibility to remove the channel to unclutter my list of channels.
« Last Edit: September 19, 2012, 05:27:24 am by rarev »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #453 on: September 19, 2012, 10:25:06 am »
Quote
1) Out of 7 variables it says it should be recording, looking in the /dataMine/ directory, the names of only 3 are listed. The ones that are missing look to be those that are reporting the 9999989.01 values on the graph, yet 2 of them are showing a green tick to suggest they're being updated?
UPDATE 1 - I think the might have found the cause for 2 of them, as a couple of device name have an '/' in them, e.g. RFX Power ELEC2/56898. Plus I could see in the logs that datamine was trying to write to it but it was unable to open the file (not surprising as it's not there) plus I also couldn't create a version of it manually with the exact same name.
UPDATE 2 - This links to question 7 I think, as I have renamed one of the devices that had '/' in the name, which after a refresh is now visible under its new name on the Config tab and surprisingly still has the required variable highlights in green, However the graph entry has not been updated with the new name and looking in the logs it's still saying that datamine is unable to write to a file with the old nam (RFX Power ELEC2/56898....txt) etc .
Ok - the '/' in the filename is an issue that I have fixed and I'll get this update released in the next day or so. As per the answer to question 7 below, the file names and display names are different. Also, datamine won't change any of its configuration when you change the device configuration (ie if you edit the device name after starting dataMine logging). DataMine doesn't use the names of devices to operate internally - it uses the device numbers which is why when you change the name it still works, but the filename is still wrong.

Quote
2) Under the Advanced tab - LastUpdate is set to zero, should that  change with each write for a recording?
No idea - this is an internal Vera variable I think - it's not a dataMine specific/controlled variable.

Quote
3) Under the Config tab - when selecting a variable there is no clock icon visible/available as far as I can see, only an empty square?
Ah - thanks. Yes, this is probably a hangover from the UI4 version where the dataMine GUI was integrated in UI4. Now this reference is broken. My ultimate plan is to remove this from the plugin and have the configuration in the dataMine GUI, but at the moment I'm fighting with bugs in the java lib I'm using...

Quote
4) there are a lot of .backup files (17 to be exact ) in the /dataMine/ directory, is that to be expected?
UPDATE - Ok, I see now, every change in selection of variables that i want to be recorded or not etc creates a back up of the dataMinejson file
Yes - you can safely delete the old ones. I will add a clean funtion to delete old backups,

Quote
5) There is no device icon appearing for the dataMine device in the UI, is that to be expected too?
UPDATE - Looking in the JSON file that was installed from the app store, I can see the following icon listed, "flashicon":"icons\/Data_Mine.swf"., yet nothing appears
This should be taken care of by the Vera plugin manager. I'll double check this...

Quote
6) Is there a way to delete a graph for a variable I selected by mistake?
No - not at the moment, but this is planned. Note that since you seem to be playing in the dataMine.json file, you can delete stuff in there. A good pointer - if you do edit the json file, I would recommend a json formatter so that you can read it better. I personally use jsonlint.com.

Quote
7) What happens if I need to rename a device, will it lose it's connection with its associated  log file in the /dataMine/ directory ?
UPDATE - As mentioned in question 1 above, it seems even if I try to change the name in the dataminjson file it is not reflected..
The config file (dataMine.json) contains separate references for 1) log file names, 2) the name displayed in the GUI, and 3) the actual link to the device. You can change the display name without impacting on the operation of dataMine.

Cheers
Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #454 on: September 19, 2012, 10:37:41 am »
Thanks so much Chris, if you get chance I had added a question 8. which is a wired one?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #455 on: September 19, 2012, 10:53:55 am »
Thanks so much Chris, if you get chance I had added a question 8. which is a wired one?

8.) I have two temperature logging files, each with the same start date/time, yet depending on the order I select them to be 'graphed' the info is presented in two very different ways. Why?
Can you provide an example of what you mean? Maybe upload a couple of screenshots? The data is passed into a graphing library, so it may be something to do with the way that interprets the data, but a couple of screenshots would help to explain the problem.

Cheers
Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #456 on: September 19, 2012, 11:08:05 am »
Can you provide an example of what you mean? Maybe upload a couple of screenshots? The data is passed into a graphing library, so it may be something to do with the way that interprets the data, but a couple of screenshots would help to explain the problem.

Sure here you go.

Both have the same date/time entry for the first reading, yet the graph renders them differently and places them far apart.

Additional questions if you have time?

a) - If I want to transfer all my current cost energy usage data, that's stored in an RRD, do I do that in one hit, or did i read i needs to create different files ?
b) - what does the [R2228] represent at the end.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #457 on: September 19, 2012, 11:46:16 am »
Hmmm - interesting. I'll need to get back to you on what I think is happening with the graphs! It's certainly not immediately obvious....

Quote
a) - If I want to transfer all my current cost energy usage data, that's stored in an RRD, do I do that in one hit, or did i read i needs to create different files ?
Yes, you need different files. The files are broken into weekly files just to avoid a single massive file and having to scan through to access the data.

Quote
b) - what does the [R2228] represent at the end.
The R means Raw, and the 2228 is the week number. In theory, R can be something else (the code isn't hooked in, but you can generate hourly and daily files as well - the idea being to speed up the generation of long timespan graphs). If you're familiar with unix time, then the 2228 comes from the current unix time divided by 604800 (the number of seconds in a week).
So, if you're familiar with messing with RRD, then I suspect you won't find it too difficult to convert the data if you wanted...  Note that RRD samples/logs data periodically where dataMine just logs when there are changes, so you will see some differences in the graphs.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #458 on: September 19, 2012, 12:04:38 pm »
So, I put your data into my system, made a couple of bogus entries in the config file, and it seems to graph just fine... Are you familiar with FireBug for logging data to/from the browser? If so, it would be really interesting to see what the request and response is...

I also don't see a difference depending on which way around I select the variables - it's all a bit strange. I'm running a different version here, but I don't think that should impact on this issue...

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #459 on: September 19, 2012, 01:14:28 pm »
An updated version of the plugin should be hitting your Vera in the next day or two. This is only a change to the plugin, not the GUI.  It has the following changes -:
  • Fixes an issue where data is not logged if there are reserved characters in the device names (ie characters that are not permitted in filenames)
  • Fixes an issue if the SetDataDirectory was not terminated with a /. The plugin now works with our without the / on the end of the directory name.
  • Adds functionality to support a Veras USB logging - ie it can disable mounting the drive and use a directory under the Vera log directory. I'll post more on this once the plugin has been released by MCV.

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #460 on: September 19, 2012, 01:18:43 pm »
Thanks Chris

Were they 'graphing' OK before you added the bogus entries?

Are you familiar with FireBug for logging data to/from the browser? If so, it would be really interesting to see what the request and response is...

Sorry no I'm not, is it straight forward to use?

What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #461 on: September 19, 2012, 01:29:38 pm »
Were they 'graphing' OK before you added the bogus entries?
I needed the bogus config entries for it to show up in the channels list, otherwise I can't graph them...

Sorry no I'm not, is it straight forward to use?
If you are using FireFox, then it's reasonably easy for what I need if you're vaguelly tech savvy. Once it's installed, it adds a button to FireFox and there's a console that can show all the traffic between the browser and the server. It would be possible to capture the request and response with this. IE may have a similar feature as you can hit F12 to enter developper tools area, but I don't tend to use it...

Cheers
Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #462 on: September 19, 2012, 01:52:36 pm »
What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?

Any thoughts on the above?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #463 on: September 19, 2012, 02:12:12 pm »
What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?

Any thoughts on the above?
Nope - unfortunately not (although I can agree with you - it's strange! ;))
It would be good to get a cope of what the server is producing. The other thing I did think of earlier is if there's something in your config file that's a bit wierd, but I really can't think of anything that could cause this.

One more thing you could look at. Graph the data both ways, then do a grep on the log file - from WinSCP, go to the terminal and type "grep dataMine /tmp/log/cmh/LuaUPnP.log". This does log data about requests coming in - I don't think it will tell me much, but you never know (and at the moment I just don't have a clue).

Chris

Offline parkerc

  • Beta Testers
  • Sr. Hero Member
  • *****
  • Posts: 2479
  • Karma: +35/-48
  • Life Moves Pretty Fast....
Re: New dataMine graphing and logging plugin
« Reply #464 on: September 19, 2012, 02:32:11 pm »
Cheers Chris.

Another strange observation with the graphs for you..

If I select a variable that has been logging the longest (Ben's Temp) first and then add another one that has not been logging as long (Jane's temp) it plots the new/additional line graph starting further down. (seems logical)

Ok now clearing the graph and I did this..

If I select a variable that was not logging the longest (Jane's Temp) and then add another one that's been logging longer, it a) does not extend the graph back to factor in the longer period, and b)  it actually plots the older one further down?

Weird?
« Last Edit: September 20, 2012, 05:30:52 pm by parkerc »