We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287580 times)

Offline teonebello

  • Sr. Member
  • ****
  • Posts: 405
  • Karma: +0/-0
  • Macs, Home Automation, Bikers
Re: New dataMine graphing and logging plugin
« Reply #225 on: March 20, 2012, 03:05:35 pm »
That's great!  :D

Offline chixxi

  • Hero Member
  • *****
  • Posts: 1036
  • Karma: +37/-14
Re: New dataMine graphing and logging plugin
« Reply #226 on: March 21, 2012, 05:59:23 am »
Glad to hear you are working on it chris! And I am glad to hear that the problems you have are apps.mios.com related, out of that I kind of interpret that you solved all the other bugs.  ;D

I am not so glad about hearing that mcv has not fixed a problem which they were aware of for at least a few month:
http://forum.micasaverde.com/index.php/topic,8974.msg58862.html#msg58862
« Last Edit: March 21, 2012, 06:10:23 am by chixxi »
Developer of Plugins: Virtual Switch, Variable Container, Popcorn Hour Remote, Vacation Ghost. => PLUGINS HAVE BEEN UNPUBLISHED BY ME.

Offline mcvflorin

  • Administrator
  • Hero Member
  • *****
  • Posts: 1755
  • Karma: +11/-3
Re: New dataMine graphing and logging plugin
« Reply #227 on: March 21, 2012, 07:50:24 am »
I am not so glad about hearing that mcv has not fixed a problem which they were aware of for at least a few month:
http://forum.micasaverde.com/index.php/topic,8974.msg58862.html#msg58862

This has just been fixed. (Mantis #2143)

Offline chixxi

  • Hero Member
  • *****
  • Posts: 1036
  • Karma: +37/-14
Re: New dataMine graphing and logging plugin
« Reply #228 on: March 21, 2012, 08:08:20 am »
wow, now you just really impressed me! thank you!

I tested that with a custom icon (png) and the path "/../../www/cmh/skins/default/icons/", it is working!
Developer of Plugins: Virtual Switch, Variable Container, Popcorn Hour Remote, Vacation Ghost. => PLUGINS HAVE BEEN UNPUBLISHED BY ME.

Offline teonebello

  • Sr. Member
  • ****
  • Posts: 405
  • Karma: +0/-0
  • Macs, Home Automation, Bikers
Re: New dataMine graphing and logging plugin
« Reply #229 on: March 21, 2012, 08:13:45 am »
that means that we should be able to install datamine from app.mios.com soon?

What about the storage in the USB?

Thanks all

Offline chixxi

  • Hero Member
  • *****
  • Posts: 1036
  • Karma: +37/-14
Re: New dataMine graphing and logging plugin
« Reply #230 on: March 21, 2012, 08:33:14 am »
What about the storage in the USB?

I am wondering about that too. I guess all the plugin files will be loaded somewhere on the vera. @chris, would it be possible that you allow specifying the path for the log storage in the plugin?

Excited to see dataMine running soon again. Thanks again for that plugin, was missing it since day one of UI5 release...
Developer of Plugins: Virtual Switch, Variable Container, Popcorn Hour Remote, Vacation Ghost. => PLUGINS HAVE BEEN UNPUBLISHED BY ME.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #231 on: March 21, 2012, 02:55:43 pm »
Quote
And I am glad to hear that the problems you have are apps.mios.com related, out of that I kind of interpret that you solved all the other bugs.
I think I posted a while ago, but the big issue is incompatability with java libs used in Vera. So, to avoid this issue, the plugin provides a separate page for the graphing, rather than embedding the graph into a device window.

Quote
would it be possible that you allow specifying the path for the log storage in the plugin?
Yes - this is already supported. It defaults to /dataMine, but it's a variable, so you can point it elsewhere.

Anyway, if everything is fixed on apps, then I should be able to get something up quite quickly - it's working on my system here so it's just a case of getting everything loaded onto apps, and configured...

Cheers
Chris

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #232 on: March 21, 2012, 08:11:12 pm »
An impressive plugin for sure -- enough to keep me holding on to UI4 until looks like UI5 can support it.  Was still wondering whether anyone had a response to query:

Quote
What file holds the information that provides the list of variables that can be charted?  As with at least one other poster, be useful to be able to clean-up 'mistakes' from this dropdown; as things are now, it would seem that once you've clicked green, they are permanently latched as part of the drop down selections for charting.  Happy if I do this manually via SSH file editing.

As well (and connected to this), as I organize data logging nodes, some renaming, and a it gets a bit disorganized over time as the csv file names and device names fall out of alignment.

Offline slushpupie

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #233 on: March 22, 2012, 01:04:44 am »
Have you considered adding the ability to upload the data periodically to a service like pachube.com or a google docs spreadsheet? They have some cool advanced data visualization features and lets you share your graphs with others (if you want).

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #234 on: March 22, 2012, 04:49:55 am »
Quote
What file holds the information that provides the list of variables that can be charted?
There isn't one. There's a file that holds the names of the variables that ARE BEING charted, but the list of variables that can be charted is read directly from the devices. The only way to manipulate this is through the config screen in dataMine.

Regarding the CSV filenames- I agree - it's a bit of a pain when / if you change a device name, things do get a bit messed up. In theory though, you shouldn't need to go and play with the CSV files. I keep meaning to add an export option to dataMine to allow the data to be exported as a CSV file, but I've just not done this yet.

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #235 on: March 22, 2012, 09:05:39 am »
Chris, thanks for response.

Sorry, my bad phrasing.  I had meant *are being* charted, and automatically linked to that, logged to csv files.  Perhaps risky to edit file you referenced for that purpose, but was curious whether this renaming could be done 'manually'.  I certainly appreciate there would be potential continuity issue with current data depending on whether matching the file names would cause dataMine to continue to log to pre-existing file or whether it would set-up a new one.

As to csvs, I need to do quite a bit of statistical manipulation and re-processing, e.g. of binary sensor information triggering in different parts of the house.  I use dataMine for the logging, and 'tactical' visualization to ensure devices working properly.  Happy so far in batch off-loading csv through SSH (got half used to that through install : -), but would welcome a more user-friendly option for this as you and others have suggested.

JohnR

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #236 on: March 22, 2012, 09:16:11 am »
Ok - yes, you can edit the config file to rename the CSVs. It is all listed in the dataMineConfig.json file - I would recommend editing it and running it back through jsonlint.com to ensure there are no syntax errors with the json (it also formats it in a more readable way than Vera does!). If you edit the filenames in this file, and also edit the files on Vera, then everything will work fine (you'll need to restart Luup to make dataMine re-read the config.

I think I will add CSV export at some stage (soon probably), but for what you're doing, it may still be better to stick with SSH. My real goal would be to have a descent analytical plugin, but in reality I think that Vera doesn't have the processing power. That said, Vera3 responds to dataMine requests a lot faster than Vera2, so who knows...

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #237 on: May 07, 2012, 04:47:21 pm »
I'm working on installing this today before I leave for the week... hoping someone can answer one question.

I'm at this step...

Quote
Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

    mkdir /dataMine
    mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

... when I try to mount the device to the dataMine folder I get this message...

Code: [Select]
root@MiOS:~# mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

mount: mounting /dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine failed: No such file or directory

... the folder dataMine does exist in the root.

I have added a 512MB USB stick and told Vera to log to it. Here's the message I get on the Logs tab...

Quote
Enabled
Mounted: read-write
Usage: 2%
Errors log:
2012-05-07_13:40:25 (/dev/sda1) : Successfully checked
2012-05-07_13:40:25 CHECKDISK: Starting to check device: /dev/sda1 with: /usr/sbin/e2fsck

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #238 on: May 07, 2012, 05:24:29 pm »
*UPDATE*

I want to check my work here - I think I figured it out.

Since the Vera mounted the USB drive as /dev/sda1 , I mounted /dataMine to /dev/sda1 also.
... is there any problem with this?

And, I suspect that I should need to do the following from the TRAC instructions, just using sda1 (?)...

Quote
Modify (or generate if it doesn't exist) the /etc/init.d.custom-user-startup file to add the line -:

    mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

Run the following commands -:

    chmod +x /etc/init.d/custom-user-startup
    /etc/init.d/custom-user-startup enable

This ensures the USB stick is remapped when Vera restarts. The second command makes the file executable, and the last command adds the symbolic link to start it on startup.

thx for the help!

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #239 on: May 07, 2012, 07:47:27 pm »
OK, the USB drive is mounted to /dataMine -- I can get to it via WinSCP.

BUT when I click the Tool button on the dataMine device, in UI5, I don't options that I think I should -- see screenshot attached.