We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287704 times)

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #255 on: June 02, 2012, 08:32:23 pm »
WOW... looks great. This is some seriously great stuff for the community. I'm really looking forward to graphing my AC usage with outside temps, and my set points. I'm in Arizona... we use a lot of AC.

Offline chris66

  • Sr. Member
  • ****
  • Posts: 318
  • Karma: +9/-8
Re: New dataMine graphing and logging plugin
« Reply #256 on: June 06, 2012, 08:31:59 am »
Great  plugin! Cannot wait to install it on UI5 ::)

Offline teonebello

  • Sr. Member
  • ****
  • Posts: 405
  • Karma: +0/-0
  • Macs, Home Automation, Bikers
Re: New dataMine graphing and logging plugin
« Reply #257 on: June 06, 2012, 08:32:56 am »
Great  plugin! Cannot wait to install it on UI5 ::)

+1

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #258 on: June 06, 2012, 08:50:56 am »
It's coming  :)

I've had a few issues with getting the install to work with apps.mios.com. The problem is that the UI is a reasonable size file (~450k), and I have a slow internet connection  :-[. This uncovered some issues with timeouts when installing apps and meant that I couldn't install dataMine from apps.mios.com. It also revealed another "problem", or hidden feature that I need to fix to stop the app installer getting into a pathalogical loop.

Anyway, @MCVFlorin has been very helpful with solving this and the problem should be fixed in the next UI5 software (this week or next!). In the meantime, once I've resolved this second issue (tonight I hope) I might release a version that will need you to use WinSCP (or similar) to put the GUI file manually onto the Vera, and once the new firmware is available I can integrate this fully into apps.mios.com.

Watch this space...

Chris

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #259 on: June 06, 2012, 09:16:27 am »
Thanks for the update, will this work with UI4 as well??

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #260 on: June 06, 2012, 09:33:40 am »
Thanks for the update, will this work with UI4 as well??

Unfortunately not. The UI would work without changes, but I've added some extra functions to the Lua file to support the new UI, so unfortunately it won't work as a system. Once I get a UI5 version released, if there's sufficient interest I can probably port the Lua back to UI4 (there aren't many changes so it shouldn't take long).

My personal feeling is that the separate GUI is a better approach as it provides more flexibility, but others may have different ideas and prefer the closer integration as in UI4 (??)...

Chris

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #261 on: June 06, 2012, 09:44:39 am »
I'll say "yes please" to a UI4 version then, everything else is working there and I have no desire to risk breaking a working heating control system by "updating" to UI5.

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #262 on: June 07, 2012, 05:39:21 pm »
I've now put a version of dataMine onto apps.mios.com for approval. Assuming MCV are happy with this, it should be in the marketplace in the next day or so...

Installation is a bit more difficult than it needs to be at the moment due to an issue with the current version of Vera software. This is meant to be fixed in the release due out any day now. In the meantime, one file needs to loaded manually, and I'll attach that here once the app is approved.

The picture attached shows the GUI. On the left is the list of channels that dataMine is recording - configuration of channels is the same as UI4 and is done through the plugin properties in UI5. Next to the channel is an icon - either a green tick, yellow warning, or red exclamation. This indicates the state of the channel - green means it's been updated in the last hour, yellow in the last day, and red is older than 24 hours. One warning about this - at the moment, when Lua is restarted, this gets reset and everything goes red (I'll fix this later).

To graph a channel, click on the channel name. This will highlight the channel, and also show an axis icon to allow you to select the primary or secondary axis. Then click the "Update Graph" button. The other button "Clear Channels" clears the selection of channels (it doesn't update the graph though!).

Along the top of the graph is a toolbar - this has buttons to select zoom in/out, display day/week/month, shift left/right, and display a real-time graph. The real-time graph is definately work in progress - it needs some work as the graph looks a bit screwy as it only updates when the data updates, and with the splined graphs, it just looks wrong! By default, pressing the real-time button will display the last hour of data, but you can change that with the buttons.

There's also a set of graph options - this should be pretty self explanatory (press the buttons if you're not sure!).

I think this should provide most of the functionality of the UI4 version, plus some. However, status channels aren't handled quite right at the moment (it's on my todo list and should be easy to add).

I do have some ideas for improvements, but let's see how we get on with this first  :)

Chris

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #263 on: June 07, 2012, 08:17:56 pm »
Awesome work Chris!

- Garrett

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #264 on: June 08, 2012, 12:48:21 pm »
The UI5 version of dataMine appears to have been approved in the marketplace, so here's some installation instructions. Due to the amount of data that dataMine generates, it really needs to use a USB stick, and therefore it's a bit more complicated to install than some plugins (sorry!). I should admit that I haven't tried this yet on a Vera3, but I'm assuming the same instructions I wrote for Vera2 are still applicable. I think the USB stick, and the json library should be installed before the plugin is installed from the marketplace (it's not manditory, but it probably makes sense).

Use WinSCP (or another SSH client) to upload the files to Vera...

The plugin needs the json library. Unfortunately, I can't include this into the app installation as apps.mios.com doesn't support shared files. To install this, go to -:

and download this to the following directory on Vera
  • /usr/lib/lua

Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.
  • mkdir /dataMine
  • mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

Modify (or generate it if it doesn't exist) the "/etc/init.d.custom-user-startup" file to add the line -:
  • mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

Run the following commands -:
  • chmod +x /etc/init.d/custom-user-startup
  • /etc/init.d/custom-user-startup enable

This ensures the USB stick is remapped when Vera restarts. It makes the file executable, and adds the symbolic link to start it on startup.

Now install the app through the marketplace (http://apps.mios.com/plugin.php?id=1088). This installs all the standard Lua device files, and should install the GUI, but due to a timeout issue with Vera (to be fixed soon) you need to load the attached "dataMineWeb.tar.gz" manually into the "/www/dm" directory using WinSCP (or similar). PLEASE NOTE: The attached file needs to be renamed - please REMOVE the .zip off the end so the filename is called "dataMineWeb.tar.gz" (the forum won't allow me to upload a gz file!). This must be done after the app has been installed - the file will already exist, and you should overwrite what's there.  Then click on the "Restart" button on UI5 - when dataMine starts, you should see a notification stating that dataMine is installing the GUI - this will take around 1 minute (at least it did on my Vera2 - it will probably take less time on a Vera3).

Once the dataMine device is loaded, click on the tool icon. You will get the main tab which has a link to the UI, and config tab. Click on the config tab, and select the variables you want to log. All changes to the selected variables should now be logged to files in the /dataMine directory.

Once the next release of UI5 is out I'll update the app to remove all this messing around with the dataMineWeb file, then updates should be nice and simple! If some kind sole wants to write a shell script to automate the USB configuration, I'm sure that would be welcome (I use linux a lot, but am far from an expert in shell scripts).

I'm sure people will let me know if you have problems ;) and I'll try and fix them as soon as possible. I do have a number of improvements planned - it's just a matter of time (I'm also trying to build an extension on the house at the moment, so my better half doesn't like to see me "playing" on the computer!). I hope to update the graphing to add/improve the current features quite soon, but other changes may take a bit more time...

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #265 on: June 08, 2012, 01:33:31 pm »
Hi Chris,
Installing now.

Question,
for the "/etc/init.d.custom-user-startup" file you have...

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

last time I installed I put in...

mount /dev/sda1 /dataMine &

... this was the only way I could get it to work. Is there a specific reason I need to use your code?


Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #266 on: June 08, 2012, 02:34:43 pm »
No - I'm not an expert in this area and I think I probably found this in an OpenWRT forum. I'd say use what works for you - of course if you've already done this on your Vera previously, then you don't need to do it again.

I'd really like to get this part of the installation made a bit easier. It's not really my area of expertese, but I think it should be possible to write a shell script to automate this (any volunteers?  :) ).

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #267 on: June 08, 2012, 07:25:45 pm »
Chris,
Under the Config, all the items are showing properly and they open when clicked... but how do you 'select' which ones you want to use? I don't see a checkbox, or highlight, etc?

thx

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #268 on: June 08, 2012, 08:08:41 pm »
Update...

After many clicks, some of the items are responding and highlighting, then providing a drop-down... but only the two I show in my screenshot... all others I tried do nothing.

Offline elordude

  • Sr. Newbie
  • *
  • Posts: 39
  • Karma: +1/-0
Re: New dataMine graphing and logging plugin
« Reply #269 on: June 08, 2012, 09:38:38 pm »
Chris,

What size USB stick do you recommend?