We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287573 times)

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #210 on: March 07, 2012, 10:43:31 pm »
Just as a quick postscript, the biggest areas of shakiness I had were the various instances in the instalation instrucitons where files and/or folder were referenced, which needed editing or modification, that I didn't have in the first place.  Being uncomfortable with this whole SSH / WinSCP environment and what was going on, I created these with some considerable lack of confidence.

Related to this was the inclusion of the instructions from member "chixxi" related to creating the 'symbolic link' to get the custom-user-startup file activated.  I presume this is now part of the official instruciton / recommendation, but I was unsure.

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #211 on: March 08, 2012, 05:02:22 pm »
First, praise where praise is due.  Chris, very impressive, and this needs to be repeated, very impressive, plugin -- can't thank you enough for contributing it to the cause.  The data files are just so damned clean -- really excellent work and smart thinking all around in the UI.  The effort I've spent flailing to get this to work will be recouped easily in streamlining our data acquisition effort from the Vera.  Can't thank you enough (ping me if you ever have an Excel annoyance that is bugging you or a financial calculation -- about all I know).  And for MCV, I bought the Vera, and stayed with UI4 based solely on reading this thread (and of course will be scared to mess with any upgrades).

As well, I have to thank quite a few folks that have posted here over last few months.  Not just the ones that have provided solutions, including Chris's own persistance, but those who have had problems that have toss out what they were.  I would never have gotten past / through the .lzo bug any other way.

Right, from post below, here are the open questions that I expect some who may benefit may benefit from:

  • I presume the instructions to this point presume USB flash memory stick plugged into USB1 (top) port.  What is the edit to make this work from USB2?
  • If someone wants to execute the recommendation from mcvflorin above, re: replacing the upload_upnp_file.sh with WinSCP SSH client, how does this work without 'wrecking' the file?  When I copy these .sh files from Vera to my local and back, the file size goes from about 38 bytes to 6,500 bytes, and the ability to download from MiOS developer console is destroyed.  Apologies in advance for being SSH clueless, but I don't think this is down to 2byte vs 1 byte on spaces with text file -- I tried about every file format / transfer option I could, and they all seemed to wreck file size, shift date, etc.
  • What file holds the information that provides the list of variables that can be charted?  As with at least one other poster, be useful to be able to clean-up 'mistakes' from this dropdown; as things are now, it would seem that once you've clicked green, they are permanently latched as part of the drop down selections for charting.  Happy if I do this manually via SSH file editing.

Right, apologize for essentially talking to myself over last 48 hours on this topic, but all good news (not without some pain, but I don't think as much as some who have blazed the trail in front of me).  Also, perhaps some incremental notations / improvements and 'lessons learned' for those who may follow me.

Having gotten jammed up last night I decided to 'clear back and simplify'.  Hence, I'm afraid I don't have a definitive position on where my problems were, but I have a strong suspicion, and it relates to something no one has yet considered on this post (and I've printed out and read through all 83 pp a couple of times).  What USB port are we presuming here for mounting?  I expect one of those 0s in the command string means USB1, and setting a 1 in there somewhere -- perhaps someone could specify -- would mount the dataMine directory to USB2?

About halfway through today's misery (more below that may help some folks) I realized I have my Insteon PLM controller patched into USB1 (top port).  USB stick in USB2.  Hmmm.  As/if it can't find the /dataMine logging directory, it can't set-up dataMineConfig.json, and you simply get a 'stalled' config window on the dataMine tab.

So I started this afternoon's effort with a clear picture that I wasn't getting the config file into \dataMine.  So I:

1.  Deleted all the files that related to the effort of mounting to USB stick
2.  Replaced upload_upnp_file.sh (as recommended for those of us with 1.1.1350 builds by mcvflorin).  Decided best to save my previous upload file to my local in the event this failed to work.
3. Decided smart to 'drop back' to 28 Nov 2011 build, as again, less variability with problems.  (When I get brave again I'll revisit changes / improvements to determine if important to me.)
4.  Determined No. 2 broke my Vera's ability to use MiOS developers install area of console
5. Reloaded previous upload.sh file, and determined that the WinSCP process was blowing these files apart.  (Note Newbie status.)
6. Now I have nothing functioning.  Dug around and found link for 1.1.1245 build, which I had determined from yesterday's effort might be better bet than 1.1.1350.
7.  Cleared back to factory settings.

Followed my own script / Word document from yesterday, using 28 Nov 2011 build, and (essentially) had no problems at all.  In process of that I CAN confirm i) the .lzo bug does not affect anyone with 1.1.1245 build; and ii) my 16 GB flash USB seems to be logging data (near as I can tell).

I think that's it for me.  If there is interest in me cleaning up my Word document with better information and some additional ideas/thoughts I had from today's post, let me know.  I'll happily chip back in and then leave to Chris and/or others as to what of it might be useful to put into revised Wiki instalation instructions and/or ReadMe file.

All the best.  Off to now reinstate my Insteon motion sensors as triggers -- they worked pretty well with Vera btw.

JohnR
« Last Edit: March 08, 2012, 05:09:03 pm by JohnR »

Offline Da_JoJo

  • Hero Member
  • *****
  • Posts: 1380
  • Karma: +16/-78
  • If something aint work, we can allways try n make
Re: New dataMine graphing and logging plugin
« Reply #212 on: March 08, 2012, 06:46:31 pm »
nice story you got there :-)
might wanna try http://www.winscp.com the datamining works in ui5 , the chartsplotting and interface to it doesnt (probably the changes in ui5 interface )
for other usb port to use you would need to set it somewhere in the plugin to usb2   i.e. change the 1 into a 2 in the plugin files
« Last Edit: March 20, 2012, 04:37:37 pm by Da_JoJo »
Vera lite (1.5.622), 2x an-158/2, dead usb pl2302 rs-232, 2x greenwave 6 port, 4x Fibaro FGD211 v1.6, FGBS001, few FGS - 221, etc. AuthomationHD 3 for android :-)
Dutch & German translator http://wiki.micasaverde.com/index.php/Special:AllPages http://support.micasaverde.com http://domotica-shop.nl

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #213 on: March 08, 2012, 10:07:04 pm »
Yeah, I'm sure only the desperate (like me) may gain some value from it.  Thanks for link -- have bookmarked it and will see how things progress.  Couldn't be happier (so far) with dataMine, however.  The logging files and organizaiton are just so clean.  I've still got original iPad, so I'll be fine with UI4 for a while I expect -- need to amortize the cost of my pain over some usage time.

As to USB, given the port syntax noted regarding the (no one can get to work) 'log to USB' option on the console, I expect 0 = USB1, and 1 = USB2.  There's a MOUNT command executed in various ways in the install for dataMine, and I expect a piece of that establishes which USB port.  If no one has quick answer on this, I'll do some digging around.

Offline Da_JoJo

  • Hero Member
  • *****
  • Posts: 1380
  • Karma: +16/-78
  • If something aint work, we can allways try n make
Re: New dataMine graphing and logging plugin
« Reply #214 on: March 09, 2012, 08:03:27 am »
yu thats probably it.. change this in the datamine installer and the datamine plugin files itself and it will use the other usb port
you can also try n list all ports with some command i forgot (been a 4 years since i touched linux-a-like stuff. do rememer that files sizes and stuf not allways correct with winscp so i used putty and totalcommander which did the job a lot better (not faultless i may add, this is due the nature of the protocol used)
Vera lite (1.5.622), 2x an-158/2, dead usb pl2302 rs-232, 2x greenwave 6 port, 4x Fibaro FGD211 v1.6, FGBS001, few FGS - 221, etc. AuthomationHD 3 for android :-)
Dutch & German translator http://wiki.micasaverde.com/index.php/Special:AllPages http://support.micasaverde.com http://domotica-shop.nl

Offline netsidedeveloper

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #215 on: March 17, 2012, 04:50:13 am »
What a great plugin.  Is it possible to use the plug in to chart activity or inactivity of a motion sensor?  I want to setup a system in my Gran's house so that we can see she is moving around....Thanks.

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #216 on: March 17, 2012, 05:59:32 am »
If you want to know if there is inactivity of the motion sensor and to be notified, you can have a look at the home care plugin. 

- Garrett

Offline netsidedeveloper

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #217 on: March 17, 2012, 07:49:02 am »
Thanks for the reply Garrett.  I'd like to be able to see the activity on a webpage aswell, is this possible?

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #218 on: March 17, 2012, 08:13:33 am »
That is where dataMine would come in as from what I gather from the homecare plugin, it just notifies the user of inactivity of the motion sensor.

- Garrett

Offline JohnR

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #219 on: March 20, 2012, 09:58:21 am »
netsidedeveloper,

Keep me posted re: where you get to with motion sensors and Vera2 (might be useful to compare notes, as there are some challenges among the options).  I'll put up a post under 'motion sensors' area.  I'm also setting up some monitoring in a home residence.

JohnR

Offline andygodber

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #220 on: March 20, 2012, 10:50:07 am »
Chris/others - am I right in reading that the graphing function is still not working on UI5?

Offline teonebello

  • Sr. Member
  • ****
  • Posts: 405
  • Karma: +0/-0
  • Macs, Home Automation, Bikers
Re: New dataMine graphing and logging plugin
« Reply #221 on: March 20, 2012, 12:46:38 pm »
Why not release this wonderful plug in the app.mios.com?

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: New dataMine graphing and logging plugin
« Reply #222 on: March 20, 2012, 12:57:20 pm »
He's currently chasing down bugs in the MCV stack that make it trickier than it should be.  See this thread for details...
    http://forum.micasaverde.com/index.php/topic,9976.0.html

Why not release this wonderful plug in the app.mios.com?


Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #223 on: March 20, 2012, 12:58:28 pm »
Because the way the plugin is build, it requires a few things that is currently broken in the app.mios.com section. This is currently being worked on by MCV. Once those issues are worked out, I am sure Chris will get it in there.

- Garrett

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #224 on: March 20, 2012, 03:02:14 pm »
Because the way the plugin is build, it requires a few things that is currently broken in the app.mios.com section. This is currently being worked on by MCV. Once those issues are worked out, I am sure Chris will get it in there.

100% correct. I've been waiting for the UI5 update since it was meant to fix a problem with plugins that use files that can't be compressed (as dataMine does). It seems that only half the problem was in Vera - the rest is in apps.mios.com, and that also needs fixing! MCV have said they'll try and do that "this week" - once this is done, I'll get the plugin released.

Cheers
Chris