We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287703 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #555 on: February 10, 2013, 04:31:34 am »
Defaults for USB or non-USB should be set up in the GUI without having to go into the command-line and do all sorts of setup.
This is a good idea. Following some discussion with @hek regarding setup, I was also starting to think along these lines. As @Aaron said, Vera doesn't offer much in the way of setup support, but I can probably put something into the GUI...

Cheers
Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #556 on: February 10, 2013, 04:40:27 am »
I want to use a USB stick in the upper slot and also install another USB stick in the lower slot for Vera USB logging.

What is is method to do this so that I am assured that datamine logs to the upper slot and Vera logging only logs to the lower slot? I am worried that the mounts will get screwed up with each other.
A very valid concern. @hek recently pointed me to a feature of USB where a unique ID is put on the USB stick. DataMine can now find the mount point based on this "UUID", and from further reading on the topic, it seems this is (nearly!) the only 100% guaranteed way to ensure mounting works properly under Linux (ie the mountpoint is the same each time).

As suggested by @gengen, I will look at providing something in the GUI to ease setup, however if you are familiar with the WinSCP program, then you can find the UUID by running the "blkid" program.  You just need to find out which ID is relevant to which USB stick, and put this into the configuration SetMountUUID. I highly recommend this configuration method...

I don't know how Vera mounts the USB stick for the logging, so I'm afraid I can't be sure that it won't get confused. If it also uses UUID, then I believe there won't be a problem.

Do you have an updated install docs somewhere I can use?
I try and keep the docs updated - they can be found here -:
http://forum.micasaverde.com/index.php/topic,13057.0.html

Cheers
Chris

Offline bucko

  • Hero Member
  • *****
  • Posts: 659
  • Karma: +27/-9
Re: New dataMine graphing and logging plugin
« Reply #557 on: February 10, 2013, 09:02:45 pm »
@CHris

Thank you for the quick response. You are really doing a good job working through the various issues and ideas with your plugin development. Again, great job here.

I came to the conclusion that Vera logging to USB seems to work if it is the only USB thing running. If I reboot my Vera 3 with USB logging, it shows CHKdisk errors, then tries to reformat the USB stick, fails again, etc. This is while ONLY using logging and Datamine uninstalled!

Once I try to add a datamine stick and set it up, USB goes bonkers, and once it fails to read it's own USB stick, it simply takes over the datamine stick and reformats it for logging. Then of course datamine fails. I am not a programmer, but it looks like Vera logging has some issues and should be looked into by MCV.

That said, I stopped USB logging, rebooted my Vera, re installed dataMine, and put in a 250 mb USB stick in the Vera UPPER slot. I ran blkid and got this ..../root$ blkid
/dev/sda: UUID="60BA-4C9C"

Also in my Vera log it reads.....
50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>
50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>

Isn't is reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Can you help to explain this please?
Vera 3, 40+ Fibaro relays & dimmers, IPUX cams, Elk M1 Gold, M1XEP, M1SP, XVOR, M1RB, Elk124, Monoprice HDMI matrix & 6 zone amp, Denon AVR, Panasonic A8000, BI DV,On Controls

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #558 on: February 11, 2013, 07:07:46 am »
Hi @bucko,
Firstly, thanks for your thanks  :)

A bit of background first - I suspect you probably know most/all of this, but it may be useful for others...

I recently became aware of a feature of USB where the device has what's called a UUID - Universally Unique ID. I'm not completely sure how unique it is since it seems to change when you reformat the drive, but it will still identify a drive without ambiguity...

From what I've read, this is almost the only way to guarantee that a drive gets mounted correctly. My original thought when I started DM was that different USB sockets in Vera have different mount points, and to some degree this seemed to hold true, but not completely. Certainly, if you have a hub connected, each time you power cycle Vera, the USB stick will end up mounting to a different point - seemingly randomly (at least with my limited understanding!).

So, I've implemented the mount system that uses the UUID (which you appear to be using), and in the next version this will become the default system for dataMine mounting USB sticks. The way it works is that dataMine runs a program called "blkid" which gives you the UUID, and the associated mount point. From this, I can find the correct mount, and mount the drive...

I don't know how Vera mounts the drive for USB logging - I might try and look through the files to see if I can work it out. However, with what I now know (as above) the mount point can change when you have multiple USB sticks, so I'm not completely surprised by what you've seen (I may have had a different view a couple of weeks back!).

My hope is that by using the UUID feature within dataMine, that at least dataMine should mount correctly every time. However, unless Vera is also using this, I'm not sure if we can guarantee that Vera won't go and use the dataMine USB stick for logging - effectively mounting it twice, and we know "bad stuff" happens if you do this.

So, on to your question (sorry it's taken a while to get there ;)).
The mount point doesn't link to a physical USB port as I previously thought - it can change each time Vera starts...

Have the problems you've seen been happening since you have been using the UUID feature, or was the corrupted disk (etc) before changing to this?

I'll see if I can find out how Vera mounts the log USB drive - this may help...

I hope that helps at least a little - I understand this isn't as simple as one would otherwise hope!

Cheers
Chris

Offline svaleb

  • Hero Member
  • *****
  • Posts: 508
  • Karma: +11/-7
Re: New dataMine graphing and logging plugin
« Reply #559 on: February 11, 2013, 07:20:55 am »
Hi Chris

My vera formatted the usbstick for vera loggging to 512  it vas a 2gb stick and I cant reformat it to 2gb.
I dont know if there has been something wrong with it from the start.
But if vera formats a 2gb stick for its own use to 500thomthing I will be very careful to put in another stick untill I am sure that it wont happen agin.
Others who have had this eksperience ???
I wil wait ontil you have solved the problem, you are doing a great job, THANK YOU.
Regards
           CE

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #560 on: February 11, 2013, 11:29:05 am »
My vera formatted the usbstick for vera loggging to 512  it vas a 2gb stick and I cant reformat it to 2gb.
I dont know if there has been something wrong with it from the start.
But if vera formats a 2gb stick for its own use to 500thomthing I will be very careful to put in another stick untill I am sure that it wont happen agin.
Others who have had this eksperience ???
I wil wait ontil you have solved the problem, you are doing a great job, THANK YOU.
Hi @svaleb,
I think you are getting confused - this is nothing to do with dataMine - the Vera logging is an internal Vera function and not something that I am doing. The wiki information here http://wiki.micasaverde.com/index.php/Store_logs_USB does indeed say that it formats up for 512M, but I would have thought that you could still reformat it....

For dataMine, you do not need to use the Vera USB logging. dataMine uses a pre-formatted USB drive, so you don't need to worry about this problem.

Cheers
Chris

Offline svaleb

  • Hero Member
  • *****
  • Posts: 508
  • Karma: +11/-7
Re: New dataMine graphing and logging plugin
« Reply #561 on: February 11, 2013, 01:30:54 pm »
Hi Chris

No, I am awere that it is not DM.
But if I put in the other stick vera might bee "confused" and format that one too.
And it was not a 2gb stick but an 8gb stick, and I cant reformat it so...bad luck.
I have seen DM work and it is great, now it is how to store it.
Thanks for your answer looking forward to a solution.
Regards
         CE

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #562 on: February 11, 2013, 01:41:18 pm »
You should be able to recover the full size of the usb drive. Either in linux or in windows. In windows you'll need to go into the disk management console. For linux, you can use fdisk.

- Garrett

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #563 on: February 11, 2013, 01:45:45 pm »
Hi Chris

No, I am awere that it is not DM.
But if I put in the other stick vera might bee "confused" and format that one too.
And it was not a 2gb stick but an 8gb stick, and I cant reformat it so...bad luck.
I have seen DM work and it is great, now it is how to store it.
Thanks for your answer looking forward to a solution.
Ok - I don't think Vera will reformat it - not unless you go into the USB logging section in UI5 and ask it to reinstall USB logging.  You should be fine to put in another stick.

Take a look at the other post I've just put in the dataMine folder, which presents my best guess as to what's happening in Vera. I think if you avoid using Vera to format the USB stick that dataMine uses, it should work ok.

The next version of DM has an easier install in the GUI that allows selection of the USB UUID and should ease configuration...

Cheers
Chris

Offline svaleb

  • Hero Member
  • *****
  • Posts: 508
  • Karma: +11/-7
Re: New dataMine graphing and logging plugin
« Reply #564 on: February 11, 2013, 02:06:56 pm »
Thanks Garretwp and Chris
Yes, I let vera format the first stick and I tryed to reformat it in windowes, but could not do it. Vera has it back in for logging now.
The other 8gb has been formatted in windowes and wil be redy for the next version of DM.
Thanks again both of you !
Regards
       CE

Offline bucko

  • Hero Member
  • *****
  • Posts: 659
  • Karma: +27/-9
Re: New dataMine graphing and logging plugin
« Reply #565 on: February 11, 2013, 08:46:00 pm »
@Chris,

On my last post I mentioned I had this report in WinScp with the blkid command.


50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>
50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>


I went to the dataMine folder by wINsCP'ing into my Vera and noticed that the data is writing to this folder and not to my USB stick in the upper Vera USB slot. DM advance tab is set with my UUID of my stick ok.  "mount point = 0" and "manual mount point = (blank) and "mount location" = /dev/sda

Isn't this reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Why is mine reporting /dev/sda while my USB is physically plugged into the upper slot? Should I go ahead and change DM mount location to /dev/sda1 even though the stick is in the upper slot? I'm a bit confused by this.

Right now I turned off Vera USB logging and only have  my DM stick on my Vera (upper slot) because Vera logging was corrupting my DM stick. I think maybe it has something to do with the fact my Vera slots seem to be reversed.

On my Vera it appears my upper slot is /dev/sda and the lower slot is /dev/sdb according to blkid. I just verified this again by plugging in my stick to Vera lower and blkid reports it as /dev/sdb. BTW, there is no "1" shown at the end from the blkid command result.

What do you make of that?

Vera 3, 40+ Fibaro relays & dimmers, IPUX cams, Elk M1 Gold, M1XEP, M1SP, XVOR, M1RB, Elk124, Monoprice HDMI matrix & 6 zone amp, Denon AVR, Panasonic A8000, BI DV,On Controls

Offline svaleb

  • Hero Member
  • *****
  • Posts: 508
  • Karma: +11/-7
Re: New dataMine graphing and logging plugin
« Reply #566 on: February 12, 2013, 01:01:46 am »
Yes, I have found out the same as bucko.
CE

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #567 on: February 12, 2013, 04:52:21 am »
Quote
On my last post I mentioned I had this report in WinScp with the blkid command.50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>
This looks fine.  

Quote
I went to the dataMine folder by wINsCP'ing into my Vera and noticed that the data is writing to this folder and not to my USB stick in the upper Vera USB slot.
??? I don't understand this? dataMine DOES write into the /dataMine directory - this is correct. The dataMine directory IS the USB stick - that's how drive mounting under Linux works. The USB mounting maps itself into the file system - you don't get a separate drive letter or whatever like you do under Windows.  

Quote
DM advance tab is set with my UUID of my stick ok.  "mount point = 0" and "manual mount point = (blank) and "mount location" = /dev/sda
Again, this looks fine.  

Quote
Isn't this reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Why is mine reporting /dev/sda while my USB is physically plugged into the upper slot? Should I go ahead and change DM mount location to /dev/sda1 even though the stick is in the upper slot? I'm a bit confused by this.
No - as I've said elsewhere, there appears to be no way of mapping a physical USB slot to a mount point such as /dev/sda1 under Linux. Others have seen that the device mount points change - especially with USB logging. From the reading I've done on this, the only way to ensure you map correctly is to use either the UUID (which is preferred) or the LABEL. I've written another post on this  http://forum.micasaverde.com/index.php/topic,13634.0.html.  

Quote
Right now I turned off Vera USB logging and only have  my DM stick on my Vera (upper slot) because Vera logging was corrupting my DM stick. I think maybe it has something to do with the fact my Vera slots seem to be reversed. 
The slots aren't "reversed" - they just don't relate directly to the mount points. How did you format the drives used by dataMine and the USB logging? IF you used the Vera USB logging to format both of them, then I think this is likely to cause problems. Again, take a look at the post I wrote on USB logging http://forum.micasaverde.com/index.php/topic,13634.0.html

  
Quote
On my Vera it appears my upper slot is /dev/sda and the lower slot is /dev/sdb according to blkid. I just verified this again by plugging in my stick to Vera lower and blkid reports it as /dev/sdb. BTW, there is no "1" shown at the end from the blkid command result.
Again, dataMine doesn't use the /dev/sd** to mount drives - it uses the UUID which is the preferred method. Also, I don't think USB logging uses the /dev/sd** either - I believe it's using the LABEL definition that you may also see in the blkid command output.  

Quote
What do you make of that?
I hope the above explains what I believe is happening. I don't profess to be an expert in Linux by any means! If others have better knowledge, I'm willing to be updated (please!). However, from everything I've read, and from discussions with other users, I believe the above to be correct. 

Cheers
Chris

Offline bucko

  • Hero Member
  • *****
  • Posts: 659
  • Karma: +27/-9
Re: New dataMine graphing and logging plugin
« Reply #568 on: February 12, 2013, 09:08:21 pm »
@Chris

The slots aren't "reversed" - they just don't relate directly to the mount points. How did you format the drives used by dataMine and the USB logging?

I formatted the stick on my PC as a fat 32 . datamine advanced shows it as a vfat as well. Under the directory it reads /tmp/log/cmh/datamine

 dataMine is working ok. But I did WinScp into my vera and looked into the directory location (tmp). It shows new files written in there. Is this ok or should ALL files be written only to the USB stick? How can I tell if files are being written to the USB stick? Do I just unplug it and look at it with my PC?

WinScp blkid does show my stick there and it's UUID is the same I put into datamine advanced tab.

Thanks
Vera 3, 40+ Fibaro relays & dimmers, IPUX cams, Elk M1 Gold, M1XEP, M1SP, XVOR, M1RB, Elk124, Monoprice HDMI matrix & 6 zone amp, Denon AVR, Panasonic A8000, BI DV,On Controls

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #569 on: February 13, 2013, 04:25:46 am »
dataMine is working ok. But I did WinScp into my vera and looked into the directory location (tmp). It shows new files written in there. Is this ok or should ALL files be written only to the USB stick?
Where abouts in tmp? I see you have your dataMine directory mapped to /tmp/log/cmh/datamine, so all dataMine files should end up there. There is a small exception to this, and that is I write a temporary file into the /tmp directory, but it should be deleted before you see it...

How can I tell if files are being written to the USB stick? Do I just unplug it and look at it with my PC?
Yes, I think the only way to be 100% sure the files are going onto the USB stick is to plug it into your PC and take a look. The "problem" is that the way this works is you create a directory in the file system, and you mount the USB into that directory. So, with or without the USB, the data will be written into that directory when you look at the listing with WinSCP (or anything else). If the USB isn't mounted correctly, then the files just go into the standard file system, if it's ok, then they go onto the USB. As far as I know, there's no way to see this directly other than to unplug the USB (any Linux experts feel free to comment).


WinScp blkid does show my stick there and it's UUID is the same I put into datamine advanced tab.
I suspect it's mounting working ok, but you do have a slightly non-standard configuration with the directory in /tmp. This is normally only suggested if you're using Veras USB logging and dataMine on the same stick...  I think you said earlier that USB logging is disabled now, but if you previously had USB logging enabled, and dataMine also mounting the USB to this directory, then this may cause problems. I'm not 100% sure what would happen - it might be ok, but you would end up with the USB logging mounting to /tmp/log/cmh, and then dataMine mounting to a subdirectory of this. I don't know enough about Unix/Linux file systems to know if this is allowed, but it seems "dodgy".

Another problem you may also be having with this configuration. I think that the /tmp/log/cmh is normally mounted to a RAMdisk (I'm not 100% sure about this, but I think it's the case). In this case, you would be mounting into a volatile directory - again, I don't know enough to say if this is good, bad, or otherwise....  Sorry.

I would suggest that you change the dataMine directory back to "/dataMine" - this would avoid any potential problems that may occur with the above uncertainties. If you are trying to do something "special", then let me know what you want to achieve and I'd be happy to give you my thoughts...

Mounting of file systems under Unix/Linux isn't as straight forward as it is under Windows - it's probably a lot more powerful, but with that comes a lot more potential for confusion! Keeping things simple is definately best  :).

Cheers
Chris