We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287705 times)

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #405 on: July 27, 2012, 11:00:08 am »
the dataMine folder is nearly empty, see attached JPG.

I'm uninstalling and removing all files/folders, LUA, etc.... then reinstalling.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #406 on: July 27, 2012, 11:14:54 am »
the dataMine folder is nearly empty, see attached JPG.

I'm uninstalling and removing all files/folders, LUA, etc.... then reinstalling.

Something very strange is happening then, and I don't think it has anything to do with dataMine directly. dataMine doesn't delete any files, so something has deleted everything from your USB drive. I don't really know what could cause this.

One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #407 on: July 27, 2012, 11:48:29 am »
One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

I'm not manually copying anything to the dataMine folder.

Yes, I do have the Vera logs also saving to the USB drive where \dataMine is ... but this should not matter, unless Vera has a very strange bug - since the only some of the dataMine files go missing.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #408 on: July 27, 2012, 11:54:26 am »
One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

I'm not manually copying anything to the dataMine folder.

Yes, I do have the Vera logs also saving to the USB drive where \dataMine is ... but this should not matter, unless Vera has a very strange bug - since the only some of the dataMine files go missing.

This definately does matter - it is not a good idea and can cause strange things to happen. This will effectively mean that you have 2 mount point for the same USB drive, and this will cause unpredictable results (as you're seeing). There have been some other posts on this issue. Looking at the image you posted, there are no other logs, and the log file that is shown is only 700kb, which means it was generated quite recently, so I suspect that this is also going "astray".

You should either disable USB logging, or install a separate USB stick for use with dataMine. This is almost certainly the cause of your problems.

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #409 on: July 27, 2012, 12:28:53 pm »
I don't understand the problem... the drive should be mounted once, but the OS... then the app(s) can access it.  Once the OS mounts it, the app(s) should see it just like any other drive.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #410 on: July 27, 2012, 12:39:21 pm »
I don't understand the problem... the drive should be mounted once, but the OS... then the app(s) can access it.  Once the OS mounts it, the app(s) should see it just like any other drive.
No - both dataMine and Vera USB logging are mounting the drive. Effectively, you now have the same file system mounted into two places, which can cause "bad things" to happen. dataMine mounts to the /dataMine directory, and USB logging mounts to /tmp/logs (or maybe /tmp/logs/cmh - I don't have this enabled, so I'm not sure). It's therefore not easy to have dataMine and logging using the same drive (although it's also not impossible, but the directory structure would have to change).

I recommend you run SSH, or the WinSCP terminal, and type "mount". If there are two mount points for the same drive, as I suspect there is as both Vera logging and dataMine will be mounting it, then you need to change something to eliminate this.

Cheers
Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #411 on: July 27, 2012, 12:59:16 pm »
here's my mount points...

rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sda1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime,errors=continue,barrier=0,data=ordered)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /dataMine type ext3 (rw,relatime,errors=continue,barrier=0,data=ordered)

... so there are two mount points with /dev/sda1 -- I don't understand why this is an issue... why can't two apps use the same mount point, but different folders in that mount point?  Windows can do this, no problem.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #412 on: July 27, 2012, 01:13:37 pm »
... so there are two mount points with /dev/sda1 -- I don't understand why this is an issue... why can't two apps use the same mount point, but different folders in that mount point?  Windows can do this, no problem.

Not really - Windows has a completely different concept. When mounting a drive under Windows, you simply refer to the drive (A: or whatever). The drive becomes a single entity.  Under Linux, you create a directory as the mount point, and then mount the drive to that directory. You've now got the same drive mounted to two places. Under the Windows concept, it's like having the same USB drive mounted as drive A:, and drive B: (which under Windows, you can't do).

I'm not a linux expert, but from what I've read, this produces unstable results. You end up with the Linux kernal trying to manage the USB stick as though it was two drives, and it's not - "bad stuff" then happens. I think most linux systems prevent this (although I've not tried on my Linux box, for obvious reasons), but OpenWRT doesn't.

Chris

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #413 on: July 27, 2012, 02:16:17 pm »
You can not have two mount points pointing to the same device. This will cause data corruption.

- Garrett

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #414 on: July 30, 2012, 09:11:06 pm »
You can not have two mount points pointing to the same device. This will cause data corruption.

I'm not sure the data is corrupting ... the dataMine channels just disappeared again and I looked in the Vera log and found...

Quote
2012-07-30_15:00:40 [is_usb_plugged] WARNING: Found UNPARTITIONED USB STICK Connected
2012-07-30_15:00:41 [erase_part_schema] WARNING: We'll RAW ERASE disc: /dev/sda, having: 984784 sectors

It looks like a problem upon reboot - like Vera has a problem with the drive... a vera bug?

BUT then dataMine must have automatically remounted and created the files it needed since at 3:01pm is the creation date of mount_tests.

So, I'm turning off Vera logging to USB to see if there is an issue... either way, I think this needs to be fixed/addressed. We need to have a way to use the USB drive for more that one thing... like logging from two different apps.  I suspect Linux can do this. Maybe dataMine can use an existing mount point that Vera uses instead of trying to mount its own?
« Last Edit: July 30, 2012, 09:13:05 pm by Aaron »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #415 on: July 31, 2012, 04:43:49 am »
Hi Aaron,
As per the previous comments about this, you can't run both logging and dataMine on the same mount point as things currently stand as this means the drive will be mounted twice. This is known the cause problems. I suspect this log text came from Veras USB logging functionality - it's detected data corruption in that file system, and it's reformatted the drive. When dataMine starts, it reinitialises the configuration files (with nothing being logged obviously).

It's not possible to have 2 mount points to the same drive. As I think I've said previously, the only option would be to put the dataMine directory under the log directory, which isn't exactly intuitive as it would be /tmp/log/cmh/dataMine. This would solve the problem as I would then not need to mount the USB, but would be a bit of a strange place to put the directory. You'd then be totally reliant on Veras USB logging for dataMine to work. If this gets turned off, dataMine will stop working. I can however easily add the option to do this.

The other option is to use a second USB stick for dataMine. In my opinion, this is cleaner, but does of course require a new USB drive.

What do you think?

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #416 on: July 31, 2012, 10:08:31 am »
I think providing the option to log dataMine using Vera's log mount point is best... even though it might not be intuitive, the more I
'm learning about Linux, that would be par for the course... since most things about Linux's file structure is in no way intuitive.

Another reason in favor of this is many of us (including myself) use Insteon, and we already have the other USB port in use for that.

Thank you!

Offline AbNa

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #417 on: August 09, 2012, 07:31:24 am »
Hi,
I seem to have a similar problem on one of my units, the channels are cleared if i reload the engine. It seems the usb does not mount correctly, I have set the correct mountpoint but it does not seem to work. The usb stick is formatted on windows(FAT32). USB logging is not enabled.

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)

Any suggestions?

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #418 on: August 14, 2012, 11:00:29 am »
@Chris
Any new updates to test?



Hi,
I seem to have a similar problem on one of my units, the channels are cleared if i reload the engine. It seems the usb does not mount correctly, I have set the correct mountpoint but it does not seem to work. The usb stick is formatted on windows(FAT32). USB logging is not enabled.

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)

Any suggestions?

This fixed my problem when I had something similar...

uninstall dataMine
remove usb drive from Vera
reformat your USB drive on your PC
unplug power from Vera
put USB drive back in same USB slot in Vera
plug Vera's power back in
reinstall dataMine





Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #419 on: August 19, 2012, 03:53:55 am »
Sorry for the slow replies. I've been working abroad for the past couple of weeks without much access to Internet  :(

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)
I don't understand what you mean by this - what DataMine icon is missing? If you're talking about the plugin icon, then if that's not there how are you configuring the plugin?

Any new updates to test?
Not yet - sorry. I blame my employer... I'm at home for the next week so if the other boss doesn't have too much for me to do at home, I'll try and get a new version out that can work with the USB logging.

Cheers
Chris