We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287576 times)

Offline alsilco

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #780 on: January 30, 2014, 06:12:26 pm »
Hiya Lights and other gurus!
You seem to be a verrry knowledgeable guys and I was hoping you could shed some light on my Datamine Bug.  Vera support Configured DM and it was working for 12 days, then suddenly stopped 4 days ago.  It was logging perfectly onto a USB Drive in the top slot, even with Vera's developed TemperatureFix app on 3x Aeon 4in1 Multisensor, the 3M CT50 Wifi Thermostat, and Weatherunderground's World Weather App. Vera's Tech's got DM up again, (sensors showing last updated as green), but all graphs are showing as Flat Lines since Jan 27th). Have scoured the forums, and see about USB issues, but I have SCP'ed into the Vera, and do not see any current dates, only the earlier ones. I just got my Vera3, and am surprised I have gotten this far!  Please find attached a screen shot from DM, after it stopped 2 days ago, showing online status as grey, but currently it shows red, and says "Vera is online, however Datamine has errors (error count 146).  When I have uninstalled and reinstalled DM, the errors go away, then start climbing again as the data is collected. Now have the Flash Drive in the lower USB, and still having exactly the same problem. Was logging mostly Temps & Humidity for an engineering project, and I'm now back at square Zero!!   Please Help if you can!! Thanx!!! Al

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: New dataMine graphing and logging plugin
« Reply #781 on: January 31, 2014, 03:42:22 am »
Look at "datamine cifs" on this forum
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline alsilco

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #782 on: January 31, 2014, 11:56:12 am »
Look at "datamine cifs" on this forum

SanderL, Saw the cifs posts, but was hoping to just get back to logging data at present. Here are a few lines from my Datamine log:

50   01/31/14 10:58:46.944   luup_log:63: dataMine: Unable to open file for write /dataMine/database/24/raw/2300.txt <0x2ca41680>
50   01/31/14 10:58:46.944   luup_log:63: dataMine: Error: '/dataMine/database/24/raw/2300.txt: Read-only file system' <0x2ca41680>
06   01/31/14 10:58:46.944   Device_Variable::m_szValue_set device: 59 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ca41680>
50   01/31/14 10:58:46.946   luup_log:63: dataMine: Logging error logging variable 62::urn:micasaverde-com:serviceId:HumiditySensor1::CurrentLevel <0x2ca41680>
50   01/31/14 10:58:46.947   luup_log:63: dataMine: Unable to open file for write /dataMine/database/25/raw/2300.txt <0x2ca41680>
50   01/31/14 10:58:46.963   luup_log:63: dataMine: Error: '/dataMine/database/25/raw/2300.txt: Read-only file system' <0x2ca41680>
06   01/31/14 10:58:46.963   Device_Variable::m_szValue_set device: 62 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ca41680>
50   01/31/14 11:00:03.442   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt <0x2cc41680>
50   01/31/14 11:01:32.957   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt __LEAK__ this:151552 start:2256896 to 0x1143000 <0x2cc41680>
50   01/31/14 11:01:32.969   luup_log:63: dataMine: Logging error logging variable 54::urn:upnp-org:serviceId:TemperatureSensor1::CurrentTemperature <0x2b641680>
50   01/31/14 11:01:32.969   luup_log:63: dataMine: Unable to open file for write /dataMine/database/10/raw/2300.txt <0x2b641680>
50   01/31/14 11:01:32.970   luup_log:63: dataMine: Error: '/dataMine/database/10/raw/2300.txt: Read-only file system' <0x2b641680>
06   01/31/14 11:01:32.970   Device_Variable::m_szValue_set device: 54 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b641680>
50   01/31/14 11:02:50.397   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt <0x2cc41680>

I SCP'ed into the Vera3 and tried to change the permissions on the "Notifications [R2299].txt" and the "2300.txt" files, but was denied in both cases.  See attached photos.

Are these files supposed to be read only?
Why is DM trying to write to read only files?
Do you know to fix this ??
Thanks for your help!
Al

Offline a-lurker

  • Hero Member
  • *****
  • Posts: 872
  • Karma: +66/-8
Re: New dataMine graphing and logging plugin
« Reply #783 on: January 31, 2014, 03:56:22 pm »
First - R2229 File permissions.jpg shows the correct permissions for this file and yes dataMine writes to these files. These particular files store the notifications and associated time stamps.

The operating system will sometimes change a device to read only if it encountered an error with it eg bad sectors, file system corruption, power fail during write, stick pulled out of box while in operation, etc.

If you execute, using SCP:

   cat /proc/mounts

you will see a line with dataMine mentioned. It should show rw, not ro.

If the latter, you can try a full power on/off of Vera and recheck. If that fails - suggest you back up all the files on the stick and copy them over to a new one and retry. Alternatively reformat existing stick and copy files back on to that and retry.

Why this might have occurred is unclear and the suggested fix is not guaranteed.

Offline alsilco

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #784 on: January 31, 2014, 04:26:20 pm »
A-Lurker,
The Cat /proc/mounts yields 14 lines, and yes, the last one is Datamine:

/dev/sda1 /dataMine vfat ro,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

I've seen lots of controversy re: corrupted USB drives getting corrupted, so do I:
1. just yank the power from the Vera3, or is there a better way to power down the unit? (Like is there a way to power down from the UI5 GUI?)
2. Is there a right way to unplug the Flash drive? Do I need to somehow unmount the Flash drive? (I do not need to remove Datamine App to remove the Flash drive, do I?)

Thanks so much,
Al

Offline a-lurker

  • Hero Member
  • *****
  • Posts: 872
  • Karma: +66/-8
Re: New dataMine graphing and logging plugin
« Reply #785 on: January 31, 2014, 05:59:32 pm »
Have a look here:

http://forum.micasaverde.com/index.php/topic,12395.msg96211.html#msg96211

and no need to uninstall the plugin. Make sure you backup everything using SCP first!

Offline mntlvr

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #786 on: February 15, 2014, 10:32:29 am »
Chris
Can you give me some information as to how you have this plugin configure itself to a Temperature sensor or a power meter.
When I set the "parent ID" to my honeywell thermostat ,whose  ID is "108" and vera3 version 1..5.622, then save and reload the plugin completely disappears. When I go to "APPS" and left click the datamine app, it show no installed plugin that it created earlier. So perhaps I am setting up the "parentID" incorrectly.


 [/quote author=Chris link=topic=7750.msg49749#msg49749 date=1315676792]
Update 6 July 2012 - UI5
dataMine on UI5 should now have a seemless install. I've also started an overview document that I'll try and keep up to date with information rather than expecting everyone to sift through what is now a long thread...

To install -:
  • Install the app as usual through UI5 the app install tab
  • Format a USB stick (on Windows works fine for me)
  • Plug the USB stick into Vera
  • Configure up one variable in the dataMine properties (see below)
  • Restart Lua

Once you've installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I'm not 100% sure. I've been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I'm told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.

Note that this should not be used with Veras USB logging enabled. If you have USB logging enabled, you should use a separate USB stick - don't be tempted to use the same drive as this can cause problems.

Original Post
I like the Vera UI - it's clean and simple (although I hope UI5 improves on this). As others have found though, reading logs to get sensor history is not useful. I also am not a fan of the model of hosting data on a separate website - I really like to have my data under my control. This is mainly for security and communication tollerance reasons.
 
I've seen other plugins that allow data to be logged, and then either use a script to produce graphs, or use an external server to process and visualise the data. This is quite possibly a more powerful solution, but I wanted the simplicity of a single unit, and the flexibility of on the fly graphing rather than fixed graphs processed daily in a script.
 
So... the dataMine plugin is born.  This logs all value changes in specified variables to log files on Vera, and provides a GUI graphing interface complete with zoom, scroll etc. The system really must be used with an external USB memory stick since it does produce a relatively large amount of data. I've mounted a 4GB flash drive (FAT32) as /dataMine, and all data files are logged here. I estimate that a 4GB stick would support the system for 10 years or more, so memory isn't really an issue.
 
In addition to the "raw" data logging, the system also logs hourly min/max, and daily min/max. This data is used to support different zoom levels so that the large raw files don't need to be heavily processed to see long timeframes. The system does work with large raw data sets, but it starts to slow down (taking maybe 10 seconds or so to return a 1 year history of temperature data). The creation of these files is still work in progress - currently only the hourly log is generated, and then only manually.
 
Installation
As I've made use of a number of javascript libraries (primarily the flotr graphing package) installation isn't quite as straight forward as a standard plugin (sorry). The usual files (XML, JSON, LUA) are loaded in the normal way through the "developpers" button in the control panel and all files specific to dataMine are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. The config file needs to be stored in another directory /dataMine, which is also where all the data files are stored.  The ZIP file also includes a set of files with approximately 1 year history of outside temperature at my house to allow you to play with the system - these can of course be deleted. There are basic installation instructions in the readme.txt file. (I had to trim down the data due to ZIP size limitations on the server!).

Use
The dataMine plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse.
The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren't hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There's also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon).
 
Limitations
  • Currently, only 1 channel can be graphed at a time. This is something I may change later - the graph library supports multiple channels, but Vera is likely to get a bit grumpy if we serve up too much data!
  • If there is no current data, the graph may not generate. This is because the default graph is currently the last day. Clicking on the month or week button should fix this.
  • The hourly and daily files are not yet generated automatically, but will be processed once per day (this is next on my todo list). I intend to do it this way rather than an "on the fly" method primarily as I find that Vera isn't quite as reliable as I would like, and I find it rebooting many times per day. Data statistics would need to be maintained over these reboots, and the easiest way to do this is to write a temporary log and process this data daily. This does however mean that current data may not be displayed on large zoom levels. This shouldn't be an issue - if you zoom in, you'll see the most up to date data.
  • Data is subsampled. This means that spikes etc may not be displayed on larger zoom levels. I don't think this is a major issue in most applications.

This is still very much work in progress (consider this an alpha version that needs further work). Feel free to give it a go and send any feedback and I'll try and fix any problems. It's worth noting that I've only had my Vera a couple of weeks, so I'm very new to it, and the Lua/Luup system, so I've probably screwed things up, or at least poorly written parts of the code, so go easy on me :) (and maybe a "use at own risk" warning is in force!).

---
A wiki, and the latest code, is available at http://code.mios.com/trac/mios_datamine.
---

Cheers
Chris

---
Attached ZIP file for UI4: 18-Jan-2012
---
[/quote]

Offline erkme73

  • Sr. Member
  • ****
  • Posts: 334
  • Karma: +11/-11
Re: New dataMine graphing and logging plugin
« Reply #787 on: February 22, 2014, 02:36:47 am »
Just installed the plugin, set the mountpoint to /dev/sda1.  I was not prompted to use specific USB drive when I ran the graphing page link.

However, right out the gate, the number of channels shows zero.  The graphing page has nothing listed.  It's green (connected) and no errors, but I can't see anything...

I hate being such a noob, but all of the 'fixes' I've read presume the reader has more experience than I do.  Is there something obvious I'm missing?  Running Vera Lite, UI5 1.5.622




Offline a-lurker

  • Hero Member
  • *****
  • Posts: 872
  • Karma: +66/-8
Re: New dataMine graphing and logging plugin
« Reply #788 on: February 22, 2014, 02:59:02 am »
Looks OK - indicates you have 512 mbyte stick installed. In your 1st image above, you need to click on "Configuration" tab (top left) and select something to graph. It's a tree type structure - just drill down to the sensors of interest on the left side and enable them on the right hand side. Don't forget to save your selections.

Offline tiwas

  • Full Member
  • ***
  • Posts: 186
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #789 on: March 04, 2014, 05:21:58 pm »
Will this work with a passive USB hub or will it need to be powered?

Cheers,

Lars

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: New dataMine graphing and logging plugin
« Reply #790 on: March 05, 2014, 02:11:17 am »
Try it. But with me before I went to CIFS it worked
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline LightsOn

  • Hero Member
  • *****
  • Posts: 754
  • Karma: +4/-3
Re: New dataMine graphing and logging plugin
« Reply #791 on: April 27, 2014, 04:41:45 pm »
Hi all,  I am noticing that every so often my data is lost. I use cifs set up and all works fine. All data colected remains on the cifs but every so often it loses conection to it, however all still works so I can open the datamine page and set items to monitor again so its like it resets but info on the cifs remains it just disconects from it.

Any ideas whats going on?

Sent from my GT-I9300 using Tapatalk


Offline sabolcik

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #792 on: April 29, 2014, 01:24:30 am »
I'm been trying to get the plugin running for the last couple of hours without any success.  Hoping that someone can assist me with getting this resolved as I'm very excited to try out the graphing.  It looks very useful.

I installed a 500MB USB stick both through a USB hub and directly to the VeraLite (note that I am running UI5).  In both cases it shows up via dmesg as being installed.

I tried configuring the setmountpoint variable to /dev/sda1 and also let the graphing startup screen prompt me for the volume name and set it for me (it also chose /dev/sda1 as the mount point).  Restarting vera didn't cause the variable mountlocation and mounttype to get updated, which I think is supposed to happen when a proper install launches.

Checking the debug panel I get the following error message numerous times - luup_log:155: dataMine: Error reading tmpfile during mountcheck <0x2bfd9680>.

At the end of the logfile I get the following information on the disks:

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 524 MB, 524288000 bytes
17 heads, 59 sectors/track, 1020 cylinders
Units = cylinders of 1003 * 512 = 513536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1   *           1        1019      511028   b Win95 FAT32
/dev/sda2            1020        1020         501+ 83 Linux
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -F6F4-DEE5
SetMountPoint   -
SetManualMount  -0
-6-===========================================================================
{"dbVersion":2,"Events":{"count":0,"last":0},"Version":"0.980","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================

I'm not sure if the "doesn't contain a valid partition table" message is targeted at my USB drive.

I have attached the complete log file from datamine.

Can anyone help?

Offline sabolcik

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #793 on: April 29, 2014, 02:01:58 am »
Very strange - I swapped out the USB drive for another one that I tried earlier.  This time when I clicked on the graph tab it showed memory was available but much less than I have on the USB drive (a few 100KB instead of 16GB).  I assumed that  meant that  I was using the vera internal memory for logging.  I poked around some more then when I got back to the datamine device it now showed 16GB.  I reinstalled my USB hub and it still worked.  As far as I know I didn't change anything with the configuration but it just started working.

Weird.  Not sure if anyone can learn anything from my heartache, but I'll let you know if this keeps working and is stable.


Offline LightsOn

  • Hero Member
  • *****
  • Posts: 754
  • Karma: +4/-3
Re: New dataMine graphing and logging plugin
« Reply #794 on: April 29, 2014, 05:16:13 am »
Hi sabolcik,

Yes I found a similar thing historically - there are some posts regarding such issues - or at least similar issues.  the system is quite picky with the type of USB stick and format of it.  I had some that worked all the time - some that worked not at all - and some that worked then broke.

I moved to full cifs and has been a huge improvement other than I issue I am having mentioned above.  all the data still in the cifs - just that data mine no longer connects to it?