We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 245332 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
New dataMine graphing and logging plugin
« on: September 10, 2011, 01:46:32 pm »
Update 6 July 2012 - UI5
dataMine on UI5 should now have a seemless install. I've also started an overview document that I'll try and keep up to date with information rather than expecting everyone to sift through what is now a long thread...

To install -:
  • Install the app as usual through UI5 the app install tab
  • Format a USB stick (on Windows works fine for me)
  • Plug the USB stick into Vera
  • Configure up one variable in the dataMine properties (see below)
  • Restart Lua

Once you've installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I'm not 100% sure. I've been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I'm told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.

Note that this should not be used with Veras USB logging enabled. If you have USB logging enabled, you should use a separate USB stick - don't be tempted to use the same drive as this can cause problems.

Original Post
I like the Vera UI - it's clean and simple (although I hope UI5 improves on this). As others have found though, reading logs to get sensor history is not useful. I also am not a fan of the model of hosting data on a separate website - I really like to have my data under my control. This is mainly for security and communication tollerance reasons.
 
I've seen other plugins that allow data to be logged, and then either use a script to produce graphs, or use an external server to process and visualise the data. This is quite possibly a more powerful solution, but I wanted the simplicity of a single unit, and the flexibility of on the fly graphing rather than fixed graphs processed daily in a script.
 
So... the dataMine plugin is born.  This logs all value changes in specified variables to log files on Vera, and provides a GUI graphing interface complete with zoom, scroll etc. The system really must be used with an external USB memory stick since it does produce a relatively large amount of data. I've mounted a 4GB flash drive (FAT32) as /dataMine, and all data files are logged here. I estimate that a 4GB stick would support the system for 10 years or more, so memory isn't really an issue.
 
In addition to the "raw" data logging, the system also logs hourly min/max, and daily min/max. This data is used to support different zoom levels so that the large raw files don't need to be heavily processed to see long timeframes. The system does work with large raw data sets, but it starts to slow down (taking maybe 10 seconds or so to return a 1 year history of temperature data). The creation of these files is still work in progress - currently only the hourly log is generated, and then only manually.
 
Installation
As I've made use of a number of javascript libraries (primarily the flotr graphing package) installation isn't quite as straight forward as a standard plugin (sorry). The usual files (XML, JSON, LUA) are loaded in the normal way through the "developpers" button in the control panel and all files specific to dataMine are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. The config file needs to be stored in another directory /dataMine, which is also where all the data files are stored.  The ZIP file also includes a set of files with approximately 1 year history of outside temperature at my house to allow you to play with the system - these can of course be deleted. There are basic installation instructions in the readme.txt file. (I had to trim down the data due to ZIP size limitations on the server!).

Use
The dataMine plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse.
The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren't hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There's also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon).
 
Limitations
  • Currently, only 1 channel can be graphed at a time. This is something I may change later - the graph library supports multiple channels, but Vera is likely to get a bit grumpy if we serve up too much data!
  • If there is no current data, the graph may not generate. This is because the default graph is currently the last day. Clicking on the month or week button should fix this.
  • The hourly and daily files are not yet generated automatically, but will be processed once per day (this is next on my todo list). I intend to do it this way rather than an "on the fly" method primarily as I find that Vera isn't quite as reliable as I would like, and I find it rebooting many times per day. Data statistics would need to be maintained over these reboots, and the easiest way to do this is to write a temporary log and process this data daily. This does however mean that current data may not be displayed on large zoom levels. This shouldn't be an issue - if you zoom in, you'll see the most up to date data.
  • Data is subsampled. This means that spikes etc may not be displayed on larger zoom levels. I don't think this is a major issue in most applications.

This is still very much work in progress (consider this an alpha version that needs further work). Feel free to give it a go and send any feedback and I'll try and fix any problems. It's worth noting that I've only had my Vera a couple of weeks, so I'm very new to it, and the Lua/Luup system, so I've probably screwed things up, or at least poorly written parts of the code, so go easy on me :) (and maybe a "use at own risk" warning is in force!).

---
A wiki, and the latest code, is available at http://code.mios.com/trac/mios_datamine.
---

Cheers
Chris

---
Attached ZIP file for UI4: 18-Jan-2012
---
« Last Edit: July 29, 2012, 04:10:24 am by Chris »

Offline utz

  • Sr. Member
  • ****
  • Posts: 274
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #1 on: September 10, 2011, 02:25:49 pm »
Looks great. I built my own graphing tool and it works for me but it is not as generic as this. I will try it out when I find some time.

A nice feature would be to plot binary variables (0 or 1, switches and things like it); especially plotting a few of these on the same chart for debugging purposes (to see when/why a scene was triggered or not).

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #2 on: September 10, 2011, 02:31:19 pm »
Yes, I'd like to get multiple plots working at some stage, but one limitation of this sort of system is that it needs Vera to serve up the data, and that's likely to limit things. I've tried to keep the system as generic and easy to use as possible - there's still quite a bit of work to do yet....

At the moment, this does graph binary data, although you currently need to mess with the config file. There's a "type" variable in the saved json config file, and if you change this to 1, it will draw the graphs assuming the data has been constant between points rather than with diagonal lines.

I've also just tried this with IE and found the graphing doesn't work. the graph library I use can be made to work with IE, but it needs a separate lib which I have included, but it still doesn't work... Something else to look at  ;)

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #3 on: September 11, 2011, 05:03:37 am »
Here's an updated version that should work with IE. I've tested it on IE8 and it seems to work ok, although I note that it takes quite a while to render the Config tab (on Firefox, this only takes a second or two!).

Chris

[Edit] File removed as I've added an updated version later in this thread.
« Last Edit: September 12, 2011, 05:45:53 pm by Chris »

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #4 on: September 11, 2011, 05:22:25 pm »
dataMine looks promising. Thanks for sharing your code.

I'm using a a physical dataMine directory in Vera's root.
Unfortunately, selecting variables doesn't work for me (Vera V2, 1.1.1338, Firefox 6.0.2, latest version of dataMine).

BTW, how did you manage to get the list of variables for each UPnP service?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #5 on: September 11, 2011, 06:03:23 pm »
When you click on the variable name, does anything happen? The variable should highlight in green (as per my picture).
I have found a bug where it doesn't save the configuration (I moved the save code into a separate function and forgot to call it in this part of the code). This would probably stop loggingworking (certainly if you restart Lua), but I think it should still respond by highting the variable. I'll post a fix tomorrow if that sounds like what's happening.

By a little poking around, I found that there's a ajax/json call that the UI makes to get all the configuration data. This includes a table with the devices, services and variables, and with a bit of messing around I've ended up with the code to select variables. Take a look at the javascript for the details (showConfig function).

Chris

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #6 on: September 11, 2011, 08:03:04 pm »
Nothing happens, variable doesn't highlight.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #7 on: September 12, 2011, 09:27:42 am »
Do you have the dataMineConfig.json file in the /dataMine directory? If not, try putting the one that came in the ZIP into the directory.

Also, can you have a look in the log file - maybe there's an error reported in there? I have a suspicion that if there's no config file, then the insert may fail as the config file will be nil. I'll try this out when I get home...

If that's not the issue, I'll add some debug messages to try and isolate the problem.

Chris

Offline radarengineer

  • Sr. Member
  • ****
  • Posts: 402
  • Karma: +4/-0
Re: New dataMine graphing and logging plugin
« Reply #8 on: September 12, 2011, 11:02:18 am »
Chris, great work especially considering you have only have the Vera a few weeks to mess around with it!

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #9 on: September 12, 2011, 01:22:17 pm »
Thanks radarengineer.

Ap15e - I've confirmed that your problem is probably caused by not having the dataMinConfig.json file in the dataMine directory. I've fixed this dependance and I'll upload a new version later tonight to fix this and a couple of other improvements.

Chris

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #10 on: September 12, 2011, 02:09:39 pm »
Code: [Select]
cp /www/cmh/plugins/dataMine/dataMineConfig.json /dataMine

Result:

Code: [Select]
01      09/12/11 20:00:32.812   LuaInterface::CallFunction_Startup-1 device 47 function initialise failed /etc/cmh-ludl/L_DataMine1.lua:48: module 'json' not found:

Looks like Vera is missing the json module (http://code.mios.com/trac/mios_genericutils/wiki/JSONLua?) ...
« Last Edit: September 12, 2011, 02:12:59 pm by Ap15e »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #11 on: September 12, 2011, 02:37:23 pm »
I'm using this module and it works fine here. I don't know if I loaded it onto the system myself or if it came preloaded.

A quick search on my system found /overlay/usr/lib/lua/json.lua. It's only dated last week, so I assume I loaded it myself (too many late nights!). I have some vague recollection of doing this via the "extra Lua files" tab, but I'm not 100% sure. I'll try and confirm this tonight...

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #12 on: September 12, 2011, 04:20:11 pm »

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #13 on: September 12, 2011, 04:43:12 pm »
Another problem:

Code: [Select]
50      09/12/11 22:39:29.513   luup_log:47: dataMine: 1:Unable to open file for read - /dataMine/EnviR Temperature - CurrentTemperature [R2175].txt <0x3c10>
Looks like your code doesn't create the logfiles.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #14 on: September 12, 2011, 05:24:09 pm »
The code does create the files - I believe this error is coming from the graphing function - i.e. when you went to graph the data out (correct?). If it didn't create the file, you'd get an error about not opening the file for write.

Can you take a look and see if there is actually a file in the /dataMine directory?  I don't know how often the EnviRs temp sensor updates, but my Everspring doesn't record a changed temperature very often. It needs at least 2 readings to have been recorded since you enabled logging which can take a while (remember, it's only logging changes). Can you enable logging for the Status variable on a light switch and turn the lights on and off a few times. That ought to generate the file, and (hopefully) it will graph. Also, you can take a look at the files it generates - they are just CSV files.

The only caveat I'd add is that the current version produces very poor graphs of binary data since I was initially focussing on sensor data where subsampling doesn't matter (it really does matter for other data!). I've now fixed this and will upload a new version tonight. So, ignoring the craph grap you'll get, take a look and see if you can ger anything to generate.