The Vera Community forums have moved!

Advanced => Plugins & Plugin Development => Programming => dataMine Plugin => Topic started by: Chris on September 10, 2011, 01:46:32 pm

Title: New dataMine graphing and logging plugin
Post by: Chris on September 10, 2011, 01:46:32 pm
Update 6 July 2012 - UI5
dataMine on UI5 should now have a seemless install. I've also started an overview document that I'll try and keep up to date with information rather than expecting everyone to sift through what is now a long thread...

To install -:

Once you've installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I'm not 100% sure. I've been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I'm told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.

Note that this should not be used with Veras USB logging enabled. If you have USB logging enabled, you should use a separate USB stick - don't be tempted to use the same drive as this can cause problems.

Original Post
I like the Vera UI - it's clean and simple (although I hope UI5 improves on this). As others have found though, reading logs to get sensor history is not useful. I also am not a fan of the model of hosting data on a separate website - I really like to have my data under my control. This is mainly for security and communication tollerance reasons.
 
I've seen other plugins that allow data to be logged, and then either use a script to produce graphs, or use an external server to process and visualise the data. This is quite possibly a more powerful solution, but I wanted the simplicity of a single unit, and the flexibility of on the fly graphing rather than fixed graphs processed daily in a script.
 
So... the dataMine plugin is born.  This logs all value changes in specified variables to log files on Vera, and provides a GUI graphing interface complete with zoom, scroll etc. The system really must be used with an external USB memory stick since it does produce a relatively large amount of data. I've mounted a 4GB flash drive (FAT32) as /dataMine, and all data files are logged here. I estimate that a 4GB stick would support the system for 10 years or more, so memory isn't really an issue.
 
In addition to the "raw" data logging, the system also logs hourly min/max, and daily min/max. This data is used to support different zoom levels so that the large raw files don't need to be heavily processed to see long timeframes. The system does work with large raw data sets, but it starts to slow down (taking maybe 10 seconds or so to return a 1 year history of temperature data). The creation of these files is still work in progress - currently only the hourly log is generated, and then only manually.
 
Installation
As I've made use of a number of javascript libraries (primarily the flotr graphing package) installation isn't quite as straight forward as a standard plugin (sorry). The usual files (XML, JSON, LUA) are loaded in the normal way through the "developpers" button in the control panel and all files specific to dataMine are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. The config file needs to be stored in another directory /dataMine, which is also where all the data files are stored.  The ZIP file also includes a set of files with approximately 1 year history of outside temperature at my house to allow you to play with the system - these can of course be deleted. There are basic installation instructions in the readme.txt file. (I had to trim down the data due to ZIP size limitations on the server!).

Use
The dataMine plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse.
The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren't hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There's also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon).
 
Limitations

This is still very much work in progress (consider this an alpha version that needs further work). Feel free to give it a go and send any feedback and I'll try and fix any problems. It's worth noting that I've only had my Vera a couple of weeks, so I'm very new to it, and the Lua/Luup system, so I've probably screwed things up, or at least poorly written parts of the code, so go easy on me :) (and maybe a "use at own risk" warning is in force!).

---
A wiki, and the latest code, is available at http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine).
---

Cheers
Chris

---
Attached ZIP file for UI4: 18-Jan-2012
---
Title: Re: New dataMine graphing and logging plugin
Post by: utz on September 10, 2011, 02:25:49 pm
Looks great. I built my own graphing tool and it works for me but it is not as generic as this. I will try it out when I find some time.

A nice feature would be to plot binary variables (0 or 1, switches and things like it); especially plotting a few of these on the same chart for debugging purposes (to see when/why a scene was triggered or not).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 10, 2011, 02:31:19 pm
Yes, I'd like to get multiple plots working at some stage, but one limitation of this sort of system is that it needs Vera to serve up the data, and that's likely to limit things. I've tried to keep the system as generic and easy to use as possible - there's still quite a bit of work to do yet....

At the moment, this does graph binary data, although you currently need to mess with the config file. There's a "type" variable in the saved json config file, and if you change this to 1, it will draw the graphs assuming the data has been constant between points rather than with diagonal lines.

I've also just tried this with IE and found the graphing doesn't work. the graph library I use can be made to work with IE, but it needs a separate lib which I have included, but it still doesn't work... Something else to look at  ;)

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 11, 2011, 05:03:37 am
Here's an updated version that should work with IE. I've tested it on IE8 and it seems to work ok, although I note that it takes quite a while to render the Config tab (on Firefox, this only takes a second or two!).

Chris

[Edit] File removed as I've added an updated version later in this thread.
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 11, 2011, 05:22:25 pm
dataMine looks promising. Thanks for sharing your code.

I'm using a a physical dataMine directory in Vera's root.
Unfortunately, selecting variables doesn't work for me (Vera V2, 1.1.1338, Firefox 6.0.2, latest version of dataMine).

BTW, how did you manage to get the list of variables for each UPnP service?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 11, 2011, 06:03:23 pm
When you click on the variable name, does anything happen? The variable should highlight in green (as per my picture).
I have found a bug where it doesn't save the configuration (I moved the save code into a separate function and forgot to call it in this part of the code). This would probably stop loggingworking (certainly if you restart Lua), but I think it should still respond by highting the variable. I'll post a fix tomorrow if that sounds like what's happening.

By a little poking around, I found that there's a ajax/json call that the UI makes to get all the configuration data. This includes a table with the devices, services and variables, and with a bit of messing around I've ended up with the code to select variables. Take a look at the javascript for the details (showConfig function).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 11, 2011, 08:03:04 pm
Nothing happens, variable doesn't highlight.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2011, 09:27:42 am
Do you have the dataMineConfig.json file in the /dataMine directory? If not, try putting the one that came in the ZIP into the directory.

Also, can you have a look in the log file - maybe there's an error reported in there? I have a suspicion that if there's no config file, then the insert may fail as the config file will be nil. I'll try this out when I get home...

If that's not the issue, I'll add some debug messages to try and isolate the problem.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: radarengineer on September 12, 2011, 11:02:18 am
Chris, great work especially considering you have only have the Vera a few weeks to mess around with it!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2011, 01:22:17 pm
Thanks radarengineer.

Ap15e - I've confirmed that your problem is probably caused by not having the dataMinConfig.json file in the dataMine directory. I've fixed this dependance and I'll upload a new version later tonight to fix this and a couple of other improvements.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 12, 2011, 02:09:39 pm
Code: [Select]
cp /www/cmh/plugins/dataMine/dataMineConfig.json /dataMine

Result:

Code: [Select]
01      09/12/11 20:00:32.812   LuaInterface::CallFunction_Startup-1 device 47 function initialise failed /etc/cmh-ludl/L_DataMine1.lua:48: module 'json' not found:

Looks like Vera is missing the json module (http://code.mios.com/trac/mios_genericutils/wiki/JSONLua?) ...
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2011, 02:37:23 pm
I'm using this module and it works fine here. I don't know if I loaded it onto the system myself or if it came preloaded.

A quick search on my system found /overlay/usr/lib/lua/json.lua. It's only dated last week, so I assume I loaded it myself (too many late nights!). I have some vague recollection of doing this via the "extra Lua files" tab, but I'm not 100% sure. I'll try and confirm this tonight...
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 12, 2011, 04:20:11 pm
Success! Had to scp http://code.mios.com/trac/mios_genericutils/export/20/json.lua to /usr/lib/lua.
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 12, 2011, 04:43:12 pm
Another problem:

Code: [Select]
50      09/12/11 22:39:29.513   luup_log:47: dataMine: 1:Unable to open file for read - /dataMine/EnviR Temperature - CurrentTemperature [R2175].txt <0x3c10>
Looks like your code doesn't create the logfiles.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2011, 05:24:09 pm
The code does create the files - I believe this error is coming from the graphing function - i.e. when you went to graph the data out (correct?). If it didn't create the file, you'd get an error about not opening the file for write.

Can you take a look and see if there is actually a file in the /dataMine directory?  I don't know how often the EnviRs temp sensor updates, but my Everspring doesn't record a changed temperature very often. It needs at least 2 readings to have been recorded since you enabled logging which can take a while (remember, it's only logging changes). Can you enable logging for the Status variable on a light switch and turn the lights on and off a few times. That ought to generate the file, and (hopefully) it will graph. Also, you can take a look at the files it generates - they are just CSV files.

The only caveat I'd add is that the current version produces very poor graphs of binary data since I was initially focussing on sensor data where subsampling doesn't matter (it really does matter for other data!). I've now fixed this and will upload a new version tonight. So, ignoring the craph grap you'll get, take a look and see if you can ger anything to generate.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2011, 05:44:29 pm
Updated software version.

This fixes a couple of bugs if the config file is not present.
It also improves the graphs for light switches and other status type data by eliminating sub sampling for graphs over shortish time periods. This automatically gets enabled for all "Status" and "Watts" variables, but I will look at better ways of doing this.

If you're upgrading, you should only need to upload the J_DataMine1.js and L_DataMine1.lua files.

Edit: File removed as there's now an updated version posted in this thread, alternatively, visit http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine)
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 13, 2011, 06:30:54 am
There are no data files in /dataMine. EnviR update interval: 6 seconds
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 13, 2011, 08:51:50 am
dataMine works by logging changes, so it's not quite as simple as "6 seconds". If it doesn't change, then it still won't log anything until the temperature does change.
That said, if you've been running this for a long time, it's unlikelt the temperature hasn't changed, so there's quite likely a problem.

Anyway, a few questions might help me out...

I can add more logging to try and work out what's happening, but there should be sufficient logging already (at least most errors are logged). It would great if you can send me any dataMine log events - they should all start with dataMine...
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 13, 2011, 04:29:40 pm
Quote
dataMine works by logging changes, so it's not quite as simple as "6 seconds".

Yes, I know - but the EnviR temperature sensor is sufficiently flaky: :)
http://forum.micasaverde.com/index.php?topic=6752.msg46748#msg46748

The latest version of your plugin seems to work:

EnviR Temperature - CurrentTemperature [R2175].txt:
Code: [Select]
1315939858,25.7
1315939876,25.5
1315939900,25.4
1315939918,25.5
1315939930,25.6
1315939936,25.5
1315939948,25.6
1315939961,25.4
1315939967,25.6
1315939991,25.7
1315939997,25.5
1315940009,25.6
1315940027,25.8
1315940047,25.7
[...]

Corresponding GDL output:

Code: [Select]
Sep 13 20:50:58 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939858 EnviR_Temperature 25.7
Sep 13 20:51:16 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939876 EnviR_Temperature 25.5
Sep 13 20:51:40 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939900 EnviR_Temperature 25.4
Sep 13 20:51:58 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939918 EnviR_Temperature 25.5
Sep 13 20:52:10 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939930 EnviR_Temperature 25.6
Sep 13 20:52:16 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939936 EnviR_Temperature 25.5
Sep 13 20:52:28 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939948 EnviR_Temperature 25.6
Sep 13 20:52:41 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939961 EnviR_Temperature 25.4
Sep 13 20:52:47 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939967 EnviR_Temperature 25.6
Sep 13 20:53:11 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939991 EnviR_Temperature 25.7
Sep 13 20:53:17 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315939997 EnviR_Temperature 25.5
Sep 13 20:53:29 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940009 EnviR_Temperature 25.6
Sep 13 20:53:47 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940027 EnviR_Temperature 25.8
Sep 13 20:54:07 192.168.x.y GDL_40_urn:upnp-org:serviceId:TemperatureSensor1/CurrentTemperature: 1315940047 EnviR_Temperature 25.7

- a perfect match.

Some observations and suggestions:

Code: [Select]
menu line for graph 1
graph1
Button: 'add graph'

menu line for graph 1
graph1
menu line for graph 2

menu line for graph 1
graph1
menu line for graph 2
graph2
Button: 'add graph'

...
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 13, 2011, 06:09:26 pm
Hey - that's great - I'm pleased it's working...  :)

The reason there are two values is to make graphs that look "correct". Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct. ie the light was off all night - it wasn't slowly turning on over night  :) So, when I come to display the graph, I add the extra values - it actually more correctly represents the data that Vera was displaying, but it's just not consistant with what's being logged. Due to the float bug in Vera, it's much easier to use the same time rather than subtract a second (or whatever) - I originally tried this, and it caused all sorts of problems.

I'd have to think about non numeric data - that's possibly not so simple.

Timezones are actually done, so you can expect to see local times in the next rev.

Multiple graphs, or multiple plots on the same graph is definately something I'm keen on. The plugin "window" is quite small though and this limits the graph size, so multiple plots seem like the way to go (I think, but I'm open to other thoughts on this as both have their benefits).


Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 13, 2011, 07:09:02 pm
Quote
Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct.

You might find the GDP source code an interesting read.

You could use different plotting algorithms for binary data (light switch), discrete data (dimmer), continuous data (temperature) and alpha(numeric) data (wind direction).

Question:
Does deactivating the logging require a LuaUPnP restart/reboot?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 14, 2011, 05:30:25 am
Quote
You could use different plotting algorithms for binary data

The flotr package doesn't have a different algorithm, so I've effectively done this myself. I may however move some of this into the JS to try and reduce the load on Vera, but JS isn't the fastest thing around either - it's just a case of apportioning the functionality to the most appropriate place in the system.  I'll try and take a look at GDP.

Quote
Question:
Does deactivating the logging require a LuaUPnP restart/reboot?
As far as I'm aware, there is no variable_unwatch function, so yes, you need to restart Luup for changes to take effect. If you're aware of another method to stop watching a variable, then I'd be happy to implement it.
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on September 14, 2011, 06:50:56 am
If you're aware of another method to stop watching a variable, then I'd be happy to implement it.

You could create a flag e.g. WATCH_VARIABLE and check it at the beginning of the callback function. If the flag is 0/false, return immediately. It's not the most elegant solution, but it works.
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on September 14, 2011, 11:49:02 am
Quote
Since we're only logging changes, graphing things like light switche status would result in "diagonal" lines which actually isn't correct.

Correct. But why are there "diagonal" lines?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 14, 2011, 01:48:48 pm
Quote
You could create a flag e.g. WATCH_VARIABLE and check it at the beginning of the callback function. If the flag is 0/false, return immediately. It's not the most elegant solution, but it works.

Actually, this is easy - I already have this flag in my config structure (i.e. logging on/off). The config structure gets scanned in the "watch" function, so I can easily add this. It's not the best solution since it still means Vera is going to the (wasted) effort of calling my plugin, but it will work.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 14, 2011, 02:01:28 pm
Quote
Correct. But why are there "diagonal" lines?

Good question! I think it's probably to do with the way I sub sample the data, and where I record the last value - these aren't in the same place, so sometimes the extra point I put in is not for the last value that was graphed, but the last value in the file.

To avoid sending too much data from Vera to the UI, I sub sample the file. It's simply ("End Time" - "Start Time") / 500 (500 being roughly the number of points on the graph). For data that's roughly evenly logged, this works fine, but for other data where you get a lot of changes over a short time, this falls down. There's a slightly modified algorithm for switch data to avoid this.

I might play around with this and see if it can be removed (or reduced). I don't know how Vera gets on with concatenating large strings which is mainly why this is implemented this way.
Title: Re: New dataMine graphing and logging plugin
Post by: kskenyon on September 20, 2011, 09:39:01 am
I have dataMine working (mostly) on a Vera2@1245 with a USB drive.

Is there a way to delete an item from being logged without editing the files directly?

One of my variables will not stay green when it is clicked in the Config list.  I assume it is not being saved for some reason.  Other variables are working well.

Is it possible to use the local time zone for graphing?  My brain is not used to working in GMT even though the calculation is simple.

Nice work.  I think this will be a popular plugin when it's ready for prime time.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 20, 2011, 11:30:41 am
Quote
Is there a way to delete an item from being logged without editing the files directly?
Clicking on the variable so that it goes yellow will disable logging. It won't however delete the data files that have already been generated. I was considering adding this, but decided to simply stop logging. Is this what you mean, or do you want the data files deleted?
--- EDIT--- Actually, in the current public version, you will also need to restart Lua for this to take effect. In the next version, this will be automatic.

Quote
One of my variables will not stay green when it is clicked in the Config list.
Can you describe what happens? You click on the variable, it initially goes green (?), but if you then close and restart dataMine, it's no longer green (?) or something else?
I had an error a few days back where somehow the config file got written with no data in it, and that caused an error that stopped variables being logged (although not just on a single variable), otherwise I've not seen any issues with this.

Quote
Is it possible to use the local time zone for graphing?
Not in the version you've got, but it is implemented in the version I'm working on at the moment. Hopefully I'll get this out in a few days time - I'm looking at ways to add multiple channels to a single graph. If it starts to take too long I'll upload an intermediate version to add this feature.

Quote
Nice work. I think this will be a popular plugin when it's ready for prime time.
Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: kskenyon on September 21, 2011, 12:05:33 am
One of my variables will not stay green when it is clicked in the Config list.
Can you describe what happens? You click on the variable, it initially goes green (?), but if you then close and restart dataMine, it's no longer green (?) or something else?
I have two TED 5000 Devices that are updating OK in the main UI4.  One seems to update the db in dataMine, the other does not.  That's why I wanted to delete the variable from the graph pull down list and try again.  It took several tries to get the variable to stay green in the Config and now it seems to stay green but does not update the database. Not sure what I did to get it to stay green.  Some combination of opening and closing windows until it stayed green.  The other one is working fine and they are otherwise identical devices. I thought about deleting everything that dataMine has created and starting over but it has been collecting data on other variables for a while.  Can I just delete a .txt file for a variable without messing everything else up?



Thanks for the upcoming fix for the TZ.
Title: Re: New dataMine graphing and logging plugin
Post by: kskenyon on September 21, 2011, 10:13:01 am
Not sure what I did but it seems to be working now.  I'd still like to delete some "mistakes" I made from the Graphing pull down menu.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 21, 2011, 10:30:08 am
Quote
I have two TED 5000 Devices that are updating OK in the main UI4
I assume they have different names? As you've probably noticed, dataMine uses the device name and variable name to make up the filename. If the devices had the same name, this may cause problems (I think you would end up with a single file with both lots of data).

Quote
It took several tries to get the variable to stay green in the Config
Strange. When you click on a variable name, it sends a message back to dataMine, and dataMine should update an internal table, and rewrite the dataMineConfig.json. If it does this (actually, just the updating of the internal table), then it should stay green. I think there may still be a bug which could cause a problem if the config file was either not initially present, or corrupted - this has been fixed (I really should upload an update). I don't know if this is the problem you've seen though...


Quote
Can I just delete a .txt file for a variable without messing everything else up?
Yes. The generation of the txt files is completely independant. Each time the variables change, Vera sends a message which dataMine receives and logs the data to a file. Each time dataMine logs the data, it opens the file, writes the data, and closes the file. So, the file will be generated at that point.
However, this won't change the logging - the configuration is stored in the dataMineConfig.json file - this file stores the variable names, file names, status (logging 'green' or 'yellow'). If you download this file, you can make it look pretty by putting it into www.jsonlint.com (http://www.jsonlint.com).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 30, 2011, 06:23:17 pm
I've added a few features to dataMine. There have been a few fixes/additions under the hood, but the main changes are -:

The system allows you to select multiple channels and place them onto the left or right Y axis to allow for channels with different scales. This is shown in the attachement. When you click on the "Select Channels" button, there are four fields of interest on each line. First, there's an arrow button. Pressing this will instantly graph this channel with no further action required. Next is the variable name. Clicking on this selects the channel, and enables two further buttons. The first button on the right is a "go" button - it will display all variables currently selected. The second button allows you to select two the left or right axis. This is extremely useful if you have different types of data (say, temperature and humidity) which would otherwise not graph well on the same scale.

It may take a little bit of getting used to, but it's a lot more powerful than a simple dropdown list! I'm  open to suggestions on improving this if it's deemed to be too complex.

If you're upgrading, note that there are a number of new PNG files to support the new buttons and features of the software, so please ensure you upload these as per the installation instructions. The LUA, JSON, XML and JS files also need to be updated via the "MiOS Developers" button.

A wiki has also been started at http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine).
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on October 11, 2011, 10:59:35 am
Hi Chris,

Could it be that the dataMineConfig.json file is missing in your ZIP file?

I installed dataMine, but I can't get it to work, reading trough the posts I noticed you were referring to dataMineConfig.json file, and that it should be present in /dataMine, but I cant find it in your ZIP...

Thanks,
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 15, 2011, 05:19:17 pm
Sorry for the slow response - I'm travelling in New Zealand at the moment...

You shouldn't need to have the json file now - dataMine should create it for you when you go into the config page and select variables to log. I did include a copy of the json file in an early zip file, but haven't bothered recently.

If you can tell me what's happening, and also take a look in the vera log file for and "dataMine" entries and I'll take a look into the problem. I won't be back home though until the 25th, so won't be able to do a lot until then, but I will be checking the forum semi regularly over the next week.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on October 16, 2011, 02:08:03 am
No problem Chris :)

I have installed datamine but I can't get it to work. I ran trough the instructions several times, deleted the created device and started the installation again, but can't find anything I missed...

I can create the device, but it does not start as it should, see pictures attached.

I am running 1338, but that should not be a problem, as it is referenced in this post.

I also noticed that the /etc/init.d/custom-user-startup file does not exist in my system.
Does that point to something I should be checking?

Thanks,
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 16, 2011, 05:19:43 pm
Quote
I can create the device, but it does not start as it should, see pictures attached.
The first thing I noticed is that the icon is a default icon - not the graph icon that should be displaying in dataMine. This could mean that there are files missing from the installation, or that the I_DataMine1.XML file isn't specified in the UpnpImplFilename in the configuration (the picture of your config doesn't show this).

It would be worth looking in the log file to see if there are any "dataMine" entries being logged. dataMine logs a status message on startup. You could also look further down the config screen to see if dataMine has created any variables - it should create one for the data directory - this will tell you if the software has been running at all.

Quote
I also noticed that the /etc/init.d/custom-user-startup file does not exist in my system.
You don't necessarily need this file - I use it to mount the USB drive when Vera starts. I couldn't get the Vera option of using a USB stick for the log files to work, so I mounted a USB stick outside of the Vera software, and pointed my /dataMine directory into the USB stick. I think by default this file doesn't exist, and you should create it with the mount command (as per the info on the wiki) to remount the USB drive when Vera starts.

However, if you have simply created a directory in the main filesystem, and aren't using a USB stick, then you don't need the startup file as won't be mounting a USB stick. I don't recommend this as dataMine will take a lot of disk space. It also results in writing to the Vera flash memory, which has limited write cycles available. You can use it like this for short term testing, but not long term.

So, I would -:
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on October 17, 2011, 12:39:48 am
Thanks for your help Chris,

•Look in the log file to see if there are any dataMine entries.
No entries found

•Check you have the implementation file loaded (through the developers menu), and configured.
I_DataMine1.XML file has been loaded and is specified in the UpnpImplFilename

•Check to see if there is a diretrory variable toward the bottom of the config screen
No variables have appeared, see screen attached

A USB stick has been mounted. I created the custom-user-startup file manually, but it did not mount the USB after a reboot. As you said this is only needed in case I need to reboot, I hoped it pointed to some package I missing on my system that is required to run datamine...

Where to look next?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 17, 2011, 01:16:35 am
I've just noticed one thing that could be the problem - you have the device_type incorrect. This is my fault - I've specified it incorrectly on the wiki page (I'll change it shortly).

device_type should be urn:schemas-cd-jackson-com:device:DataMine:1 - I've specified servicId.

I'm not 100% sure that this will resolve the issue as there are a few things I can see that I think should work, but I'm not really sure how Vera works under the hood so let's see how it goes.
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on October 18, 2011, 03:18:42 pm
The device type correction gave some progress...datamine is starting...but something is still wrong....

- Vera keeps saying;  Running Lua Startup
- The Setup Tab gives me a list of devices and their variables, and I can select them (they become green), but when I go to a different ab, and then return to the Setup Tab, they are not green green any longer. Is this as intented or should a previously selected line still appear green?
- Datamine is logging data in folder /Datamine
- When selecting the Select Channels button on the Graph Tab, an empty list is displayed
- The  L_DataMine1.lua file shows as only file in the /overlay/etc/cmh-ludl folder without .lzo extionsion, no clue if this could be related to the Running Lua Startup message.

What do you think?

Thanks,
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on October 19, 2011, 02:21:02 am
More news this morning...

The Select Channels drop down list has been populated with selected variables (did took a couple of hours on my system) and datamine produces a graph, but only for 1 value. Displaying 2 channels on 1 or 2 axles does not seems to be working.

Vera still is continiously notifying that it is Running Lua Startup
 
Selected values in the Setup Tab now remain green once selected. When unselecting they turn yellow.

Is there a way to remove variables from the channels drop down list?

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: readw on October 21, 2011, 06:14:00 am
Hi All,

I am very unfamiliar with openwrt.

In the install instruction it advises that,

md /dataMine
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine


If I am logged onto the root directory of vera and try the command md /dataMine i get the following error

-ash: md: not found

What am I doing wrong?

thanks

Warren
Title: Re: New dataMine graphing and logging plugin
Post by: futzle on October 21, 2011, 07:09:58 am
Instead of md use mkdir
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on October 30, 2011, 06:53:55 am
dataMine is failing after changing some settings via Config tab:

Code: [Select]
10/30/11 11:49:06.837   ^[[31;1mluup_log:42: GoogleWeather: Google Weather #42 starting up with id ^[[0m <0x402>  10/30/11 11:49:06.940   luup_log:47: dataMine: Initialising Data Mining System <0x402>
10/30/11 11:49:06.986   ^[[31;1mLuaInterface::CallFunction_Startup-1 device 47 function initialise failed /usr/lib/lua/json.lua:122: Unterminated JSON
10/30/11 11:49:06.987   ^[[31;1mLuImplementation::StartLua running startup code for 47 I_DataMine1.xml failed^[[0m <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 31, 2011, 09:49:25 am
Sorry for the delay in responding to this thread - I've been "down under" in New Zealand and have just recovered from the trip...

Quote
dataMine is failing after changing some settings via Config tab
I've seen this happen too - when it happened to me, it was caused by the json file being written incorrectly. I use a JSON library to turn the Lua tables into json to save the config, and if something bad happens here, then you end up with this. After it happened to me, I planned to rename the old version of the dataMineConfig.json rather than just overwrite it, but I don't recall if I have completed this or not. Take a look in the /dataMine directory to see if there's a backup, or take a look in the json file to see if it's correctly formatted (or indeed, if it has anything in it).

I'll take a look at trying to better handle this, although it's unclear to me what I can do at the moment (other than keep a backup of the config).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 31, 2011, 09:53:38 am
@Jackpost - are you still experiencing issues with the "Running Lua Startup" message and getting multiple graphs plotted?

Quote
Is there a way to remove variables from the channels drop down list?
Yes, and no. Currently, if you change the color to yellow in the config list, then it is not being logged anymore, but you can't completely remove it from the list. The reason I don't completely remove it is to allow users to disable logging, while still keeping, and being able to graph, old data. I will probably add another button to the config screen to allow users to remove the channel completely.
Title: Re: New dataMine graphing and logging plugin
Post by: Jackpost on November 01, 2011, 02:00:37 pm
The Running Lua Startup message neever went away and getting multiple graphs plotted did not work eighter, I did not find anything wrong when follwoing the installation instructions...

This weekend my Vera froze, after a reboot it was available for a short time but kept getting frozen again...so I reverted to a backup. I didnt reinstall DataMine.

Thanks for your help getting a solution, for me a reporting tool like datamine is a big gap in the current Vera UI...
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 01, 2011, 05:18:57 pm
Quote
Take a look in the /dataMine directory to see if there's a backup, or take a look in the json file to see if it's correctly formatted (or indeed, if it has anything in it).

The JSON file is empty, there is no backup file.
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on November 03, 2011, 07:55:06 pm
I'm having difficultly getting the datamine plugin to work, It shows up, but I can't highlight anything on the config tab.
It also didn't seem to create a config file in /datamine
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 06, 2011, 06:32:58 pm
Have installed dataMine over weekend, all went reasonably well (big learning curve!) but just found there is a "stack overflow at line: 0" which I suspect mught be caused when I am accessing remotely. This didn't show up when I was at home.

The greater challenge is the "dataMine Startup LUA Failed" message which I think is leading to non viewability of any graphs - ie I don't think it's pulling any data. I've scoured the forum, reinstalled files, checked and double checked, rebooted Vera unit, and have come to a dead end. I did see a topic on Startup LuA failed but not dataMine-related. I also have used this link http://code.mios.com/trac/mios_datamine to install, as it appears to be the main source of info/instruction.

All else works - lights turn on and off when they should, remotes trigger after dusk, scenes work, so none of that has been affected.

What would the best next step be - is this a "log with MCV" item, or are they only likely to deal with their product, rather than a separate function? Or have I just missed something obvious (just not to me!)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 07, 2011, 01:18:00 pm
Quote
I'm having difficultly getting the datamine plugin to work, It shows up, but I can't highlight anything on the config tab.
It also didn't seem to create a config file in /datamine
I would say  the LUA file isn't running for some reason. Can you take a look in the logfile and see if there are and "dataMine" entries in there. You should at least see dataMine initialisation messages. If not, make sure the L_dataMine1.LUA file is loaded using the developpers tab (sorry - you've probably done this right, but it's the only thing I can think of right now).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 07, 2011, 01:27:43 pm
Quote
just found there is a "stack overflow at line: 0" which I suspect mught be caused when I am accessing remotely. This didn't show up when I was at home.
By this, do you mean you get a pop-up in Internet Explorer saying stack overflow? I've also see this - I hadn't attributed it to dataMine though (yet!). This happens on my work computer when I try and access remotely (it runs IE) but does it even if I don't click on dataMine. I assume this error is a JavaScript error, and given that dataMine java doesn't load until you click on the dataMine properties button, I assumed it wasn't dataMine... This doesn't happen on other computers though (running IE or Firefox).
I should also add that dataMine won't work remotely due to the way javascript files are loaded.

Quote
The greater challenge is the "dataMine Startup LUA Failed" message which I think is leading to non viewability of any graphs
This was reported by another user, but they uninstalled dataMine so I never looked into the issue.
Normally when LUA crashes, it should put an error into the log file - can you take a look and see if there'a anything there? The very first thing dataMine does when it starts is write an entry into the log saying Initialising Data Mining System so if you search for that and see if there's an error showing in the next few lines. It should give a line number for the error which will hopefully help me find the problem. I'm by no means a LUA expert programmer and I'm not 100% sure how LUA handles different errors other than it generally results in crashing the application as you've seen (other apps will stay running ok). If I can find the error, I can add a check to stop this and report the issue properly.

My guess is that MCV won't be too interested unless I can find a problem that's in their code. I'll try and find the issue if you can stick with me while we work on it.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 07, 2011, 09:19:35 pm
Hi Chris, thanks for your replies. I'm looking forward to seeing what dataMine can provide for me - my main motivation in getting a Vera system is to cut down our $3,000 per year electricity bill!

I'll check out the logfile when I get home later tonight - I do have the L_dataMine1.LUA loaded (just checked) but will see what the logfile provides.

The Stack Overflow popup only appears when I log in via work computer, and it only started appearing after I loaded dataMine. Using IE at work, Chrome at home. It's not an issue, since I can still see/access/use the dashboard after "Ok"ing the message to clear it. I figured dataMine wouldn't work remotely - I saw that the graphics/images didn't load when trying to access from work.

I'd seen the other user comments you mentioned - pity nothing came of that though.

Will come back to you with the log file contents - hopefully it will shed some light on the issue. Thanks again, both for your reply now, and the work you have put in to build what appears to be a useful addition to the system.
Title: Chris I need you advice
Post by: wichiepr on November 08, 2011, 06:06:57 pm
I am having problem to run datamine,
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 09, 2011, 05:31:17 am

Quote
The greater challenge is the .....

OK - first thing is to ask you to remember my status over there on the left <Chris glances at "Newbie", nods knowingly>  ;)

So - with that in mind I am struggling to get hold of the suggested log file - have checked in /tmp/log, /tmp/log/cmh and even run a search thru all directories for file named logfile or dataMine with no luck. There is a dataMine directory sitting in /root although I am now wondering if I mounted the USB stick properly. FWIW L_dataMine1.LUA is in place.

'scuse my ignorance in all things coding...
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 09, 2011, 01:26:08 pm
Hey, no probs - I shouldn't have blindly expected you to find the log actually - it's a bit obscure...

Look in /tmp/log/cmh - the file you want is LuaUPnP.log. This is the file where all the debug and other information ends up, and there are a few dataMine messages that go in here as well.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 10, 2011, 06:25:26 am
Hey, no probs - I shouldn't have blindly expected you to find the log actually - it's a bit obscure...

Look in /tmp/log/cmh - the file you want is LuaUPnP.log. This is the file where all the debug and other information ends up, and there are a few dataMine messages that go in here as well.

Hmm - the below few lines seem to tell a story - there's a few other dataMine entries too. What's the opinion? (Still wondering about whether I mounted teh USB stick properly...)


01   11/10/11 21:02:20.046   LuaInterface::CallFunction_Startup-1 device 11 function initialise failed [string "..."]:7: module 'L_DataMine1' not found:
   no field package.preload['L_DataMine1']
   no file './L_DataMine1.lua'
   no file '/usr/share/lua/L_DataMine1.lua'
   no file '/usr/share/lua/L_DataMine1/init.lua'
   no file '/usr/lib/lua/L_DataMine1.lua'
   no file '/usr/lib/lua/L_DataMine1/init.lua'
   no file '/etc/cmh-ludl/L_DataMine1.lua'
   no file '/etc/cmh-lu/L_DataMine1.lua'
   no file './L_DataMine1.so'
   no file '/usr/lib/lua/L_DataMine1.so'
   no file '/usr/lib/lua/loadall.so' <0x402>
01   11/10/11 21:02:20.047   LuImplementation::StartLua running startup code for 11 I_DataMine1.xml failed <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 10, 2011, 01:32:19 pm
I don't think it's USB related.

I haven't seen this error before, but it looks like the MIOS system that loads LUA files can't find the L_DataMine1.lua file. So, my guess is that the L_DataMine1.lua file isn't loaded correctly. This file should be loaded using the "developers" tab, and not put somewhere else (e.g. on the USB stick or in the plugins directory) with the likes of the JS files and image files.

If it is loaded correctly, then one other possibility is that there a case sensitivity issue. I suspect (being Linux) that the filename has to have exactly the same case as in the error message. This should be correct, but sometimes Windows changes case (since it doesn't care) and this screws up Linux.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 10, 2011, 07:53:08 pm
I don't think it's USB related.

I haven't seen this error before, but it looks like the MIOS system that loads LUA files can't find the L_DataMine1.lua file. So, my guess is that the L_DataMine1.lua file isn't loaded correctly. This file should be loaded using the "developers" tab, and not put somewhere else (e.g. on the USB stick or in the plugins directory) with the likes of the JS files and image files.

If it is loaded correctly, then one other possibility is that there a case sensitivity issue. I suspect (being Linux) that the filename has to have exactly the same case as in the error message. This should be correct, but sometimes Windows changes case (since it doesn't care) and this screws up Linux.

Yeah - I WinSCP'd and looked at those directories above and files as stated are not there. I had definitely loaded the L_DataMine1.lua file via the Dev tab - and have since reloaded it, but with the same results. I wasn't sure if it is possible to delete the file first then reload, but it is still showing the same result.

I also ran a search for the file and it only sits in .etc/cmh-ludl/ and .overlay/etc/cmh-ludl/ with content appearing to be exactly the same in both cases.

Would it help to see the complete LuaUPnP.log file in case it has some other message (that I missed) - if so, I can send it thru.

PS I have also F5'd the screen to refresh and reload, tried different browsers (usually use Chrome but IE and FF have same result), turned off and back on the Vera unit.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 12, 2011, 06:28:20 am
Quote
I also ran a search for the file and it only sits in .etc/cmh-ludl/ and .overlay/etc/cmh-ludl/
That's where it should be when loaded with the developers app. I assume it has exactly the same case, and doesn't have lzo appended to the end? If so, that would be consistant with my configuration.

Something doesn't add up. If I look at the error message, I see the following -:
Quote
no file '/etc/cmh-ludl/L_DataMine1.lua'
And yet this file does exist in this directory? I'm not sure that the problem is (directly) a dataMine problem - I don't know if MCV are monitoring this thread, but I'd certainly welcome input from them. I don't know how Vera loads Lua files, but to me it looks like you've got enough to start the program. I don't think looking at the rest of the log will help (at least, I don't think it will help me - MCV may know more, but it seems to indicate the Vera isn't loading the dataMine plugin files at all, so until we get over that hurdle, the log isn't going to help me.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 12, 2011, 08:09:18 am
Quote
I also ran a search for the file and it only sits in .etc/cmh-ludl/ and .overlay/etc/cmh-ludl/
That's where it should be when loaded with the developers app. I assume it has exactly the same case, and doesn't have lzo appended to the end? If so, that would be consistant with my configuration.

Something doesn't add up. If I look at the error message, I see the following -:
Quote
no file '/etc/cmh-ludl/L_DataMine1.lua'
And yet this file does exist in this directory? I'm not sure that the problem is (directly) a dataMine problem - I don't know if MCV are monitoring this thread, but I'd certainly welcome input from them. I don't know how Vera loads Lua files, but to me it looks like you've got enough to start the program. I don't think looking at the rest of the log will help (at least, I don't think it will help me - MCV may know more, but it seems to indicate the Vera isn't loading the dataMine plugin files at all, so until we get over that hurdle, the log isn't going to help me.

OK - this might be progress - there is lzo appended on the file in both locations - as is the case with the other files in each directory. Are you saying it should be without the lzo? See screenshot attached for info - both directories hold exactly the same files.

In relation to your second point, the program starts, ie I can open using the spanner, and have four tabs - datamine, graph, config and advanced. If I click on graph it sits there with the "thinking" icon and nothing happens. Clicking on config it shows the various devices, although lacks the specific info that you have showing in green on your http://code.mios.com/trac/mios_datamine page. Then I can click back into graph and it shows all the menu items, but clicking "select channels" brings up the yellow box has no data in it.

So what seems to be happening is the program runs, but it is not pulling down any data to display.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 12, 2011, 08:44:03 am
Quote
OK - this might be progress - there is lzo appended on the file in both locations - as is the case with the other files in each directory. Are you saying it should be without the lzo?
On my system, there is no lzo on the LUA files - it is appended to the other files though (i.e. the XML/json files). I don't know the mechanism that the files end up with LZO - it's a compression system I think, so I guess somewhere in the Vera code, it does this, but I don't know why LUA files don't have this. We'd need MCV to comment on this (I've asked them to take a look at this thread).
You could try renaming the file to remove the LZO, but I think this won't work. Alternatively (possibly better) is to upload a copy of the file into the /etc/cmh-ludl directory using WinSCP and see what happens.

I assume you're still getting the error with the continuous "Error during LUA startup" message?

Quote
I can open using the spanner, and have four tabs - datamine, graph, config and advanced
This at least means it's loading the other files ok. This stuff is defined in the JSON file.

Quote
Then I can click back into graph and it shows all the menu items, but clicking "select channels" brings up the yellow box has no data in it.
The yellow box etc is generated by the javascript, so again, this file is being read. The population of the list requires the LUA code to be running.

Quote
I also ran a search for the file and it only sits in .etc/cmh-ludl/ and .overlay/etc/cmh-ludl/ with content appearing to be exactly the same in both cases.
Just a comment on this - I think that there's only 1 file, but Linux has the ability to link/map directorys into multiple places so that they appear twice (or more). I haven't tried to confirm this, but I think MCV has a single directory (possibly the overlay diectory) where they store all the Vera specific stuff, and they then map it across to the appropriate place on startup. (that's just an educated guess - not confirmed). I think when you upload a file to one place, you'll find it in both...



Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 12, 2011, 09:06:57 am
Quote
On my system, there is no lzo on the LUA files - it is appended to the other files though (i.e. the XML/json files). I don't know the mechanism that the files end up with LZO - it's a compression system I think, so I guess somewhere in the Vera code, it does this, but I don't know why LUA files don't have this. We'd need MCV to comment on this (I've asked them to take a look at this thread).
You could try renaming the file to remove the LZO, but I think this won't work. Alternatively (possibly better) is to upload a copy of the file into the /etc/cmh-ludl directory using WinSCP and see what happens.
................................
I assume you're still getting the error with the continuous "Error during LUA startup" message?

I've loaded a copy of the file into each directory, and refreshed, which has got rid of the Error during LUA startup message - which seems to have worked.  :D

Quote
The yellow box etc is generated by the javascript, so again, this file is being read. The population of the list requires the LUA code to be running.

So assuming we're up and running, does data show straight away, or is it after a period of time?

Quote
Just a comment on this - I think that there's only 1 file, but Linux has the ability to link/map directorys into multiple places so that they appear twice (or more). I haven't tried to confirm this, but I think MCV has a single directory (possibly the overlay diectory) where they store all the Vera specific stuff, and they then map it across to the appropriate place on startup. (that's just an educated guess - not confirmed). I think when you upload a file to one place, you'll find it in both...

OK - I just refreshed both directories in WinSCP and files are still there, same size as when I pasted them in.

Thanks, Chris, I appreciate you guiding me thru this. It's great to have the forum, and help within it - I had a houseful of x10 gear a few years ago and there was nothing like this. Thanks again  :)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 12, 2011, 09:52:09 am
Quote
I've loaded a copy of the file into each directory, and refreshed, which has got rid of the Error during LUA startup message - which seems to have worked.  :D
That's great news - I'm now interested to find out why your system seems to have added the lzo to the files where mine didn't! I'd have to assume that the other user who had the same error was also due to this. Hopefully MCV can comment.

Quote
So assuming we're up and running, does data show straight away, or is it after a period of time?
You should now be able to go into the config tab, and select some variables to monitor. When you click on them, they should go green which means that the LUA file is really running. Also, when you do this you should get the dataMineConfig.json file written into the /dataMine directory.
With that done, you will be recording data every time the variable changes, and you should get data files written into the /dataMine directory, and you should be able to graph them out. If you record something like a lightswitch status or power level, then you can turn it on and off a few times, and you should be able to confirm everything is working.

Quote
Thanks, Chris, I appreciate you guiding me thru this
No problem - it looks like we're nearly there, and it's always interesting to get to the bottom of a problem.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 12, 2011, 06:22:00 pm
Quote
That's great news - I'm now interested to find out why your system seems to have added the lzo to the files where mine didn't! I'd have to assume that the other user who had the same error was also due to this. Hopefully MCV can comment.

Yes, seems strange - I know I (eventually) followed the directions to the tee - I'm the one in the family who reads instructions on all things!. The lzo issue may well be the same for the lost member, too.

Quote
You should now be able to go into the config tab, and select some variables to monitor. When you click on them, they should go green which means that the LUA file is really running. Also, when you do this you should get the dataMineConfig.json file written into the /dataMine directory.
With that done, you will be recording data every time the variable changes, and you should get data files written into the /dataMine directory, and you should be able to graph them out. If you record something like a lightswitch status or power level, then you can turn it on and off a few times, and you should be able to confirm everything is working.

Ahaa - that, too is working - variables selected, all good. It didn't seem to kick in straight away last night but all good this morning.

Quote
No problem - it looks like we're nearly there, and it's always interesting to get to the bottom of a problem.

True - as I said earlier, big learning curve for me - and from what I've read, you're only new-ish to the system/code too - great work, Chris - this is a really useful group of functions! Hopefully I can reduce our power bill to a more reasonable level.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 13, 2011, 08:50:26 am
Hi Chris. Thanks for the hard work with your dataMine code. I installed today - clear instructions and all v. straightforward, thanks for that...

For starters I'm monitoring a Horstmann HRT4-ZW and ASR-ZW thermostat and boiler actuator.

I have logging enabled for 3 channels - thermostat setpoint and current temperature, and boiler actuator operating mode.

Having changed setpoints / increased and decreased temp. I can see the log files being created and have a noddy graph :) :)

Q. However one problem I have is that the boiler actuation mode is logged textually -

Code: [Select]
1321190664,HeatOn
1321190884,Off
1321191080,HeatOn
1321191097,Off
1321191129,HeatOn
1321191528,Off

This seems to be playing havoc with the axis limits when I try to graph it.

I'd also like to be able to override the x,y axis limits as currently it seems to be auto-generating them between the minimum and maximum logged values?

Any thoughts on how to deal with this?

Thanks!

Alex
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on November 14, 2011, 05:36:28 am
@willstar

What firmware version do you have?
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 14, 2011, 07:47:39 am
@willstar

What firmware version do you have?

It tells me it's the latest version, 1.1.1350 .
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on November 14, 2011, 08:07:32 am
There were some changes in our internal repository, and somehow some builds used the cgi-bin files (like the one that uploads the files on Vera) from the UI5 repository. 1350 is one of the builds affected. The only difference between these files is that the UI4 file doesn't compress the Lua files, but the UI5 file does.

To fix this, SSH into Vera and delete /www/cgi-bin/cmh/upload_upnp_file.sh. Then, using a program like WinSCP, put the attached file in its place.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 14, 2011, 08:23:41 am
There were some changes in our internal repository, and somehow some builds used the cgi-bin files (like the one that uploads the files on Vera) from the UI5 repository. 1350 is one of the builds affected. The only difference between these files is that the UI4 file doesn't compress the Lua files, but the UI5 file does.

To fix this, SSH into Vera and delete /www/cgi-bin/cmh/upload_upnp_file.sh. Then, using a program like WinSCP, put the attached file in its place.

OK, done - thanks for your help on this
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 16, 2011, 04:49:25 am
Quote
Q. However one problem I have is that the boiler actuation mode is logged textually

For anybody who runs into this issue, I was using HVAC_UserOperatingMode1 with ModeStatus. If instead I use SwitchPower1 with Status I can retrieve a 0,1 boolean value which gives me the boiler heating status...
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 16, 2011, 08:06:49 am
You could transform a non-numerical UPnP variable to a numerical one by luup.variable_watching the non-numerical UPnP variable and creating/setting a user defined numerical variable - but I'm not sure whether the dataMine plugin would list user defined variables that are not part of an UPnP device specification.

The GDL/GDP plugin does support non-numerical variables.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 16, 2011, 08:12:58 am

Thanks Ap15e. That's helpful to know.
Title: Re: New dataMine graphing and logging plugin
Post by: utz on November 16, 2011, 08:22:51 am
but I'm not sure whether the dataMine plugin would list user defined variables that are not part of an UPnP device specification.

it does (if I understand correctly). I monitor variables that I put in as a hack which are not properly part of the specification; but they are listed and it works.
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 16, 2011, 03:40:36 pm
Code: [Select]
local MY_NUMERICAL_VARIABLES_SID = 'urn:my-numerical-variables'
local HVACO_SID                  = 'urn:upnp-org:serviceId:HVAC_UserOperatingMode1'
local HVACO_DEVICE_ID            = 11

luup.variable_watch( 'alphanumerical2numerical', HVACO_SID, nil, HVACO_DEVICE_ID )

function alphanumerical2numerical( lul_device, lul_service, lul_variable, lul_value_old, lul_value_new )

 if ( lul_service == HVACO_SID ) and ( lul_variable == 'ModeStatus' )
  then

   if lul_value_new == 'HeatOn'
    then
     luup.variable_set( MY_NUMERICAL_VARIABLES_SID, tostring( lul_variable ) .. 'Numerical', '1', lul_device )
   elseif lul_value_new == 'Off'
    then
     luup.variable_set( MY_NUMERICAL_VARIABLES_SID, tostring( lul_variable ) .. 'Numerical', '0', lul_device )
   end

  end

end
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 17, 2011, 08:50:00 am
Oh cr@p - dataMine Startup LUA failed message is up again. It did seem to coincide with my installing another half a dozen appliance switches this evening and then trying to add them in the confiig tab. Will have to go thru the previous messages again to rectify - although did just swap out the L_DataMine1.lua file again to see if it sorted the message out - but no go.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 17, 2011, 08:55:08 am

Something I've noticed... every time I go to the the dataMine graphing tab the first time it fails to load. When I go to another tab and back to the graphing tab it's fine. Any ideas?
Title: Re: New dataMine graphing and logging plugin
Post by: utz on November 17, 2011, 09:09:23 am

Something I've noticed... every time I go to the the dataMine graphing tab the first time it fails to load. When I go to another tab and back to the graphing tab it's fine. Any ideas?

yes, a known bug in UI4. Happens with all js tabs.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 17, 2011, 09:12:58 am

Ah thanks. I've seen some comments about UI5 in passing. Is that still in beta?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 19, 2011, 09:07:37 am
Appologies for not being too active here over the past week - I'm travelling a lot with work at the moment.

I've uploaded a new version of the Flotr libraries into the repository (http://code.mios.com/trac/mios_datamine). This fixes an anoying bug with the mouse tracking which meant the wrong point was listed if the background screen was scrolled down from the top.

I'm currently looking at a better way to graph binary status varibles by putting all binaries onto a single axis and vertically aligning them. This seems to be the most requested issue at the moment.

The other issue I want to look at is graphing of non-numeric data. This is more difficult since the Flotr package doesn't support non-numerics. This means somewhere converting the data into a numerical value that the user can understand... I'm open to suggestions on how to do this - conceptually, it's not difficult to code something, but anything that's done at graph time is going to slow down the response which will make the system less usable...
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 19, 2011, 11:48:25 am
What about a simple mapping file?

<service id>,<variable name>,<value>,<new numerical value>

Display (the list of) <value> - <new numerical value> in the legend.
Log  <value> and <new numerical value>.
If a user doesn't like your default mapping, the user could change the mapping in the configuration file.

... and provide a method for graphing alphanumerical data that cannot be mapped to a discrete number of numerical values (e.g., earthquake region, weather condition, ...).  :)
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on November 22, 2011, 04:14:01 am
@Chris: Everything working fine here on Vera2 Firmware 1.1.1338. Thank you for this absolutely great plugin!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 25, 2011, 02:51:05 pm
I've uploaded a new version to the repository. The L_DataMine.LUA and J_DataMine.JS files have been updated to add an offset when graphing multiple binary variables on a single channel.  I haven't completely tested this so any comments welcome.

I've also put an offset into the vertical axis to move traces away from the top/bottom of the chart as this often ended up with parts of the plot missing where it ran across the top of the plot. Again, any comments welcome about this.
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 26, 2011, 05:48:50 am
Chris, repository as in here?  http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine)  I just couldn't get/see the updated files if this is the location.
Cheers,
David

I've uploaded a new version to the repository.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 06:05:21 am
Help Please :-)

I am setting up Datamine and it's a big learning curve for me

As far as I can tell all the files are in the right place

DataMine seems to start OK

I can select the channels I want to log and they turn green

It says "recording 4 channels"

Come back to it later and it's now saying


Total Channels 0   
Channels Recording 0   


Go back into setup and the config tab and they are blank again, the channels are there but the green has gone

Top of the dashboard page is

dataMine[50]:
Running Lua Startup

which does not go away

The /etc/cmh-ludl/ directory shows all the files there with Izo on the end apart from the L_datamine1.lua file

I saw the bits on Izo earlier in the thread but am not sure what I am meant to do about it!

Any help appreciated

Thanks

Nick

Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 08:52:00 am
Following up on myself I've copied the log lines relating to DaraMine startup in case they give you some clues what is happening. Something is unhappy. Lines pasted at end of this message.

From the instructions I had trouble in this area

Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

mkdir /dataMine
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine
Modify the /etc/init.d.custom-user-startup file to add the line -:

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

That init.d.custom file didn't seem to be there

Any help very welcome

thanks

Nick




11/26/11 10:46:27.139   luup_log:50: dataMine: Initialising Data Mining System <0x402>
06   11/26/11 10:46:27.141   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: DataDirectory was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402>
50   11/26/11 10:46:27.142   luup_log:50: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>
06   11/26/11 10:46:27.143   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0x825aa0/NONE duplicate:0 <0x402>
06   11/26/11 10:46:27.145   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402>
01   11/26/11 10:46:27.147   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 09:39:55 am
Are the log entries you've pasted from the first time you ran dataMine, or does this same thing get logged each time? It looks like it's from the first time it was run as it's setting up some variables that should only get set the first time (ie. the data directory).

The fact that you got the "logging 4 variables" status shows that it was possible for it to write the config file. However, if the config file was located somewhere that didn't get remapped if Vera restarted, then this may explain why the status went back to 0 logged varibles.

Quote
01   11/26/11 10:46:27.147   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>
This message is interesting - I don't know what it means as it's coming from the lua engine, but it doesn't look good!

Quote
That init.d.custom file didn't seem to be there
That's actually quite normal - you should just be able to create the file and put the appropriate line into it.

Quote
I saw the bits on Izo earlier in the thread but am not sure what I am meant to do about it!
Nothing - it looks like you've got the correct filenames otherwise you'd have a different problem.

If you look in the /dataMine directory, is there anything in there? Does it map to the USB stick? I think you should be able to remove the USB stick and put it into your PC to see the files - I've not tried this myself though!

It seems that all the main files are in the right place or you would get as far as you are, so I'd suggest looking at the USB stick and the mapping of the directory.

I've added a couple of new messages into the startup function. I'm not sure they will help, but they might. Can you grab the latest L_DataMine1.lua file and load this to your Vera and take a look at the logs.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 10:00:07 am
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September

Am I missing something about getting the new file?

Struggling here, out of my depth rather with Linux.

THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening

How can I tell if it's writing to that rather than Vera's flash? I don't know enough about what I am looking for in Win Scp

Those log file extracts were a while back, I'll try and get some more

It seems that it thinks it's recording data until I press the refresh button on the dashboard display and then it goes away and shows zero again

Something is happening to show the items in the config file, for them to change colour, and for the same ones to show in the graph tab even though I can't seem to get graphs out of it

Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??

Nearly there I am sure but it's finding that last bit

Thanks for your help, much appreciated

Nick



Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 10:19:42 am
Quote
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September
Sorry - I didn't update the ZIP - just the code repository. Take a look here -: http://code.mios.com/trac/mios_datamine/browser

Quote
THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening
Ok - that's probably a good indicator that it's going to the USB stick...

Quote
Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??
Yes - the config file is the dataMineConfig.json file, and it should be in the same directory as the data is stored (ie. /dataMine). Is this file in the directory?

More log file dumps would be good...
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 11:18:36 am
Quote
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September
Sorry - I didn't update the ZIP - just the code repository. Take a look here -: http://code.mios.com/trac/mios_datamine/browser


Missing knowledge my end, have downloaded the new files from there but having trouble putting them into Vera with the developers route. Tried it in Safari and Chrome and it's not playing. Put them there in Safari in the first place.

Quote
THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening

Ok - that's probably a good indicator that it's going to the USB stick...



Something is, might be Vera logs rather than Datamine ones though.

How do I tell what is the stick and what is the Vera with WinSCp?


Quote
Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??
Yes - the config file is the dataMineConfig.json file, and it should be in the same directory as the data is stored (ie. /dataMine). Is this file in the directory?

/what/datamine???

I see /overlay/www/cmh/plugins/datamine

that's got all the PNG files etc in it but not the dataMineConfig.json file

the LuaUPnP file time is the same as the one I sent the extract from before

I'm struggling here

Nick

Done a file search on Vera with Win Scp and not found it anywhere, seems to suggest it can't find where to write things



More log file dumps would be good...
[/quote]
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 11:53:39 am
Quote
/what/datamine???

I see /overlay/www/cmh/plugins/datamine
The /overlay directory is holding the extra plugin files. What you should find in WinSCP is a directory called /dataMine - i.e. from the root directory.  This is the directory that ought to be linked into the USB drive - if this directory doesn't exist, then you're not going to get too far, and this would probably explain at least some of what's going on (such as your "logging 4 channels" going to 0).


Quote
How do I tell what is the stick and what is the Vera with WinSCp?
You can't directly see this in WinSCP - Linux makes things nice and seemless...
The command "mount -v" will do what you want (I think). You can run this command by using the WinSCP terminal (under Commmands | Open Terminal menu).

When I did this it actually showed that my system isn't linking to my USB drive, so I need to look further into this. This means everything is going into the Vera drive, which will ultimately fill up...  I'll take a look at this and try and work out what's happening - it did used to work, so presumably it means the USB isn't getting mounted on startup.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 01:29:26 pm
Quote
/what/datamine???

I see /overlay/www/cmh/plugins/datamine
The /overlay directory is holding the extra plugin files. What you should find in WinSCP is a directory called /dataMine - i.e. from the root directory.  This is the directory that ought to be linked into the USB drive - if this directory doesn't exist, then you're not going to get too far, and this would probably explain at least some of what's going on (such as your "logging 4 channels" going to 0).

Had failed to create that, that explains a lot

Have now created it and can't tell if it is mounted on the USB or it's on Vera.


Quote
How do I tell what is the stick and what is the Vera with WinSCp?
You can't directly see this in WinSCP - Linux makes things nice and seemless...
The command "mount -v" will do what you want (I think). You can run this command by using the WinSCP terminal (under Commmands | Open Terminal menu).

Tried the mount command, don't know if it worked

When I did this it actually showed that my system isn't linking to my USB drive, so I need to look further into this. This means everything is going into the Vera drive, which will ultimately fill up...  I'll take a look at this and try and work out what's happening - it did used to work, so presumably it means the USB isn't getting mounted on startup.

Not just me then :)

I now have it claiming to be logging and staying there when I press save instead of losing it, having somewhere to save it helps!

Not getting graphs yes, there is a yellow box that appears when I go to the graph tab and the channels that I selected on the config tab are there but I can't click anything that makes a graph appear

I can't seem to select logging interval in the config tab either.

The /datamine directory has 8 files in it and a "lost and found" directory which all looks like the other log location was

LuaUPnP.log is there dated a few mins ago

Looks like a file is being created with one of the devices I am logging as a file name, that's a good sign, there are a few pairs of values in it as well.

Still don't seem to be able to draw a graph though

Definate progress though, it was me failing to create the output directory that was causing a lot of the trouble

I still have

dataMine[50]:
Running Lua Startup

top right of the dashboard, something is unhappy

Definately nearer though

Nick

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 01:39:19 pm
Quote
Not just me then :)
No - I've been working through this, and it's not quite as simple as I'd thought to get the custom-user-startup file to run on startup - I'll write that up in a separate post in a minute...

Quote
I can't seem to select logging interval in the config tab either.
You can't select logging interval - dataMine will log data when it changes - not at a specific intrval. So, you won't have any data logged until it has changed - if you're logging a switch status, then flick them on/off a few times, and you should get data logged.

Quote
I still have

dataMine[50]:
Running Lua Startup

top right of the dashboard, something is unhappy
Can you take a look in the log file and send me any entries when dataMine starts - there must be something wrong, and the log is the best (only?) place to work out what.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 01:49:56 pm
I've recently noticed that the custom-user-startup wasn't being executed on startup, and my dataMine data was being logged into the local Vera drive. This isn't good since a) it's not especially big, and b) it probably has limited write cycles.

So, after a lot of chasing, I've worked out that 2 more commands are required.
Code: [Select]
chmod +x /etc/init.d/custom-user-startup
/etc/init.d/custom-user-startup enable

The first command (chmod) sets the file permissions to make the file executable. I'm not 100% sure this is required, but I think it is.

The second command makes a symbolic link in the /etc/rc.d directory. This is what actually gets the file to run when Vera starts.

When Vera is running, you can check that the USB stick is mounted and mapped to the /dataMine directory by running the command -:
Code: [Select]
mount -v
On my system, the following is shown -:
Code: [Select]
/dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine type vfat (rw)
One point to note is that if you find you also have this issue, you should copy all the files from your dataMine directory onto your PC before you remap the data to the USB stick. Otherwise, you'll loose access to the files since the USB stick will be mapped over the other files.

(I should add, I'm not a Linux expert so if there's someone out there who knows more about this than me, feel free to comment)
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 01:53:21 pm
Looking at Log file


11/26/11 18:10:10.592   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/26/11 18:10:10.726   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 18:10:10.727   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
06   11/26/11 18:10:10.728   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 2 #hooks: 0 upnp: 0 v:0x825c78/NONE duplicate:1 <0x402>
06   11/26/11 18:10:10.729   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 2 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
01   11/26/11 18:10:10.742   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>


Hopefully that means something to you

It is logging time and data for the items I am logging, just don't seem to have the graphing in there yet

Will look at the data mount stuff later, get it working first.

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 02:28:24 pm
So, this has the same error as in your previous log. It's also not logging the extra stuff I added earlier today, which is a "startup complete" message.

The only thing I can think of is that some of your other files are out of date (maybe the I_DataMine1.xml file). Can you make sure the latest files are loaded from the repository. 
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 02:49:42 pm
I hadn't updated the file at that stage, remember it was failing to upload via the developers option then.

Screenshot of my version of the graph tab attached

I have just SCPd it and will try again

OOps new message

Startup Lua failed

Graph tab now blank

Config Selections not coloured

Looks like there is a Lua crash report in there, can I send you the whole log file by e-mail, it is 2.5 meg

Looks like changing the Lua  file and the two Flotr files that I have just updated has crashed it somewhere, not sure how to restart it. The log extract below seems to be the crash

Nick (out of his depth)



2011-11-26 19:25:57 - LuaUPnP Terminated with Exit Code: 245



2011-11-26 19:25:57 - LuaUPnP crash

01 2011-11-26 17:53:7 caught signal 11 <0x400>
01 2011-11-26 18:9:55 caught signal 11 <0x400>
01 2011-11-26 18:10:29 caught signal 11 <0x400>
01 2011-11-26 18:12:10 caught signal 11 <0x400>
01 2011-11-26 18:30:40 caught signal 11 <0x400>
01 2011-11-26 18:41:40 caught signal 11 <0x400>
01 2011-11-26 18:43:55 caught signal 11 <0x400>
01 2011-11-26 18:58:24 caught signal 11 <0x400>
01 2011-11-26 19:22:55 caught signal 11 <0x400>
01 2011-11-26 19:25:57 caught signal 11 <0x400>
---------------exited-------------
  PID USER       VSZ STAT COMMAND
    1 root      1444 S    init       
    2 root         0 SW   [keventd]
    3 root         0 SWN  [ksoftirqd_CPU0]
    4 root         0 SW   [kswapd]
    5 root         0 SW   [bdflush]
    6 root         0 SW   [kupdated]
    8 root         0 SW   [mtdblockd]
   95 root         0 SWN  [jffs2_gcd_mtd4]
  119 root      1444 S    init       
  157 root      1692 S    syslogd -C256
  159 root      1436 S    klogd
  172 root       708 S    /sbin/hotplug2 --override --persistent --set-worker /lib/hotplug2/worker_fork.so --set-rules-file /etc/hot
  313 root         0 SW   [khubd]
  523 root         0 SW   [usb-storage-0]
  524 root         0 SW   [scsi_eh_0]
  593 root         0 SW   [kjournald]
  624 root         0 SW   [loop0]
  942 root      1456 S    crond -c /etc/crontabs -l 5
  948 root      1120 S    /usr/s

Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 03:17:44 pm
Still tinkering

Changed "L_DataMine1.lua" back to the original and it now gives me

dataMine[50]:
Running Lua Startup

rather than the other error and the devices are back in the yellow box again on the graph tab. Still no graph though

Hope this makes some sense

Log file shows at the time I swapped the file back and pressed refresh

11/26/11 20:01:22.077   luup_log:50: dataMine: Initialising Data Mining System <0x402>
02   11/26/11 20:01:22.080   ZWaveJobHandler::SetDeviceStatus XXX2 0/0/0/0 <0x803>
50   11/26/11 20:01:22.683   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 20:01:22.684   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/26/11 20:01:22.685   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 20:01:22.686   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/26/11 20:01:22.687   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/26/11 20:01:22.688   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/26/11 20:01:22.689   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
06   11/26/11 20:01:22.690   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 7 now: 7 #hooks: 0 upnp: 0 v:0x827238/NONE duplicate:1 <0x402>
06   11/26/11 20:01:22.691   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 7 now: 7 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
01   11/26/11 20:01:22.692   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>

Hope this makes sense??

Nick


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 05:34:23 pm
One thing that I note in your picture is that you don't have any of the icons on the graph page - they are just showing up as some sort of default icon (at least that's how it looks). If that's the case, then the other problem may be that the other files (PNG, JS etc) aren't in the correct location /www/cmh/plugins/dataMine. If this wasn't correct, then you won't get a graph.
What browser are you using? I've tried the libraries on IE and Firefox, so there's a (small) possibility that there's an issue here as well.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 26, 2011, 05:49:27 pm
I've now got a version of dataMine working that will graph "non numerical" data. I don't have a lot of this sort of data, so I'd like a bit of feedback before I release it to the repository. (refer to the files attached to this post).

Below is an example of what it looks like. It's worth noting that dataMine builds up a lookup table between the text and numbers, so it may do some strange stuff if you try and graph data that hasn't fully built up the lookup. At some stage, I'll post an explanation of how to change the default values, but for now I just want to see if it works.

So, if you have this sort of data, and are feeling game, then feel free to give this a go - I welcome any feedback.  One issue I have noticed is that if you try and plot this sort of data on the right hand y axis, it doesn't work. I don't know if this is something I'm doing wrong, or if it's a limitation of the library - I'll work this out over the next few days.

<< There's an error in this version as it doesn't correctly change graph modes. I don't seem to be able to remove the zip file below - I'll load another version in a new post >>
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 26, 2011, 06:05:14 pm
One thing that I note in your picture is that you don't have any of the icons on the graph page - they are just showing up as some sort of default icon (at least that's how it looks). If that's the case, then the other problem may be that the other files (PNG, JS etc) aren't in the correct location /www/cmh/plugins/dataMine. If this wasn't correct, then you won't get a graph.

Spotted one error on my part, the lack of a capital M in the directory name. I have changed it now and still get the same results.

How do I force things to restart, does the refresh symbol on the dashboard work for that???

What browser are you using? I've tried the libraries on IE and Firefox, so there's a (small) possibility that there's an issue here as well.

Using Chrome or Safari, both the same, tried IE6 and the spanner symbol doesn't do anything there, data logging is working nicely now, it's just the graphics missing

More progress, tried Chrome again now the directory name is right and it's giving me the custom symbols, allowing me to select channels, just failing to draw the graph!

Browser comes into it and Safari is bad news, one for the instructions page there, I normally use Chrome anyway, Vera just seemed a little happier with Safari when I was originally setting up and I stuck with it.

I'll try Firefox tomorrow and see if that is any happier drawing graphs, we are very close now, thanks for your patience is talking me through it all

Still got "running Lua Startup" at the top though

Nick



Title: Re: New dataMine graphing and logging plugin
Post by: willstar on November 26, 2011, 09:04:36 pm
I have just uploaded the newer versions of the two files you mentioned, Chris. Have been getting that old "Startup Lua Failed" message again (before I uploaded new files. Interestingly I copied (replaced) the L_DataMine1.lua in etc/cmh-ludl and it shows a date/time stamp of 27/11/11 10.07am when it is 11.56am here - even when I refresh and restart dataMine the file remains unchanged. Also when I checked the log file I cannot find any reference to dataMine at all - strange.

Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)

Wondering if I should just start from scratch again...
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 03:13:37 am
Hi Chris

I tried Firefox, still no graphs but it tried to draw them.

Thinking aloud it's logging data in the /dataMine directory and that data is appearing ok, what I don't see are the "downsampled files" that you talk about. The dataMine directory is shown on the screen shot attached.

I'm sure we are on the home straight now, the data is being captured and the graph plugin is showing, it's just the last bit now

Many thanks for the help and all the good work, it looks just what I hoped Vera would do when I bought it!

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 05:35:55 am
@willstar
Quote
Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)
I'm not sure what's going on - I don't think this is an issue with dataMine itself (although I stand ready to be corrected). I think this must be something strange with the new Vera version you're running or something. I've certainly not had any problems loading updated software through the developers tab - I've not tried it directly with WinSCP.

@nickrwym
Quote
what I don't see are the "downsampled files" that you talk about
No - this isn't actually implemented yet. It's in there, but as I haven't worked out a couple of things, it's currently disabled - I've had more requests for other graph types than the historical data, so that's taken my time so far.

In your graph screen, do you see all the buttons with correct icons in them? The image you sent yesterday didn't look correct.
Are you still getting the error about Lua Startup with the same isue logged in the log file? Either your problem is associated with this (but as most of the software seems to be working, I'm not sure), or the other issue is that the flotr.js files aren't being loaded. These files should be in the same directory as the PNG icons - hence my question about the icons.


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 05:46:24 am
I've found minor a bug in the version of the software that graphs text data. It doesn't correctly set the data into this mode - I'd done this manually during testing, and then forgot to add the functionality.

So, attached is an update - sam caveats apply - the software will need to run for a while to learn the different text data values.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 11:49:26 am
@willstar
Quote
Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)
I'm not sure what's going on - I don't think this is an issue with dataMine itself (although I stand ready to be corrected). I think this must be something strange with the new Vera version you're running or something. I've certainly not had any problems loading updated software through the developers tab - I've not tried it directly with WinSCP.


Win SCP copies it across fine, I was just having trouble loading it with the developer tab. It may have been a browser issue but it worked originally so I don't know.

Putting the old version back in made it work better than the new version so that one is in now.


@nickrwym
Quote
what I don't see are the "downsampled files" that you talk about
No - this isn't actually implemented yet. It's in there, but as I haven't worked out a couple of things, it's currently disabled - I've had more requests for other graph types than the historical data, so that's taken my time so far.

That's why I don't see them then!

In your graph screen, do you see all the buttons with correct icons in them? The image you sent yesterday didn't look correct.

Yes, that was the lack of a capital M in the middle of the directory name, my error
 
Are you still getting the error about Lua Startup with the same isue logged in the log file? Either your problem is associated with this (but as most of the software seems to be working, I'm not sure), or the other issue is that the flotr.js files aren't being loaded. These files should be in the same directory as the PNG icons - hence my question about the icons.

Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen, it's the same in Firefox as Chrome.

The data files are there for each of the items that I have selected to log, both in the pull down list and in the /dataMine directory as  a set of pairs of values, once the graphing is sorted I will be able to look at the data. Looking at it I am not convinced some are updating properly, the energy monitor is though.

Hopefully I am nearly there, still got the "Running Lua Startup" message though

Still not getting graphing but do have all the data selection and logging working.

Nick



Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 01:12:07 pm
Quote
Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen
To me, this sounds like the JS files aren't being loaded. The fact that you get the "wait" indicator, indicates to me that the UI is requesting the data from Vera, and the data is probably being returned, but not graphed.

Do you have FireBug installed? This is a FireFox addon that allows you to see the data flying around on the network between the browser and the Vera. With this, you'd be able to see if Vera/dataMine was returning graph data or not - we'd be able to narrow it down a bit more.

Quote
still got the "Running Lua Startup" message though
I'm really not sure what that's about. I'll take a look at the code and try and add some more debug/log data into the software.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 02:25:42 pm
Nick,
Please load this version of L_DataMine1.lua. It adds some more logging into the startup. The problem I've got is that from the current logging, it looks like everything except the last log message is happening. The last command before this log is to register a handler that handles the registering of variables (i.e. when you click on a variable and it goes green).

I also need to get more log data. I need all the dataMine log entries. You can do this by typing the following command into the terminal window in WinSCP
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
So, load this version, restart Lua, then try to graph something, run the above command and send me the data...
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 02:35:17 pm
Nick,
Please load this version of L_DataMine1.lua. It adds some more logging into the startup. The problem I've got is that from the current logging, it looks like everything except the last log message is happening. The last command before this log is to register a handler that handles the registering of variables (i.e. when you click on a variable and it goes green).

I also need to get more log data. I need all the dataMine log entries. You can do this by typing the following command into the terminal window in WinSCP
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
So, load this version, restart Lua, then try to graph something, run the above command and send me the data...

Thanks for your persistence with this

I uploaded the new file using Winscp, pressed the refresh on the dashboard and still get the starting message

Tried to draw a few graphs and pressed a few of the options on the graph page and grepped the log as requested, log follows


/dataMine$ grep dataMine /tmp/log/cmh/LuaUPnP.log
50   11/27/11 17:57:59.596   luup_log:50: dataMine: ID: 2 Start: 1322330279 Stop: 1322416679 Sample: 123 <0x8d808>
09   11/27/11 19:06:29.416   JobHandler_LuaUPnP::Run device 50 dataMine room 2 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x7e3828 upnp: 0 <0x400>
50   11/27/11 19:06:31.187   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/27/11 19:06:31.918   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:06:31.919   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:06:31.920   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:06:31.921   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:06:31.922   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:06:31.923   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:06:31.924   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
50   11/27/11 19:06:31.926   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[KWH] <0x402>
09   11/27/11 19:29:02.121   JobHandler_LuaUPnP::Run device 50 dataMine room 2 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x7e3618 upnp: 0 <0x400>
50   11/27/11 19:29:03.587   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/27/11 19:29:04.198   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:29:04.200   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:29:04.201   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:29:04.202   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:29:04.204   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:29:04.205   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:29:04.206   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
50   11/27/11 19:29:04.207   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[KWH] <0x402>
50   11/27/11 19:29:04.210   luup_log:50: dataMine: Registering handlers 1 <0x402>
50   11/27/11 19:29:04.211   luup_log:50: dataMine: Registering handlers 2 <0x402>
50   11/27/11 19:29:04.211   luup_log:50: dataMine: Registering handlers 3 <0x402>
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
50   11/27/11 19:29:41.187   luup_log:50: dataMine: ID: 3 Start: 1322335781 Stop: 1322422181 Sample: 123 <0x400d>
50   11/27/11 19:29:57.863   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422197 Sample: 84 <0x3c10>
50   11/27/11 19:29:59.738   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422199 Sample: 84 <0x3c10>
50   11/27/11 19:30:01.338   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422201 Sample: 84 <0x400d>
50   11/27/11 19:30:02.467   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422202 Sample: 84 <0x400d>
50   11/27/11 19:30:03.658   luup_log:50: dataMine: ID: 3 Start: 1322335803 Stop: 1322422203 Sample: 123 <0x3c10>
50   11/27/11 19:30:04.722   luup_log:50: dataMine: ID: 3 Start: 1322335804 Stop: 1322422204 Sample: 123 <0x400d>
50   11/27/11 19:30:05.528   luup_log:50: dataMine: ID: 3 Start: 1322335805 Stop: 1322422205 Sample: 123 <0x400d>
50   11/27/11 19:30:06.313   luup_log:50: dataMine: ID: 3 Start: 1322335806 Stop: 1322422206 Sample: 123 <0x400d>
50   11/27/11 19:30:12.016   luup_log:50: dataMine: ID: 3 Start: 1322335812 Stop: 1322422212 Sample: 123 <0x3c10>
50   11/27/11 19:30:12.022   luup_log:50: dataMine: ID: 4 Start: 1322363013 Stop: 1322422212 Sample: 84 <0x3c10>

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 02:40:39 pm
Quote
Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen
To me, this sounds like the JS files aren't being loaded. The fact that you get the "wait" indicator, indicates to me that the UI is requesting the data from Vera, and the data is probably being returned, but not graphed.

Could be

Do you have FireBug installed? This is a FireFox addon that allows you to see the data flying around on the network between the browser and the Vera. With this, you'd be able to see if Vera/dataMine was returning graph data or not - we'd be able to narrow it down a bit more.

No, I've got wireshark on one of the machines though so I can packet sniff if that helps?

Quote
still got the "Running Lua Startup" message though
I'm really not sure what that's about. I'll take a look at the code and try and add some more debug/log data into the software.

Your help is much appreciated

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 03:18:07 pm
Quote
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
Ok - it gets stranger....
The first line shows that the startup is completing correctly, so I don't know why the "Startup error" message is still there.
The second line shows that the Lua file is receiving requests for graph data - and presumably it's sending it back.

I also added something to display an error if the graph library isn't there, and as that's not displayed, I can rule that out.

Maybe you can use Wireshark to confirm that Vera is returning graph data - it's all text, so it'll be easy to find if you're familiar with Wireshark filters. Maybe there's something funny in the response (he says clutching at straws).

Sorry - I'm not being especially helpful  :(
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 04:22:59 pm
Quote
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
Ok - it gets stranger....
The first line shows that the startup is completing correctly, so I don't know why the "Startup error" message is still there.

The exact message is

dataMine[50]:
Running Lua Startup

It's informative rather than an error??


The second line shows that the Lua file is receiving requests for graph data - and presumably it's sending it back.

Sounds good

I also added something to display an error if the graph library isn't there, and as that's not displayed, I can rule that out.

Sounds good too

Maybe you can use Wireshark to confirm that Vera is returning graph data - it's all text, so it'll be easy to find if you're familiar with Wireshark filters. Maybe there's something funny in the response (he says clutching at straws).

Not very familiar, used it once or twice but not a regular tool

I've packet sniffed a "graph creation attempt" and can see it ask for the data

243   13.608733   192.168.1.104   192.168.1.107   HTTP   639   GET /port_3480/data_request?id=lr_dmData&channel1=2&channel2=5&channel3=0&channel4=0&start=0&stop=0&format=json HTTP/1.1

It then has lots of data flowing down from 107 which is Vera, it seems to fall into groups of 3 with one of these first

244   13.610461   192.168.1.107   192.168.1.104   TCP   60   http > 57781 [ACK] Seq=1 Ack=586 Win=6435 Len=0

followed by 2 of these which I assume are more data associated with the first packet that wouldn't fit into single packets?

300   16.610676   192.168.1.107   192.168.1.104   TCP   1514   [TCP segment of a reassembled PDU]

As I don't know what it looks like when it's working properly this isn't showing me much apart from lots of data flowing when asking for a graph!

Will the firefox extension help more??

Nick



Sorry - I'm not being especially helpful  :(

You are being very helpful by persevering

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 05:28:52 pm
Quote
dataMine[50]:
Running Lua Startup

It's informative rather than an error??
Ah - yes, that's normal - I assume this just flashes up briefly and then goes away...

Quote
Will the firefox extension help more??
Yes, FireBug is very good at displaying this sort of this - it'll display it nicely as text and marries up the request and response so you don't have to work out the "mess" that Wireshark shows.

I still think that there's an issue with the graph library not being loaded - everything else seems to be working. The Wireshark message you showed with the "channel" data is the request from the UI to Vera, and obviously Vera is returning something and I can't see why it would be incorrect. The only other suggestion I can make is to make sure the L_DataMine1.js file is up to date.

Again, FireBug will be able to confirm any issues in the UI quite quickly as it will show any javascrip errors (such as a function not being loaded).
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 27, 2011, 06:22:54 pm
Quote
dataMine[50]:
Running Lua Startup

It's informative rather than an error??
Ah - yes, that's normal - I assume this just flashes up briefly and then goes away...

It stays there permanently

Quote
Will the firefox extension help more??
Yes, FireBug is very good at displaying this sort of this - it'll display it nicely as text and marries up the request and response so you don't have to work out the "mess" that Wireshark shows.

Downloaded and installed, what settings do I need to capture what we need to see??

I still think that there's an issue with the graph library not being loaded - everything else seems to be working. The Wireshark message you showed with the "channel" data is the request from the UI to Vera, and obviously Vera is returning something and I can't see why it would be incorrect. The only other suggestion I can make is to make sure the L_DataMine1.js file is up to date.

Again, FireBug will be able to confirm any issues in the UI quite quickly as it will show any javascrip errors (such as a function not being loaded).

Sounds good, give me a clue as to what parameters we want to look at and I'll try a capture

Just uploaded J_Datamine1.js, fixed the upload via developers fault which occurred when I made the change suggested in reply 67 from Florin, reverted back to the original file and can upload again

I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???

Just reuploaded all the "developer tab" files and it made no difference, it tries to draw a graph but there is nothing there, just blank white once the timer goes.

So near but so far :-(

Nick

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 27, 2011, 06:35:32 pm
Quote
Downloaded and installed, what settings do I need to capture what we need to see??
There's a "Firebug" button down in the bottom right corner of Firefox - press this.

To view all the traffic between the UI and Vera, click on the "Console" tab. Open up data mine, select a channel to graph, and look in the console window for something that looks like this
Code: [Select]
GET http://192.168.2.232/port_3480/data_request?id=lr_dmData&channel1=1&channel2=0&channel3=0&channel4=0&start=0&stop=0&format=json
I want to see the response which will be displayed under this line if you click on it. It would be best to graph something that doesn't have too much data!

The other thing that you might see in the bottom right corner (in red) is an error message. If there are any script errors, then Firebug may report "1 Error". If you click on this, I think it will tell you the line number etc.

Let me know what you find with this and we'll see where we go next.

Quote
I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???
Yes - sorry, that was a typo.

Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 28, 2011, 07:57:15 am

Hi, I'm a bit confused as to why the temperature data int the graph isn't being sub-sampled. Is it something to do with the interval?

Thanks, Alex
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 28, 2011, 08:14:23 am
Quote
I'm a bit confused as to why the temperature data int the graph isn't being sub-sampled.
What makes you think it's being subsampled? DataMine can subsample data, but I don't think that's happening here. The only way to tell is to look at the raw TXT files in the dataMine directory - the first parameter is in seconds, so if there's a big jump between subsequent lines, then dataMine is graphing what is recorded.

What sort of sensor is this coming from? I am using a couple of the ST814 (I think that's the model) Temp/Humidity sensors, and this is about what I see from them. They only send out updates occasionally, and for humidity, only when there is a 5% change (I think it's 1 deg for temperature).

I am starting to use the One Wire sensors from EDS as they aren't bound by various limitations in Vera (e.g. the artificial 1 degree resolution that Vera imposes) and the plots are much nicer.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 28, 2011, 08:17:24 am
What sort of sensor is this coming from?

Rather simplistically I was expecting curves :) Yes it's a thermostat which is sending temperature sensor readings intermittently with a granularity of 1°C.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 28, 2011, 09:28:10 am
Quote
Rather simplistically I was expecting curves
Ah - ok, that's something different. This is unfortunately the problem with Vera only providing sensor data at low resolution. Currently, dataMine assumes that the data stays constant between changes - this is effectively what Vera will show in the UI. It was necessary to do this in order to properly display status values (e.g. light switch on or off), but it's not (necessarily) correct for sensors.

One of the next things on my "todo" list is to change this. I want to add extra configuration parameters in the config window to allow things like this to be set on a per channel basis. It still won't be curves (at least not initially), but it won't look like the Manhattan skyline.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on November 28, 2011, 09:36:42 am
Quote
Rather simplistically I was expecting curves
It still won't be curves (at least not initially), but it won't look like the Manhattan skyline.

Great stuff :)
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 28, 2011, 10:21:31 am
Quote
Downloaded and installed, what settings do I need to capture what we need to see??
There's a "Firebug" button down in the bottom right corner of Firefox - press this.

To view all the traffic between the UI and Vera, click on the "Console" tab. Open up data mine, select a channel to graph, and look in the console window for something that looks like this
Code: [Select]
GET http://192.168.2.232/port_3480/data_request?id=lr_dmData&channel1=1&channel2=0&channel3=0&channel4=0&start=0&stop=0&format=json
Lots of traffic, various errors which I don't think are DataMine, and very little traffic when asking for a graph


I want to see the response which will be displayed under this line if you click on it. It would be best to graph something that doesn't have too much data!

More to follow later when I have more time

The other thing that you might see in the bottom right corner (in red) is an error message. If there are any script errors, then Firebug may report "1 Error". If you click on this, I think it will tell you the line number etc.

No error there

Let me know what you find with this and we'll see where we go next.

Quote
I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???
Yes - sorry, that was a typo.

Good, means I am starting to get to know what is there!

I had a major cockup last night, I replaced all the MIOS developer files with the latest ones, went to bed and checked it in the morning.

The /DataMine directory was empty, whatever I had done had broken it completely and it had deleted all the history that had been building up.

I went back and replaced these files with an older version and got it working again, data is logging now.

Question, is there a copy of a set of the Flotr files that definately work in a place I can get to them, I took them from the link that was on the original install hence they were not from your collection.

Can you put a copy in the depositry so I can get known good ones??

Somewhere among the 6 files that go into the plugins directory there is a faulty file in one of the various incarnations of the program, that is what broke things last night for me. Luckily I had all the various options saved and was able to rebuild.

My gut feel is t replace the Flotr files with known good ones from an installation that is drawing graphs OK



Nick
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 28, 2011, 10:54:20 am
Changed the Flotr files for different ones, the ones I had in before were

flotr2.min.js  size 1,296,520.

I have replaced it with one of 118,430

A vastly different size

THe other file, flotr2.ie.min.js

was 216,566, is now 20,997 in the replacement

Running an attempt at graphing while Firebug is running comes back with an error

syntax error
http://192.168.1.107/cmh/plugins/dataMine/flotr2.min.js
Line 1

The first few lines of the file from the Firebug window are

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 
 

 
  <!-- get vcs info from config file, and set up i18n -->
  <!-- custom variables. -->


  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7" />
    <link rel="stylesheet" href="/stylesheets/trac_print.css" media="print" type="text/css" charset="utf-8" />
    <!--[if lte IE 6]>
      <link rel="stylesheet" href="/stylesheets/trac_ie.css" type="text/css" charset="utf-8" />
    <![endif]-->
    <title>
      flotr2.min.js in Lib
     â€“ Datamine Graphing and Logging Plugin
    </title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <!--[if IE]><script type="text/javascript">window.location.hash = window.location.hash;</script><![endif]-->

I hope this makes some sense to you!

Nick


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 28, 2011, 12:24:39 pm
It looks like when you copied the file from the repository, you've got some extra stuff for free (i.e. all the html headers). I'll update the ZIP file on the wiki, and I've also attached it here. I'd really like to work out how to link the repository directly to the wiki so you can download a zip of the repository, but I've not managed this yet.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 28, 2011, 12:44:57 pm
Not sure if my last post got through

I have success! I can draw graphs in Firefox!

Many thanks for your persistence, we got there and it looks a very useful plugin.

Information gained along the way, it doesn't work in Chrome

I look forward to trying to make sense of what my heating system achieves and to optimising the controls. Just need to get the Everspring temperature sensor reliable now!

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 28, 2011, 02:02:42 pm
Chris,

Thanks for adding support for non-numerical data. Some examples attached.
Title: Re: New dataMine graphing and logging plugin
Post by: utz on November 28, 2011, 05:33:23 pm
Could you please add the following line to L_DataMine1.lua at around line 150 in function watchVariable:
   
Code: [Select]
       if(logfile == nil) then
luup.log(DATAMINE_LOG_NAME .. "Didn't find variable!")
return
end

-- luup.log(DATAMINE_LOG_NAME .. varRef .. "---" .. varRef.Name)

--> if(varRef.Lookup == nil) then varRef.Lookup = {} end <--

if(tonumber(lul_value_new) == nil) then
if(varRef.Lookup[lul_value_new] == nil) then
local max = 0
for k,v in pairs (varRef.Lookup) do

I was logging this non-numerical data before and after updating the charts cant find the new variable Lookup in the json data and it throws an error. Most people I guess wont come across that as they did not log non-numerical data before your upgrade.

Nice work, I like this graphing feature.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 28, 2011, 05:45:28 pm
Utz - I assume putting it a couple of lines further down is also ok (I can't see a problem with that). It is only required then and I like to keep anything "extra" out of the main code for performance reasons (not that it will make a lot of difference!).
So, I could put it after the following line I think -:
Code: [Select]
if(tonumber(lul_value_new) == nil) then
Title: Re: New dataMine graphing and logging plugin
Post by: utz on November 28, 2011, 05:56:40 pm
Sure, that works too. You know your code better anyway, I stopped thinking about it when it worked ...
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on November 29, 2011, 04:50:02 am
I just tried your new update from the november 28th.

I just replaced all the files I had with the new ones you uploaded. After restarting Luup (or rebooting the Vera) the UI4 says in the upper right corner "dataMine[24]: Startup Lua Failed".

By using http://10.0.0.X/cgi-bin/cmh/log.sh?Device=LuaUPnP during Luup Engine Restart I found the following:

Code: [Select]
01 11/29/11 10:20:31.837 LuaInterface::CallFunction_Startup-1 device 24 function initialise failed error loading module 'L_DataMine1' from file '/etc/cmh-ludl/L_DataMine1.lua':
/etc/cmh-ludl/L_DataMine1.lua:1: unexpected symbol near 'Á' <0x402>
01 11/29/11 10:20:31.838 LuImplementation::StartLua running startup code for 24 I_DataMine1.xml failed <0x402>

Can you maybe help Chris? It looks like an error in L_DataMine1.lua, but I almost can't believe that...


EDIT: Solved by redownloading the file from the repository and reuploading it to the vera again.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 29, 2011, 06:12:07 am
I tend to agree that it looks like an error in L_DataMine1.lua - maybe the file got corrupted somewhere along the way? I interpret the error as an unexpected character at line 1, which would (I think) be consistant with a corrupted file.

In the first instance, I'd check that the file looks ok on your PC, and then try and reload it to Vera using the MiOS Developers | Luup Files tab.
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on November 29, 2011, 06:53:28 am
As you told I redownloaded the file and then reuploaded it to the vera. And it worked, I was sure I tried that before posting, but as it seems I didn't.

Thank you very much for you very quick reply and your help, and again thank you for the plugin!
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on November 29, 2011, 01:59:53 pm
Having trouble getting this setup.
I've tried to read all the posts up to this point , but I must be missing something.
I've downloaded the latest file above dataMine-28-11-11.zip and opened up all of the "Luup Files"
to verify that they look ok and uploaded to vera, I then scp the rest of the files to /www/cmh/plugins/dataMine and /www/cmh/skins/default/icons as specified in the readme
I've setup my usb device using the vera usb log command then created a link in /dataMine
lrwxrwxrwx    1 root     root           21 Oct 22 13:40 dataMine -> /tmp/log/cmh/datamine

as per other posts when setting up the device I changed the devicetype
 from:   DeviceType: urn:cd-jackson-com:device:DataMine1:1
to: urn:schemas-cd-jackson-com:device:DataMine:1

upon first load I see the following in the logs
01 11/29/11 12:37:51.341  LuaInterface::CallFunction_Startup-1 device 91 function initialise failed [string "..."]:7: module 'L_DataMine1' not found:
 no field package.preload['L_DataMine1']
 no file './L_DataMine1.lua'
 no file '/usr/share/lua/L_DataMine1.lua'
 no file '/usr/share/lua/L_DataMine1/init.lua'
 no file '/usr/lib/lua/L_DataMine1.lua'
 no file '/usr/lib/lua/L_DataMine1/init.lua'
 no file '/etc/cmh-ludl/L_DataMine1.lua'
 no file '/etc/cmh-lu/L_DataMine1.lua'
 no file './L_DataMine1.so'
 no file '/usr/lib/lua/L_DataMine1.so'
 no file '/usr/lib/lua/loadall.so' <0x402>
01 11/29/11 12:37:51.343  LuImplementation::StartLua running startup code for 91 I_DataMine1.xml failed <0x402>

I ssh'd in and could not find L_DataMine1.lua in /etc/cmh-ludl all files in the directory are .lzo , so mine is listed as L_DataMine1.lua.lzo
I ran touch L_DataMine1.lua
I restarted and now get the following error
01      11/29/11 12:44:10.280   LuaInterface::CallFunction_Startup-1 device 91 function initialise failed [string "..."]:10: attempt to index global 'dm_plugin' (a boolean value) <0x402>

Can anyone advise as to what I may be missing ?
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 29, 2011, 02:33:40 pm
Quote
I ssh'd in and could not find L_DataMine1.lua in /etc/cmh-ludl all files in the directory are .lzo , so mine is listed as L_DataMine1.lua.lzo
I ran touch L_DataMine1.lua
Can anyone advise as to what I may be missing ?

Having just gone through this long sage I may be able to help

Of the 6 required files that are loaded into /etc/cmh-ludl through the "developers" route 5 end up as .izo and one stays without it

The L_DataMine1.lua does NOT have Izo on it in my working installation.

Try using Win SCP to put an extra copy in without it going through the developers tab, that will mean it does not get converted to Izo.

MCV posted on this on around page 4 of this thread, there were changes within the Vera build that affected how it handled the files, I tried their file modification and it made things worse so the WinSCP route seems safer.

I may be heading off at a tangent and am sure Chris will jump in if I am.

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 29, 2011, 05:41:40 pm
Out of the blue (nothing has been changed):

Code: [Select]
dataMin[47]
Startup Lua Failed

It boils down to:

Code: [Select]
50      11/29/11 23:23:43.325   luup_log:47: dataMine: Initialising Data Mining System <0x402>
01      11/29/11 23:23:44.433   ^[[31;1mLuaInterface::CallFunction_Startup-1 device 47 function initialise failed /usr/lib/lua/json.lua:213: Failed to scan constant from string {"Variables":[{"Type [...]
 at starting position 1907^[[0m <0x402>
01      11/29/11 23:23:44.434   ^[[31;1mLuImplementation::StartLua running startup code for 47 I_DataMine1.xml failed^[[0m <0x402>

dataMineConfig.json:

Code: [Select]
{
            "Type": 0,
            "Device": 4,
            "Service": "urn:upnp-ap15e-com:serviceId:DAD1",
            "Alpha": 1,
            "Name": "DAD - EarthquakeRegion",
            "Archive": "DAD - EarthquakeRegion",
            "FirstRec": 1322409712,
            "Variable": "EarthquakeRegion",
            "Logging": 1,
            "Lookup": {
                "": 3,
                "SOUTHERN GREECE": 11,
                "ST. MARTIN REGION, LEEWARD ISL.": 10,
                "DODECANESE ISLANDS, GREECE": 7,
                "ALBANIA": 8,
                "BOSNIA AND HERZEGOVINA": 2,
                "EASTERN NEW GUINEA REG., P.N.G.": 9,
                "SOUTHERN ITALY": 24,
                "EASTERN TURKEY": 5,
                "PUERTO RICO REGION": 13,
                "LUZON, PHILIPPINES": 22,
                "IONIAN SEA": 4,
                "CZECH REPUBLIC": 23,
                "CANARY ISLANDS, SPAIN REGION": 15,
                "KEPULAUAN SULA, INDONESIA": 21,
                "GREECE": 1,
                "CENTRAL ITALY": 20,
                "CRIMEA REGION, UKRAINE": 18,
                "AEGEAN SEA": 12,
                "CRETE, GREECE": 19,
                "NORTH OF ASCENSION ISLAND": 17,
                "AZORES ISLANDS, PORTUGAL": 6,
                "SERBIA": 16,
                "NEW IRELAND REGION, P.N.G.": 14
            }
        },
        table,
        {
            "Type": 0,
            "Device": 4,
            "Service": "urn:upnp-ap15e-com:serviceId:DAD1",
            "Name": "DAD - DoseRateStationStatus",
            "Archive": "DAD - DoseRateStationStatus",
            "FirstRec": 0,
            "Variable": "DoseRateStationStatus",
            "Logging": 1,
            "Lookup": [
               
            ]
        },

Error:

Code: [Select]
Parse error on line 100:
...        },        table,        {   
---------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on November 29, 2011, 06:09:03 pm
Clearly the word "table" simply shouldn't be there, but I don't understand how it actually got there! In dataMine, I just manipulate the data as a Lua array, and use the json library to write the data to the config file.

I assume this happened "out of the blue" when a new text value was received, and dataMine then updates the config file. When it does this, it generates a backup of the old file (with a .backup extension). So, you can either try removing the word "table", or try the backup. (you've probably already done this).

Of course, of more interest is why it happened, and that I'm not so sure about. I don't know a lot about the Lua language - I'm a little suspicious about its memory management as it's possible to read a whole file into memory and then process it. It may be fine, but coming from a "C" background where you need to take care of memory management yourself, I'm suspicious that somethings not robust. I suspect that this problem is in the json library - not that that's especially useful...

I might take a look at the json library and see if I can spot anything, but I really don't want to get into looking for errors in an existing library (especially since I don't consider myself any sort of expert in Lua!).

Any suggestions from those more knowledgable in the ways of Lua would be most welcome.
Title: Re: New dataMine graphing and logging plugin
Post by: Ap15e on November 29, 2011, 07:07:10 pm
I've attached the last known good configuration and the bad configuration.
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on November 29, 2011, 08:25:53 pm
nickrwym thanks! I scp'd L_DataMine1.lua in and now it works, now I just need to figure out how to setup all I want to track.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on November 30, 2011, 09:19:01 am
Glad to hear it, I'm very pleased with the plugin, makes Vera do what I had hoped it would out of the box.

My OW server has just arrived, more things to play with this evening!

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Qrp on December 04, 2011, 09:32:20 am
Nick -

Are you using the files in the wiki zip  or updated ones?

Is your USB stick getting mounted after a reboot, because mine isn't?

I presume you are using the device type description given by Chris around page 4, and not the one that still appears in the install wiki?

Thanks,
Chip
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on December 04, 2011, 10:04:17 am
I had trouble finding the right files, some worked and some didn't. I ended up with some from the zip and some the code repositry I think but can't remember which were what.

The file sizes of the ones in Vera that work are

in /www/cmh/plugins/dataMine

flotr2.min.js   118,430

flotr2.ie.min  20,997

THose two draw the graphs

The other versions didn't want to work for me

The graphics files that go in there don't stop it working if they are wrong, they just don't look right without the,

Watch out when making the /dataMine directory from the root, there is a capital M in the middle and that matters

in /etc/cmh-ludl

I have the following, 5 are converted into .izo files in the process of being uploaded via the developer tab, 1 isn't and sometimes it gets converted with some builds of Vera and that stops it working. In that case use Win SSP to copy L_DataMine1.lua into the directory

D_DataMine1.json.lzo   939
D_DataMine1.xml.lzo  378
I_DataMine1.xml.lzo 516
J_DataMine1.js.lzo  6,520
L_DataMine1.lua  23,729  (not Izo)
S_DataMine1.xml.lzo  294

I havn't positively checked the USB stick but it's data light is flashing which is a good sign

I used the updated device type

Once it's working there are data files being written in /dataMine that you can see with ssp.

It is browser sensitive, it does not draw graphs in Safari, Chrome, IE6.

It works in Firefox, on an Ipad running IOS 5.01 and on an Iphone running IOS5.

It's an excellent program once you've got it right, well worth the effort.


I'm getting good data on the heating system that I set out to rebuild. Indoor temperature, outdoor temperature, and energy consumed are all being logged and can be easily graphed, I am now working on getting the controls right, I'm planning on pulse width modulating the heating at something like a 30 minute repeat rate with the on time varied between 5 mins on 25 off and on continuously in 5 minute increments. I am finding that outside temperature is a much better indicator of what duty cycle is needed than inside temperature and may well end up with some fairly empirical timings set by outside temperature and use the inside temperature just to validate the result, this will lead to a steadier inside temperature than letting the heating react directly to the inside temperature. The thermal time constant of the house and heating system is much slower than I expected. The rads are still full of hot water after the heat pump shuts off of course and it takes considerable time before it sinks

Early days but looking good so far, time to start writing Luua for a temperature control scene though so been reading into Luua.

Nick




Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 04, 2011, 11:43:05 am
The files in the zip that you can download from the wiki should be fine - there shouldn't be any difference with the ones in the repository (at the moment). I will try and keep the ZIP file on the wiki up to date since it seems that downloading from the repository directly can lead to confusion...

Quote
I presume you are using the device type description given by Chris around page 4, and not the one that still appears in the install wiki?
I have updated the wiki with the extra commands required to get the USB stick to mount properly on startup after I found this wasn't working properly. I think this should be correct - if you think there's something missing, then please let me know and I'll fix it.
Title: Re: New dataMine graphing and logging plugin
Post by: Qrp on December 05, 2011, 09:45:58 pm
I can't get graphs to draw. I've used the files in the current .zip file and the engine starts and seems to run. I can mount the USB stick to /dataMine, and the engine is populating it, but after a reboot that mount is gone, as though the custom-user-startup file is not being executed (though properties appear to be set right).

I can choose parameters to log in the config screen, but when I go to the graph tab I can see text but no icons and nothing will graph. Using Firefox 6.02.

Ideas?

Thanks



Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 06, 2011, 03:33:17 am
I would say that the extra files that need to be in the "plugin" directory (i.e. all the PNG files, and the flotr.*.js files) aren't in the correct place. My reason for saying this is that in your screendump, the buttons at the top of the window don't show the icons and this would cause the problem that you're seeing.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on December 06, 2011, 04:42:55 am
Look out for the capital M in the middle of dataMine, it's needed in the directory name.

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Qrp on December 06, 2011, 11:19:37 am
Still no joy. I've checked and double checked directory names and permissions, file locations and permissions, and made sure all files were from the latest zip.

The engine doesn't seem to be able to load the icon images in plugins, and the custom-user-startup file is not being executed. I can mount the USB stick to /dataMine manually, and the directory is populated, but on a reboot the association is gone again. And of course no graphing occurs.

It does seem that the plugins directory and the custom-user-startup are not being found and used by the engine, but I can't fathom why!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 06, 2011, 12:52:24 pm
In WinSCP, can you click on the TERMINAL button, and run the following command -:
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
This will give the dataMine log and might provide a pointer to what's going on. As Nick pointed out, in Linux, everything is case sensitive, so you need to be take care that nothing changes when you copy files over.

Note that I did find a number of things that stop the USB being remapped on startup - I've add the points to the wiki, but I think the extra commands required are
Code: [Select]
chmod +x /etc/init.d/custom-user-startup
/etc/init.d/custom-user-startup enable

Title: Re: New dataMine graphing and logging plugin
Post by: Qrp on December 06, 2011, 10:18:27 pm
Dataminer seems to be working fine now. There was a space at the end of /www/cmh/plugins/dataMine that was keeping the engine from finding the directory.

However, for some reason \etc\init.d\custom-user-startup is not being executed on reboot. I executed the chmod and enable commands that Chris specified, and they seemed to complete fine. I can execute the file manually from Telnet, and the USB stick will be mounted. But still no execution on a reboot.

What tells OpenWRT to execute this file on boot? Maybe something else is damaged?

Thanks,
Chip
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on December 07, 2011, 02:25:05 am
...However, for some reason \etc\init.d\custom-user-startup is not being executed on reboot...

You need to ad a symbolic link to the custom-user-startup file in "/etc/rc.d": ln -s source_file link_name

So in our case using ssh to login to the vera execute the commands:
Code: [Select]
cd /etc/rc.d/
ln -s /etc/init.d/custom-user-startup S99custom-user-startup

At least that worked in my case and I had exactly the same problem like you when I tried to let a stick being mount on bootup for dataMine.
Maybe go in the directory /etc/rc.d and first see if you have anything there referring to /etc/init.d/custom-user-startup
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 07, 2011, 08:01:21 am
Hi Chris,

I'm seeing some strangeness with the graphing (I think I have your latest code on there at present).

The day and week view is ok. The month view is wrong though.

I checked back through the data files for this channel and all the values look ok.

Any thoughts?

Thanks,

Alex
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on December 07, 2011, 08:22:06 am
Chris, another challenge for you to consider  ;)

Still get Startup LuA Failed message, have reloaded all files, etc - tried different versions of L_DataMine1.lua all to no avail. Checked the log, entry is below - can you throw any light on this at all please?

2/07/11 23:10:37.174   LuaInterface::CallFunction_Startup-1 device 21 function initialise failed error loading module 'L_DataMine1' from file '/etc/cmh-ludl/L_DataMine1.lua':
   /etc/cmh-ludl/L_DataMine1.lua:1: unexpected symbol near '<' <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on December 07, 2011, 08:23:23 am
I didn't realize till now, but I can actually confirm Data not showing correctly in Month view. And Data is available for complete month.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 07, 2011, 08:30:03 am

@willstar

I don't know if this is any help, but a '<' at the start of the file could indicate you've somehow got some HTML in there, e.g. <head>

If you log into Vera over SSH and cat the file with something like cat /etc/cmh-ludl/L_DataMine1.lua then you should be able to determine if it looks correct?

Cheers, Alex
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 07, 2011, 08:57:50 am
Quote
The day and week view is ok. The month view is wrong though.
Ah - I've been found ok!  ;)
This is a partially implemented feature. It was one of the things I initially worked on, but priorities changed as people were asking for graphing non-numeric data, and better support for status variables... The issue is that processing a months data (and sometimes less if the data changes often) can take a huge amount of time, and this can cause Vera to go bye-byes (i.e. Lua restarts). So, the concept I have is to generate historical data - min/max over a period are recorded for hourly and daily data. Then, depending on zoom level dataMine selects the appropriate data set. This speeds things up a huge amount!

So, this is largely implemented, but it needs to run each day - I need to hook this in somehow. If you want to try it out, then go to the config screen, go to the variable, and click on the little clock icon. This will tell Vera to go away and generate the historical data. It can take a few "seconds" (or tens of seconds) depending on the size of the data set. Doing this will make the monthly views work.

I'll try and sort this out next...

Quote
You need to ad a symbolic link to the custom-user-startup file in "/etc/rc.d": ln -s source_file link_name
I'm far from an expert in Linux, and know even less about OpenWRT, but I found something that said that the by running "/etc/init.d/custom-user-startup enable" it would automatically put in the link - it seemed to work for me. If I understand correctly, it looks at the start of the file to get the priority, and then generates the correct name in the rc.d directory.

Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 07, 2011, 09:02:29 am
Quote
If you want to try it out, then go to the config screen, go to the variable, and click on the little clock icon.

That works thanks :)
Title: Re: New dataMine graphing and logging plugin
Post by: Qrp on December 07, 2011, 09:24:19 pm
@chixxi Thanks for the information on creating the symbolic link, it worked

@Chris Might be good to put that in the install wiki in case others run into the same difficulty.

Thanks to everyone who helped, and to Chris for putting this together in the first place.

Chip
Title: Re: New dataMine graphing and logging plugin
Post by: willstar on December 09, 2011, 10:22:27 pm

@willstar

I don't know if this is any help, but a '<' at the start of the file could indicate you've somehow got some HTML in there, e.g. <head>

If you log into Vera over SSH and cat the file with something like cat /etc/cmh-ludl/L_DataMine1.lua then you should be able to determine if it looks correct?

Cheers, Alex

Thanks Alex - I had a look and there is some html there - <head> etc - although that appears in all the versions of the lua files (datamine) that I have tried copying in via SCP - weird.

I'm thinking I might start from scratch (again - did this before) but I think there's a simpler solution somewhere. If nothing else it is helping me up that steep learning curve!
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on December 10, 2011, 05:49:31 am
I tried your plugin in UI5. See Screenshots attached.

It is "almost" working. I did no re install the plugin after the update. The device is still available and I can choose channels etc. However, when trying to view any data (Screenshot2) it seems that the graphs are not generated and the view drops back to Screenshot5. Trying to get something out of the log right now:

Code: [Select]
50 12/10/11 11:48:50.289 luup_log:24: dataMine: ID: 2 Start: 1323427730 Stop: 1323514130 Sample: 123 <0x8008>

50 12/10/11 16:36:19.112 luup_log:24: dataMine: Initialising Data Mining System <0x402>
01 12/10/11 16:36:19.159 LuaInterface::CallFunction_Startup-1 device 24 function initialise failed /usr/lib/lua/json.lua:122: Unterminated JSON encoded object found at position in [] <0x402>
01 12/10/11 16:36:19.160 LuImplementation::StartLua running startup code for 24 I_DataMine1.xml failed <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on December 10, 2011, 07:57:56 am
It is most likely due to the fact that the plugin needs to be updated for UI5. Changes where made on how the plugs work with UI5. Since Chris is not a Beta tester, he would not have known this or been able to make the changes. So Chris have to make the proper changes to make it work.

- Garrett


UI to UI5 migration (http://wiki.micasaverde.com/index.php/UI4_UI5_Migration) has some info on changes especially for plugins.
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on December 10, 2011, 08:06:58 am
It is most likely due to the fact that the plugin needs to be updated for UI5. Changes where made on how the plugs work with UI5. Since Chris is not a Beta tester, he would not have known this or been able to make the changes. So Chris have to make the proper changes to make it work.

Yes, please don't get me wrong. I was expecting that some things would not work after the update, just wanted to let chris know. I can offer my help if he needs somebody to test. No need to hurry. I am great full for every plugin developed.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 10, 2011, 08:11:01 am
MCV have sent me UI5, but I've been busy with work over the past week, and I've also had builders start an extension on my house so I've not had the change to look at it yet. I'll try and have a look tonight...
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on December 10, 2011, 08:30:40 am
Looking forward to the update Chris. I would like to get it running on my box. But since I have been running UI5 for quite a while, I could not do so.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 10, 2011, 04:04:48 pm

I just updated to UI5 and all my logged data files have gone. Might be worth backing them up yourselves before updating just in case
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 10, 2011, 04:23:04 pm
Quote
I just updated to UI5 and all my logged data files have gone. Might be worth backing them up yourselves before updating just in case
Maybe the USB stick wasn't remapped? Or maybe it wasn't mapped properly previously and now that the system has updated the files have gone.

I "upgraded" to UI5 this afternoon, and I can no longer access my Vera at all, so unfortunately I'm unable to work on this until I get some support from MCV. Sorry.
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 10, 2011, 04:26:50 pm
It depends on your definition of correctly mapped. It was mapped, it worked. imho software should not be deleting files on external media.

That said this is the least of my worries as very little now appears to be working under UI5. Sigh. I knew I shouldn't have upgraded. Every time.
Title: Re: New dataMine graphing and logging plugin
Post by: utz on December 22, 2011, 09:35:37 am
Chris,

any news on dataMine on UI5? I use this for logging which I like, so I only want to upgrade to UI5 if this one is working...

Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 22, 2011, 11:01:52 am
Sorry - no UI5 as yet... I had a load of problems when I upgraded, so I'm back on UI4. I've ordered a Vera3 which should be here very early in the new year and I hope to get dataMine working on UI5 then.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 22, 2011, 02:06:23 pm

I've just spotted that my reinstalled copy of dataMine, since downgrading from UI5 isn't running. I get this json error too -

Code: [Select]
50      12/22/11 19:04:17.088   luup_log:13: dataMine: Initialising Data Mining System <0x402>
01      12/22/11 19:04:17.122   LuaInterface::CallFunction_Startup-1 device 13 function initialise failed /usr/lib/lua/json.lua:122: Unterminated JSON encoded object found at position in [] <0x402>
01      12/22/11 19:04:17.123   LuImplementation::StartLua running startup code for 13 I_DataMine1.xml failed <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: ajlennon on December 22, 2011, 02:12:43 pm

I see what it was. Somehow I had an empty dataMineConfig.json file in /www/cmh/plugins/dataMine. I don't know how that came to be but when I deleted it and reconfigured the dataMine device all seems to be working (at least I get no errors logged any more).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 01, 2012, 08:42:04 am
I've updated dataMine to add the ability to select graph types. This can be configured for each variable in the configuration page.

Two options are currently available -:

I've attached the updated ZIP here, and also updated the version on the wiki.
Title: Re: New dataMine graphing and logging plugin
Post by: andygodber on January 01, 2012, 11:14:24 am
Is it now working with UI5?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 01, 2012, 11:26:04 am
Quote
Is it now working with UI5?
No - sorry. I have a Vera 3 arriving (hopefully) any day now. Due to issues I had upgrading my Vera 2, I don't want to loose my online system (again) so I need to wait for the Vera 3. Once it arrives, getting dataMine working on UI5 will be at the top of todo list...
Title: Re: New dataMine graphing and logging plugin....and Homeseer?
Post by: andygodber on January 01, 2012, 04:23:20 pm
Chris - Understood  ;)

Anyone else, do you know if there is an equivalent plugin/functionality for HomeSeer?
Title: Re: New dataMine graphing and logging plugin
Post by: kskenyon on January 18, 2012, 12:44:36 pm
L_DataMine1.lua has a typo at line 188.  Lua will not start up.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 18, 2012, 02:55:34 pm
Sorry- I thought I'd uploaded the new version a couple of weeks ago. T'is now updated on the repository and on the first post of this thread.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on January 20, 2012, 05:47:27 am
Hello,

I am trying dataMine for the first time and I have not been able to get it running successfully. I followed the instructions but when it load screen both the graph and config screen just display the status bar Opening....'

I did notice the D_DataMine1.xml contained urn:schemas-cd-jackson-com:device:DataMine:1 whereby the instructions state
urn:schemas-cd-jackson-com:device:DataMine1:1.

I would appreciate any help that you can provide me.

Regards
Tim
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 20, 2012, 08:50:00 am
I am trying dataMine for the first time and I have not been able to get it running successfully. I followed the instructions but when it load screen both the graph and config screen just display the status bar Opening....'
This is indicative of the javascript files not loading, which could be because they aren't in the correct directory (??). Are you running this on UI4 or UI5 (dataMine doesn't currently work on UI5, and I think at least part of the problem is to do with JS file compatability).

Quote
I did notice the D_DataMine1.xml contained urn:schemas-cd-jackson-com:device:DataMine:1 whereby the instructions state
urn:schemas-cd-jackson-com:device:DataMine1:1.
You're right - the "device_type" should be set to "urn:schemas-cd-jackson-com:device:DataMine:1". I'll update the instructions (I'm surprised this hasn't been noticed before - thanks for pointing it out). I don't think this will be your problem though, and I guess you've already tried this anyway.
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on January 20, 2012, 05:13:22 pm
This is indicative of the javascript files not loading, which could be because they aren't in the correct directory (??). Are you running this on UI4 or UI5 (dataMine doesn't currently work on UI5, and I think at least part of the problem is to do with JS file compatability).

I retried from the start to confirm the loaded correctly and where in the correct directory structure. Still the same. You are correct, I ran logging to the USB and it indicates the error below

50   01/21/12 11:30:27.171   luup_log:170: dataMine: Initialising Data Mining System <0x402>
50   01/21/12 11:30:27.173   luup_log:170: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>
06   01/21/12 11:30:27.174   Device_Variable::m_szValue_set device: 170 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x9a9fe0/NONE duplicate:1 <0x402>
06   01/21/12 11:30:27.175   Device_Variable::m_szValue_set device: 170 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
50   01/21/12 11:30:27.176   luup_log:170: dataMine: Reinitialising configuration structure <0x402>
50   01/21/12 11:30:27.180   luup_log:170: dataMine: Startup complete <0x402>
02   01/21

I can see the usb is mounted but still no sucess. Any thoughts.

Regards
Tim
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on January 20, 2012, 08:05:56 pm
Hello Chris,

I found there were older version, so used the 28 Nov 2011 which worked. I loaded only:

L_DataMine1.lua
D_DataMine1.xml
S_DataMine1.xml
I_DataMine1.xml
D_DataMine1.json
J_DataMine1.js

Tim
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 21, 2012, 06:43:14 am
Hi Tim,
That's great. If you've got an old version, you might also want to look at changing the other two JS files (the flotr.*.js files). This was also upgraded at some stage to fix a minor, but anoying bug relating to scrolling.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on January 21, 2012, 07:12:40 am
Chris,

Thanks and will do. I have it up and running and you're done a great job. I really appreciate the effort you have put into to give a very usable and useful application.


Tim
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 24, 2012, 05:18:43 pm
Just a quick update for those waiting on a UI5 version of dataMine. I now have a version working, but it will have to operate a little differently than it is (or was!) on UI4. The big problem I've hit, and haven't managed to overcome, is that on UI5 the jQuery library is used as well as the Prototype library. There are some compatibility issues between these libraries, and this stops the graph library from working (the issues are well documented on the net).

So, what I've done is to move the graph window onto a separate web page - in some ways, this might actually be better as the window then isn't constrained by limitations to display inside the small properties page (ie I could make the graph larger or add more "stuff"). The configuration window remains the same - i.e. it's accessed exactly the same through the properties page. I've also got a link to the graph page inside the dataMine plugin properties.

The graph page itself will be on [Vera IP address]/dataMine.

The only downside to this is that as it operates outside the UI5, it's not accessible remotely (although this doesn't work on UI4 due to the way Vera works - I don't know if this is fixed on UI5, so it may not be possible anyway!).

Once I get a few issues sorted with apps.mios.com, I'll get this version released. Once I've got a version out there, I might have a look at "better" integrating it to UI5 if that's what people want.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: weissertigerxl on January 24, 2012, 05:23:51 pm
Hi Chris,

the plugin is just what i searched.

But I have the same issues:

a) only opening when clicking at configuration button
b) When I look at the logfile: I saw the same error when startup dataMine... cannot open

78: dataMine: Initialising Data Mining System <0x402>
78: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>

I check the dir at the usb stick:

After reboot vera by switching off, I change to dir /dataMine/ and check rights for ... but I can touch a file using putty and root.

Do You have any ideas?
Your wrote in a previous answer about jason file and javascript files. Where do You mean, that they should be?

Thanks for help!

Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on January 24, 2012, 11:10:54 pm
On the code page ther is the zip file from 28 Nov 2011. I deployed this and it works fine.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 25, 2012, 06:49:52 am
The dataMineConfig.json file should be stored in the directory /dataMine/. When you first load dataMine this file will not exist, so you will get this error message, but it shouldn't be a problem. You should still be able to go to the config screen in the dataMine properties, and enable logging of variables. When you enable logging, you should see the config file being written into the /dataMine directory (along with a backup file).

Javascript, and other files, are stored elsewhere - only the config file, and the data files are in the /dataMine directory.

A quick "fault finding" guide for other installation issues -:
SymptomResolution
Continual "Opening" messageProbably, the J_dataMine.js file isn't loaded.
No button icons on graph screenPNG files aren't in the correct place (/www/cmh/plugins/dataMine)
You get the buttons, and can get a list of channels to graph, but there's no graphFLOTR JS files aren't in the correct place (/www/cmh/plugins/dataMine)
Title: Re: New dataMine graphing and logging plugin
Post by: jmmorgan on February 11, 2012, 10:59:05 am
Chris:
This is a very nice plugin and you did a great job.  I had the same problem with continual "Opening" with the J_DataMine.js file from the current zip 18-01-12.  When I used the file from the 28-11-11 zip, it worked fine.  I do not know if you want to look into this further to help ease the installation.
Thanks for the great job.
John
Title: Re: New dataMine graphing and logging plugin
Post by: haveitall on February 19, 2012, 11:40:38 pm
readme says:
   DeviceType: urn:cd-jackson-com:deviceId:DataMine1:1
http://code.mios.com/trac/mios_datamine/wiki#no1 says:
   DeviceType urn:schemas-cd-jackson-com:device:DataMine:1

I think the second one is right - confirm?

... I'm stuck on 'opening....'
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on February 19, 2012, 11:53:06 pm
readme says:
   DeviceType: urn:cd-jackson-com:deviceId:DataMine1:1
http://code.mios.com/trac/mios_datamine/wiki#no1 says:
   DeviceType urn:schemas-cd-jackson-com:device:DataMine:1

I think the second one is right - confirm?

... I'm stuck on 'opening....'

The correct details are:
urn:schemas-cd-jackson-com:device:DataMine:1

If you look at the previous post, I and other had some issues with the recent updates and I used the files dated 28 Nov 2011.
Title: Re: New dataMine graphing and logging plugin
Post by: haveitall on February 20, 2012, 11:37:21 am
Thanks for the confirmation...

Last night I uploaded the 'old J_DataMine1.js and it didn't seem to make a difference.  Though, I am also unclear on if it overwrites the old... doesn't really say and you can't 'delete' the file from the GUI.  I suppose there is a way to hand-jam it through SSH - though I am a bit leary of running roughshod and deleting things.

I'll keep poking around.

Other thing that seem weird.  I formatted the USB by turning on logging, I got a bunch of errors when I tried to run the mount and mapping - so I turned off logging.  The dataMine directory is now empty, and the mount/map commands don't throw errors.  But why is the DataMine dir empty?
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on February 20, 2012, 11:50:49 am
Try uploading the files from Nov - all off them  - uploading will over write the previous files.
You also have to manually copy the Json.lua file to /usr/lib/lua.
Title: Re: New dataMine graphing and logging plugin
Post by: haveitall on February 20, 2012, 05:22:54 pm
 ??? ??? ???
Ok, so it hasen't been working all day.  I went in and modded a few other of my luups, and suddently it is working?

I assume something was restarted?  ( I rebooted multiple times with no affect)

Whatever - it's progress!
Title: Re: New dataMine graphing and logging plugin
Post by: haveitall on February 20, 2012, 09:54:28 pm
So, I may be the slow kid here, but I didn't know that closing and restarting my browser  was CRITICAL to the instal process.  Everything is working now, and I probably didn't need to spend nearly as much time on this if I had restarted the browser a few times.

The reason it 'suddenly started working' is that I switched to my laptop.  Back on my desktop it 'suddenly stopped working'.  Restarted the browser and I was good to go.

After I had it working on the 11-11 version:
I went back and reloaded the 1-12 version, restarted the browser.  Didn't work. (stuck on 'opening....').  Reloaded the 11-11 version, Didn't work (stuck on 'opening....').  Then Restarted browser and it worked.

Cheers to Chris for putting this together.  This plugin should be standard on Vera - it is really one of the main reasons I bought it.  (I had cobbled something together with an Arduino and some crazy wiring, but this is much cleaner!)
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on February 25, 2012, 04:37:33 am
hello guys

i was trying to install the plug in

i followed everything till i reached this

"Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

mkdir /dataMine
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine
Modify (or generate if it doesn't exist) the /etc/init.d.custom-user-startup file to add the line -:

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &
Run the following commands -:

chmod +x /etc/init.d/custom-user-startup
/etc/init.d/custom-user-startup enable
This ensures the USB stick is remapped when Vera restarts. The second command makes the file executable, and the last command adds the symbolic link to start it on startup"

i didn't understand what to do here...will someone explain please?


thank you
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on February 25, 2012, 05:10:37 am
Try installing the files dated 28 Nov 2011. There have been a number of users that have encountered issues with the most recent addition. Also you must reload you system several (and again) times if required. You also may need to enter all the details in the adavance tabs if it does not default. The core details are: 

Device type = urn:schemas-cd-jackson-com:device:DataMine:1
Device File = D_DataMine1.xml
Impl File = D_DataMine1.xml
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on February 25, 2012, 08:51:43 am
u don't recommend to install this plug ins?

i did what u have said
i got it working
but i didn't understand how save them on the usb
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on February 25, 2012, 12:14:18 pm
Add a few devices to monitor in datamine. Restart Lua, it may take a why. SSH you unit and check the new datamine folder. If you followed the steps and it worked, there should be files in that folder.
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on February 29, 2012, 04:01:41 am
Sorry but till now i couldnt make it works :(
here is what i have done:
- uploaded the files
- created the device
after i SSH my vera, i couldn't find "Datamine" folder in the /www/cmh/plugins/dataMine , so I CREATED it...
i have copy pasted all the files as mentioned
i have pasted the icon to /www/cmh/skins/default/icons as well
as well as json file

The device is shown and everything seems well
after that i didn't understood what to do!!

i have added devices but after a while they are removed somehow :(
would any1 help
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on February 29, 2012, 05:07:52 am
Hello,

This is under the root directory, run command. Note dataMine.

mkdir /dataMine

Run command
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

Modify (or generate if it doesn't exist) the /etc/init.d.custom-user-startup file to add the line -:

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

Run the following commands -:
chmod +x /etc/init.d/custom-user-startup
/etc/init.d/custom-user-startup enable

Restart Lua
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on March 01, 2012, 04:43:33 am
Thank you Brientim
everything worked this time, except for that i couldn't find the "CVS" files which i should the data from
I removed the USB from vera and connected to my PC, found only a json file which contains the devices i am tracking.

how can i read the data?
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on March 01, 2012, 12:28:54 pm
Click on the spanner and go to the graph tab > select channels > then select your item. On the right side of the item an icon appear, click on that icon. You can select more than one item. If the graph tab just appears to be doing nothing when you first select it, click on the config tab and wait for that to render and then click back on the graph tab.


Connect  to Unit using SSH (Don't unplug the USB unless you power down and power up after reconnecting - that is what that commands were for),
Once you configure an item to be  recorded, it should create a text file.  The text file is the file name store the data. If they at not there, I can create them manually. I could send an example if required.
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on March 03, 2012, 06:17:58 am
THANK YOU :D

at last!!!

one last question:

how can i read those?
1330533978,1
1330533981,0
1330534622,1
1330534624,0
1330536520,1
1330536521,0

and for example, why my DOOR sensor has three files not one?

Do you recommend any better plugin to monitor and save devices status to an extrenal device? (USB,Server...)
Title: New dataMine graphing and logging plugin
Post by: futzle on March 03, 2012, 06:29:06 am
therealabdo,

Look up "Unix epoch time" for the meaning of those 13* numbers. The rest should be self-explanatory.
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on March 03, 2012, 07:15:51 am
You can look at the generic data plotter(GDP); however, you will need to send it to a syslog server. It compiles a lot of data. It is not better or worse, it is what best fits your needs and what you hardware you have available.

A simple monitoring could be via notifications and a email capture or the alternative do do the same via the snd app. http://forum.micasaverde.com/index.php/topic,6656.0.html

I suppose the question should be How to do this; detail what this is you want to do. There are a lot of good advise already in th forum. Use keywords to search and that way you may find what you are trying to achieve is already answered.

But keep asking and I'll help as best I (we the forum) can.

For GDP see http://forum.micasaverde.com/index.php/topic,7333.msg46575.html#msg46575

http://www.epochconverter.com/
Title: Re: New dataMine graphing and logging plugin
Post by: therealabdo on March 04, 2012, 05:09:48 am
Thank you very much

i am now looking into those plotters. Actually my problem is that i live in a place which has an unstable internet!
so i need always to save the most significant notifications somewhere. this datamine is the second program i am trying. but both r not professional

thank you and regards
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 06, 2012, 07:00:24 pm
Chris et al.

I am distinctly not a developer, so apologies in advance for just how clueless I am.  I can half follow instructions and hack around with Excel.  I'm brand new to the entire Vera2 scene, but getting up some sort of learning curve over the last few days, so more apologies.  Chris, I am really impressed with what you've pulled off with dataMine, initially in a short stretch, and you've stuck with it.  I'd like to use it, but I'm scared on two fronts: i) that I am insufficiently competent to get it installed properly (and would need help I might not get); and ii) once I do, whether it will allow me to achieve my main objective.

What I need to be able to do is stream csv-formatted data files, generally monitoring wattage, switch events, and on/off states of motion detectors (latter I'll probably run with Insteon network into Vera2 for cost reasons).  When I'm through, probably 100s of devices perhaps.  (Don't ask, but a bit of proto development for my company.)

What I've managed to do to this point via reading all the help some of you guys have placed around the forum and on the Wiki is i) pull the LuaUPnP.log file out of my device via WinSCP; ii) load it into Excel; and iii) parse out the "now:" "watts" values from EnergyMetering1; and chart 86,400 seconds for a given day.  Of course I would start to write VBA routines to ferret through the text files and extract data -- the current approach just to test something is doable from the Aeon Smart Energy Switch.  (I did this all just from inspection -- can't say I could figure out the syntax of these statements well at all, e.g. what the five parameters are in the "Log" statement, etc.)

Chris, what you have is potentially a much more efficient approach that might (already) get me much of the way to where I need to go (and from what I can tell the closest and best thing anyone has -- correct me if anyone thinks I might look elsewhere)  As I said earlier, I'm just a bit scared I might not even be able to get it installed, but presume eventually I will prevail, perhaps with a little help.  Once I do, will dataMine allow me to offload some reasonably structured data sets on/to my PC?

If so, Chris, using your tool I'm sure would be the best route, and perhaps I might even contribute some value back from my own world.  Let me know what you think -- I'd really appreciate any guidance you might have,

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on March 06, 2012, 08:50:59 pm
Hello John.

Have a look at the previous post in and you may also want to look at GDP
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on March 07, 2012, 01:56:10 am
hey chris.

Any news on the UI5 support? I am missing your plugin so much! I know you didn't have a good start on the vera3 and UI5, but has there been any progress?

Thanks for the update.
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on March 07, 2012, 02:49:44 am
Yes, we really need some graph in UI5!!!!!!
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 07, 2012, 07:39:35 am
Breintim, thanks for that.  I've got a slightly better clue on a few things, having managed to login via WinSCP since first reading (most of) this thread -- lot in it.  Hence a little less scared perhaps, and I missed this latest stuff re: GDP and getting data out in text files) -- thanks for the steer.  To be honest, I have even achieved victory mounting a USB, at least via the console, but I'm sure I'll get there.

Will keep you posted.  (On other matters, one of the reasons I went with Vera2 / UI4 was seeing some of the backward compatibility problems encountered by the plugins -- figured not smart for me to be on the bleeding edge on this sort of stuff.)
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on March 07, 2012, 07:46:00 am
@JohnR,

Well done. Everyday is a school day in my world. That is to say, live and learn. Good luck and there are always very good people in this forum willing to assist.
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 07, 2012, 09:39:46 am
Tim, many thanks.

Much obliged and wish me luck.  May be an adventure, but I think the investment will be worth it.  Should save me being 'on my own' parsing the main log file -- which has a structure not entirely conducive to structured data logging (as Chris also quickly noted way back when, when he stared at it).

For anyone who gets this far / notices this post, who also has the vaunted Newbie status (which in my case should be permanent), simply from reading through these 14 pages, I would summarize a few pointers as follows (the experienced guys will correct / amplify anything I've gotten wrong, but here's some advice from the clueless wing of the forum) . . .

Pay attention to actually using you shift key properly -- it's dataMine (and this system seems to care about the capital M and no capital d.  From my experience in some OSs, might be good get this right the first time, as some systems don't 'take' changes like simple capitalization changes that easily once set improperly.  It would seem Linux really cares, so pay attention to capitalization wherever anything crops up in the instructions.

Try the latest dataMine build if you want or think new features are key.  However, as/if you have problems, please note that the 28 Nov 2011 build looks to work much more consistently for folks.  Consider accordingly, but need to be very careful as to what bits and pieces you pick from different versions -- there is traffic on this early in this post.  It's all a little chaotic, but expect you can ferrett through required files, look at time stamps / file sizes, etc. and confirm you've got what you need where it should be.  (Take note of some bloody 'zip file bug' caused by newer Vera firmware builds, see 14 Nov 2011 post from mcflorin; whether this is Izo, izo, or lzo ultimate file extension I'll only know when I see these files -- in this thread search on any of these possibilities to see 'hits' on the issue.)

There seems to be some variation on who can, and who can't, get the USB to mount from the console 'log to USB' option.  However, there seems to be some traffic re: this being a pre-set 500 MB partition anyway (God knows whether that's still current), so if you have a LOT of data, consider the route Chris has stipulated.  He (and others) seem to be able to format FAT32 (NEW stick if get any problems) on Win7 machine and plug it into the back of machine and deal with the instructions carefully from WinSCP.

Again, wish me luck -- I may need it.  If this all goes without a hitch, I will post back accordingly.  Otherwise, for sure I'll be back  :)

John
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 07, 2012, 10:31:06 pm
OK, well that was hard work (for me) getting to where I did today on this.  Can't say how impressed I am (and thankful) for the sweat and pain encountered by those that have gone before me -- see further below for my own contribution to the effort.  The various guidance in the earlier posts were essential additions to even the most recent install instructions -- in my opinion.  I hadn't expected it to be easy, and I'm not sure I'm there yet, but . . .

I don't have anything at this point that is obviously not working.  Nor, for that matter is dataMine exactly working (yet).  My console is stitting there with "Running Lua Startup" for the past hour.  Since this seems to appear even when dataMine is functioning, and can magically disappear, I am not sweating that (yet).

So I went to the Config tab, where I have been staring at the "Opening Config" bar for about an hour as well.  Deciding there is a lot of initial work related to getting to the Config tab (I think I read this from Chris in the early stages), and figuring this might have to do with mapping the USB, I'm going to let that cook overnight and see what the morning holds, perhaps after another reboot.  I gave it a shot with a 16 GB (not new) USB stick, but freshly formatted FAT32 Win7, full format.  We shall see -- after a few failed attempts with logging to USB from console, and figuring this would only get me a 500 MB partition anyway, I decided to go this route.

So, if tomorrow brings no joy, I have a few ideas.  First off, I figured no harm in trying latest 18 Jan 2012 build, but I may drop back to 28 Nov 2011.

Also, the 16 GB stick may be just too ambitious.  If I had had a 4 GB lying around I would have used it.  Hence, off to Staples tomorrow to get a new one, ideally unformatted, of 4 GB capacity and see what then happens.

Failing all that, there were a few spots in the installation, where I was unsure of what I was supposed to do.  To indicate what these may have been (and to provide a potentially slightly easier route to those who may follow), I have attached my own form of amended Wiki install.  I pasted the Wiki install into Word and turned track changes on, so you can see what I've added / changed -- take anything from me with a grain of salt, but perhaps some elements of guidance for edits going forward that might get less experienced folks through the install a bit easier.  Of course, looking at my Newbie tag, I didn't simply make this stuff up, but simply tried to cobble together from what others who seemed to have provided solid guidance.  This at least indicates what I've done.

I can/will of course have a look at the LuaUPnP.log file for anything that might provide a steer, if I still have problems in the morning.  It's clear that Chris's recommendation for testing whether the USB is mounted, is not currently working in my case, so I think the system is still chewing on the 16 GB monster I plugged in.

More tomorrow,

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 07, 2012, 10:43:31 pm
Just as a quick postscript, the biggest areas of shakiness I had were the various instances in the instalation instrucitons where files and/or folder were referenced, which needed editing or modification, that I didn't have in the first place.  Being uncomfortable with this whole SSH / WinSCP environment and what was going on, I created these with some considerable lack of confidence.

Related to this was the inclusion of the instructions from member "chixxi" related to creating the 'symbolic link' to get the custom-user-startup file activated.  I presume this is now part of the official instruciton / recommendation, but I was unsure.
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 08, 2012, 05:02:22 pm
First, praise where praise is due.  Chris, very impressive, and this needs to be repeated, very impressive, plugin -- can't thank you enough for contributing it to the cause.  The data files are just so damned clean -- really excellent work and smart thinking all around in the UI.  The effort I've spent flailing to get this to work will be recouped easily in streamlining our data acquisition effort from the Vera.  Can't thank you enough (ping me if you ever have an Excel annoyance that is bugging you or a financial calculation -- about all I know).  And for MCV, I bought the Vera, and stayed with UI4 based solely on reading this thread (and of course will be scared to mess with any upgrades).

As well, I have to thank quite a few folks that have posted here over last few months.  Not just the ones that have provided solutions, including Chris's own persistance, but those who have had problems that have toss out what they were.  I would never have gotten past / through the .lzo bug any other way.

Right, from post below, here are the open questions that I expect some who may benefit may benefit from:


Right, apologize for essentially talking to myself over last 48 hours on this topic, but all good news (not without some pain, but I don't think as much as some who have blazed the trail in front of me).  Also, perhaps some incremental notations / improvements and 'lessons learned' for those who may follow me.

Having gotten jammed up last night I decided to 'clear back and simplify'.  Hence, I'm afraid I don't have a definitive position on where my problems were, but I have a strong suspicion, and it relates to something no one has yet considered on this post (and I've printed out and read through all 83 pp a couple of times).  What USB port are we presuming here for mounting?  I expect one of those 0s in the command string means USB1, and setting a 1 in there somewhere -- perhaps someone could specify -- would mount the dataMine directory to USB2?

About halfway through today's misery (more below that may help some folks) I realized I have my Insteon PLM controller patched into USB1 (top port).  USB stick in USB2.  Hmmm.  As/if it can't find the /dataMine logging directory, it can't set-up dataMineConfig.json, and you simply get a 'stalled' config window on the dataMine tab.

So I started this afternoon's effort with a clear picture that I wasn't getting the config file into \dataMine.  So I:

1.  Deleted all the files that related to the effort of mounting to USB stick
2.  Replaced upload_upnp_file.sh (as recommended for those of us with 1.1.1350 builds by mcvflorin).  Decided best to save my previous upload file to my local in the event this failed to work.
3. Decided smart to 'drop back' to 28 Nov 2011 build, as again, less variability with problems.  (When I get brave again I'll revisit changes / improvements to determine if important to me.)
4.  Determined No. 2 broke my Vera's ability to use MiOS developers install area of console
5. Reloaded previous upload.sh file, and determined that the WinSCP process was blowing these files apart.  (Note Newbie status.)
6. Now I have nothing functioning.  Dug around and found link for 1.1.1245 build, which I had determined from yesterday's effort might be better bet than 1.1.1350.
7.  Cleared back to factory settings.

Followed my own script / Word document from yesterday, using 28 Nov 2011 build, and (essentially) had no problems at all.  In process of that I CAN confirm i) the .lzo bug does not affect anyone with 1.1.1245 build; and ii) my 16 GB flash USB seems to be logging data (near as I can tell).

I think that's it for me.  If there is interest in me cleaning up my Word document with better information and some additional ideas/thoughts I had from today's post, let me know.  I'll happily chip back in and then leave to Chris and/or others as to what of it might be useful to put into revised Wiki instalation instructions and/or ReadMe file.

All the best.  Off to now reinstate my Insteon motion sensors as triggers -- they worked pretty well with Vera btw.

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: Da_JoJo on March 08, 2012, 06:46:31 pm
nice story you got there :-)
might wanna try http://www.winscp.com (http://www.winscp.com) the datamining works in ui5 , the chartsplotting and interface to it doesnt (probably the changes in ui5 interface )
for other usb port to use you would need to set it somewhere in the plugin to usb2   i.e. change the 1 into a 2 in the plugin files
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 08, 2012, 10:07:04 pm
Yeah, I'm sure only the desperate (like me) may gain some value from it.  Thanks for link -- have bookmarked it and will see how things progress.  Couldn't be happier (so far) with dataMine, however.  The logging files and organizaiton are just so clean.  I've still got original iPad, so I'll be fine with UI4 for a while I expect -- need to amortize the cost of my pain over some usage time.

As to USB, given the port syntax noted regarding the (no one can get to work) 'log to USB' option on the console, I expect 0 = USB1, and 1 = USB2.  There's a MOUNT command executed in various ways in the install for dataMine, and I expect a piece of that establishes which USB port.  If no one has quick answer on this, I'll do some digging around.
Title: Re: New dataMine graphing and logging plugin
Post by: Da_JoJo on March 09, 2012, 08:03:27 am
yu thats probably it.. change this in the datamine installer and the datamine plugin files itself and it will use the other usb port
you can also try n list all ports with some command i forgot (been a 4 years since i touched linux-a-like stuff. do rememer that files sizes and stuf not allways correct with winscp so i used putty and totalcommander which did the job a lot better (not faultless i may add, this is due the nature of the protocol used)
Title: Re: New dataMine graphing and logging plugin
Post by: netsidedeveloper on March 17, 2012, 04:50:13 am
What a great plugin.  Is it possible to use the plug in to chart activity or inactivity of a motion sensor?  I want to setup a system in my Gran's house so that we can see she is moving around....Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on March 17, 2012, 05:59:32 am
If you want to know if there is inactivity of the motion sensor and to be notified, you can have a look at the home care plugin. 

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: netsidedeveloper on March 17, 2012, 07:49:02 am
Thanks for the reply Garrett.  I'd like to be able to see the activity on a webpage aswell, is this possible?
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on March 17, 2012, 08:13:33 am
That is where dataMine would come in as from what I gather from the homecare plugin, it just notifies the user of inactivity of the motion sensor.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 20, 2012, 09:58:21 am
netsidedeveloper,

Keep me posted re: where you get to with motion sensors and Vera2 (might be useful to compare notes, as there are some challenges among the options).  I'll put up a post under 'motion sensors' area.  I'm also setting up some monitoring in a home residence.

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: andygodber on March 20, 2012, 10:50:07 am
Chris/others - am I right in reading that the graphing function is still not working on UI5?
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on March 20, 2012, 12:46:38 pm
Why not release this wonderful plug in the app.mios.com?
Title: Re: New dataMine graphing and logging plugin
Post by: guessed on March 20, 2012, 12:57:20 pm
He's currently chasing down bugs in the MCV stack that make it trickier than it should be.  See this thread for details...
    http://forum.micasaverde.com/index.php/topic,9976.0.html

Why not release this wonderful plug in the app.mios.com?

Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on March 20, 2012, 12:58:28 pm
Because the way the plugin is build, it requires a few things that is currently broken in the app.mios.com section. This is currently being worked on by MCV. Once those issues are worked out, I am sure Chris will get it in there.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 20, 2012, 03:02:14 pm
Because the way the plugin is build, it requires a few things that is currently broken in the app.mios.com section. This is currently being worked on by MCV. Once those issues are worked out, I am sure Chris will get it in there.

100% correct. I've been waiting for the UI5 update since it was meant to fix a problem with plugins that use files that can't be compressed (as dataMine does). It seems that only half the problem was in Vera - the rest is in apps.mios.com, and that also needs fixing! MCV have said they'll try and do that "this week" - once this is done, I'll get the plugin released.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on March 20, 2012, 03:05:35 pm
That's great!  :D
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on March 21, 2012, 05:59:23 am
Glad to hear you are working on it chris! And I am glad to hear that the problems you have are apps.mios.com related, out of that I kind of interpret that you solved all the other bugs.  ;D

I am not so glad about hearing that mcv has not fixed a problem which they were aware of for at least a few month:
http://forum.micasaverde.com/index.php/topic,8974.msg58862.html#msg58862
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on March 21, 2012, 07:50:24 am
I am not so glad about hearing that mcv has not fixed a problem which they were aware of for at least a few month:
http://forum.micasaverde.com/index.php/topic,8974.msg58862.html#msg58862

This has just been fixed. (Mantis #2143 (http://bugs.micasaverde.com/view.php?id=2143))
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on March 21, 2012, 08:08:20 am
wow, now you just really impressed me! thank you!

I tested that with a custom icon (png) and the path "/../../www/cmh/skins/default/icons/", it is working!
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on March 21, 2012, 08:13:45 am
that means that we should be able to install datamine from app.mios.com soon?

What about the storage in the USB?

Thanks all
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on March 21, 2012, 08:33:14 am
What about the storage in the USB?

I am wondering about that too. I guess all the plugin files will be loaded somewhere on the vera. @chris, would it be possible that you allow specifying the path for the log storage in the plugin?

Excited to see dataMine running soon again. Thanks again for that plugin, was missing it since day one of UI5 release...
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 21, 2012, 02:55:43 pm
Quote
And I am glad to hear that the problems you have are apps.mios.com related, out of that I kind of interpret that you solved all the other bugs.
I think I posted a while ago, but the big issue is incompatability with java libs used in Vera. So, to avoid this issue, the plugin provides a separate page for the graphing, rather than embedding the graph into a device window.

Quote
would it be possible that you allow specifying the path for the log storage in the plugin?
Yes - this is already supported. It defaults to /dataMine, but it's a variable, so you can point it elsewhere.

Anyway, if everything is fixed on apps, then I should be able to get something up quite quickly - it's working on my system here so it's just a case of getting everything loaded onto apps, and configured...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 21, 2012, 08:11:12 pm
An impressive plugin for sure -- enough to keep me holding on to UI4 until looks like UI5 can support it.  Was still wondering whether anyone had a response to query:

Quote
What file holds the information that provides the list of variables that can be charted?  As with at least one other poster, be useful to be able to clean-up 'mistakes' from this dropdown; as things are now, it would seem that once you've clicked green, they are permanently latched as part of the drop down selections for charting.  Happy if I do this manually via SSH file editing.

As well (and connected to this), as I organize data logging nodes, some renaming, and a it gets a bit disorganized over time as the csv file names and device names fall out of alignment.
Title: Re: New dataMine graphing and logging plugin
Post by: slushpupie on March 22, 2012, 01:04:44 am
Have you considered adding the ability to upload the data periodically to a service like pachube.com or a google docs spreadsheet? They have some cool advanced data visualization features and lets you share your graphs with others (if you want).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 22, 2012, 04:49:55 am
Quote
What file holds the information that provides the list of variables that can be charted?
There isn't one. There's a file that holds the names of the variables that ARE BEING charted, but the list of variables that can be charted is read directly from the devices. The only way to manipulate this is through the config screen in dataMine.

Regarding the CSV filenames- I agree - it's a bit of a pain when / if you change a device name, things do get a bit messed up. In theory though, you shouldn't need to go and play with the CSV files. I keep meaning to add an export option to dataMine to allow the data to be exported as a CSV file, but I've just not done this yet.
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on March 22, 2012, 09:05:39 am
Chris, thanks for response.

Sorry, my bad phrasing.  I had meant *are being* charted, and automatically linked to that, logged to csv files.  Perhaps risky to edit file you referenced for that purpose, but was curious whether this renaming could be done 'manually'.  I certainly appreciate there would be potential continuity issue with current data depending on whether matching the file names would cause dataMine to continue to log to pre-existing file or whether it would set-up a new one.

As to csvs, I need to do quite a bit of statistical manipulation and re-processing, e.g. of binary sensor information triggering in different parts of the house.  I use dataMine for the logging, and 'tactical' visualization to ensure devices working properly.  Happy so far in batch off-loading csv through SSH (got half used to that through install : -), but would welcome a more user-friendly option for this as you and others have suggested.

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 22, 2012, 09:16:11 am
Ok - yes, you can edit the config file to rename the CSVs. It is all listed in the dataMineConfig.json file - I would recommend editing it and running it back through jsonlint.com to ensure there are no syntax errors with the json (it also formats it in a more readable way than Vera does!). If you edit the filenames in this file, and also edit the files on Vera, then everything will work fine (you'll need to restart Luup to make dataMine re-read the config.

I think I will add CSV export at some stage (soon probably), but for what you're doing, it may still be better to stick with SSH. My real goal would be to have a descent analytical plugin, but in reality I think that Vera doesn't have the processing power. That said, Vera3 responds to dataMine requests a lot faster than Vera2, so who knows...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 07, 2012, 04:47:21 pm
I'm working on installing this today before I leave for the week... hoping someone can answer one question.

I'm at this step...

Quote
Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

    mkdir /dataMine
    mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

... when I try to mount the device to the dataMine folder I get this message...

Code: [Select]
root@MiOS:~# mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

mount: mounting /dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine failed: No such file or directory

... the folder dataMine does exist in the root.

I have added a 512MB USB stick and told Vera to log to it. Here's the message I get on the Logs tab...

Quote
Enabled
Mounted: read-write
Usage: 2%
Errors log:
2012-05-07_13:40:25 (/dev/sda1) : Successfully checked
2012-05-07_13:40:25 CHECKDISK: Starting to check device: /dev/sda1 with: /usr/sbin/e2fsck
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 07, 2012, 05:24:29 pm
*UPDATE*

I want to check my work here - I think I figured it out.

Since the Vera mounted the USB drive as /dev/sda1 , I mounted /dataMine to /dev/sda1 also.
... is there any problem with this?

And, I suspect that I should need to do the following from the TRAC instructions, just using sda1 (?)...

Quote
Modify (or generate if it doesn't exist) the /etc/init.d.custom-user-startup file to add the line -:

    mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

Run the following commands -:

    chmod +x /etc/init.d/custom-user-startup
    /etc/init.d/custom-user-startup enable

This ensures the USB stick is remapped when Vera restarts. The second command makes the file executable, and the last command adds the symbolic link to start it on startup.

thx for the help!
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 07, 2012, 07:47:27 pm
OK, the USB drive is mounted to /dataMine -- I can get to it via WinSCP.

BUT when I click the Tool button on the dataMine device, in UI5, I don't options that I think I should -- see screenshot attached.
Title: Re: New dataMine graphing and logging plugin
Post by: bfromdruten on May 11, 2012, 06:53:44 pm
Does someone know if this plugin is working correctly with UI5?

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on May 11, 2012, 09:33:20 pm
I do not believe the developer ported it over to UI5 yet.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 11, 2012, 09:56:10 pm
It would be great if Chris would chime in... it has been a while since he posted and I'd like to know if UI5 is going ot be supported soon. I have a Vera3 and in need of his awesome work  ;D
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on May 23, 2012, 01:27:35 pm
Chris, just from me, sorry so pathetic re: thanking you for effort in your earlier reply.  I haven't even waded into the renaming exercise on the CSVs yet (after two months) largely because I'm being killed by hardware availability issues more than software logging -- Brultech GreenEye Monitor and Aeon Smart Energy (Micro) switches.  Hard to know which of these two outfits is better at the "should be just a matter of weeks" game.

Your plugin, which is in a 'don't mess with something working', and hence UI4, set-up is important enough to me, that I haven't even thought about shifting up.  In the event MCV is reading this, important software tool to be able to log data like this to USB port on the Vera.

Thanks as ever for contribution to the cause and your software.  In fact, it is because I caught this post that I bought the Vera in the first place.

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 23, 2012, 04:22:59 pm
It is unfortunate that Chris seems to have abandoned this project, and possibly the community. He has not posed here in 2 months - even though he has been on the forums and as soon as 4 days ago.

I suspect that porting to UI5 is probably a decent amount of work but should not be even close to a total rewrite.

seeing this plugin die is very sad.
Title: Re: New dataMine graphing and logging plugin
Post by: JohnR on May 24, 2012, 03:53:59 pm
Well, possible I suppose, as these 'hobby efforts' often suffer from a level of enthusiasm that I was actually impressed with how long Chris kept at it and engaged.  Motivation is interest and 'newness' more than anything that's fun once it becomes a job.  (I know that's the case with me.)

However, could well be Chris is simply caught up with day job, trying to ensure rent / mortgage paid, etc.  Not on here much just because he wouldn't have much to say other than 'sorry I've been off air for so long, but I will get back to this when I can'.

As/when this data logging becomes central to business for me/us, I'll pick up pieces and figure out who might be able to with what UI, if we have any reliability issues, whether need to move to different system, who might be available to be paid to get what done, etc.  For now, I'm not rocking the boat or messing with anything on my Vera set-up.  I'm just lucky that the boat's not being rocked by any other pressures at the moment.

But also agreed, would be a real shame if somehow this plugin couldn't gain somehow more of a life beyond Chris's yeoman efforts,

JohnR
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on May 25, 2012, 04:24:54 am
I just wanted to let you know that Chris hasn't abandoned this plugin and he's working on a new version of the plugin. He'll give you more details when he gets back on the forum.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on May 25, 2012, 04:47:43 am
That is great news MCVFlorin.

- Garrett
Title: New dataMine graphing and logging plugin
Post by: teonebello on May 25, 2012, 04:53:24 am
I just wanted to let you know that Chris hasn't abandoned this plugin and he's working on a new version of the plugin. He'll give you more details when he gets back on the forum.

Maybe without the need of USB or ssh the system?

It would be great


---
I am here: ?2b5x2k
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on May 25, 2012, 12:48:25 pm
very happy to hear that!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on May 31, 2012, 01:39:00 pm
Hi all,
Unfortunately I had to "disappear" for a month or so for a close family bereavement, so I was in New Zealand for a month without access to Vera (and I live in the UK), and since returning things have been a "bit busy". Anyway, while there I played around in my spare time on my PC and have come up with a different plugin. (actually, the Lua plugin is nearly the same, the web interface is completely different). This will provide a full web interface (ie not embedded in a UI window), which gives more room for graphs etc. It also opens up other avenues for future improvement, but the basic stuff is all but working.

Hopefully I'll get this out in the next week or so. I've now got the web install working (due to the number of files, it's too difficult to put everything through the marketplace system).

Unfortunately it still requires messing with USB files etc. This is basically unavoidable I think unless you just want to log a small amount of data. The installation should be easier though...

More to follow, and appologies for the lack of recent time with this...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on June 01, 2012, 03:35:59 am
Welcome back Chris. We had missed you

I had a strange effect on the one wire server which I have posted on that thread. It seems to need an active internet connection for Vera to read temperature successfully from the one wire server.

I look forward to the updated datamine. If we could access the graphs remotely it would be even better!

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 02, 2012, 07:11:40 pm
I had a strange effect on the one wire server which I have posted on that thread. It seems to need an active internet connection for Vera to read temperature successfully from the one wire server.
Hi Nick,
Yes, I briefly saw that post, but didn't really think too much about it. I've seen strange "networky things" happening with Vera in the past, so it may be that. when I get a chance, I'll take another look (time's a bit short at the mo).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 02, 2012, 07:19:01 pm
I just thought I'd post a couple of pictures of the new dataMine. As I said previosly, the plugin configuration side will look largely the same as it does on UI4 - you select the channels in the same way. Where things change completely is the web UI - this will now be at http://my-vera/dm rather than in the UI5 window. The initial driver for this was just getting dataMine to work under UI5.

Compatibility between different javascript libraries was a big issue, and it wasn't something I could easily overcome. However, it has another big benefit - it allows a cleaner, larger user interface that isn't restricted by UI5 limitations. Currently, it's quite limited in what it can do - roughly similar to the UI4 plugin (actually it's lacking a few features still compared to UI4), but I have some ideas for improvement when I get the time.

So, the attached pics show the current UI - I'll hopefully have this released in a few days (I'm just sorting out moving my operational unit from Vera2/UI4 to Vera3/UI5 to make sure everything still works...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on June 02, 2012, 07:42:24 pm
Hello Chris,

Welcome back and that looks great. Looking forward to seeing the final implementation. It addresses several of the shortfalls with Vera and will provide a great reporting capability that is really lacking.

Regards
Tim
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 02, 2012, 08:32:23 pm
WOW... looks great. This is some seriously great stuff for the community. I'm really looking forward to graphing my AC usage with outside temps, and my set points. I'm in Arizona... we use a lot of AC.
Title: Re: New dataMine graphing and logging plugin
Post by: chris66 on June 06, 2012, 08:31:59 am
Great  plugin! Cannot wait to install it on UI5 ::)
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on June 06, 2012, 08:32:56 am
Great  plugin! Cannot wait to install it on UI5 ::)

+1
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 06, 2012, 08:50:56 am
It's coming  :)

I've had a few issues with getting the install to work with apps.mios.com. The problem is that the UI is a reasonable size file (~450k), and I have a slow internet connection  :-[. This uncovered some issues with timeouts when installing apps and meant that I couldn't install dataMine from apps.mios.com. It also revealed another "problem", or hidden feature that I need to fix to stop the app installer getting into a pathalogical loop.

Anyway, @MCVFlorin has been very helpful with solving this and the problem should be fixed in the next UI5 software (this week or next!). In the meantime, once I've resolved this second issue (tonight I hope) I might release a version that will need you to use WinSCP (or similar) to put the GUI file manually onto the Vera, and once the new firmware is available I can integrate this fully into apps.mios.com.

Watch this space...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on June 06, 2012, 09:16:27 am
Thanks for the update, will this work with UI4 as well??

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 06, 2012, 09:33:40 am
Thanks for the update, will this work with UI4 as well??

Unfortunately not. The UI would work without changes, but I've added some extra functions to the Lua file to support the new UI, so unfortunately it won't work as a system. Once I get a UI5 version released, if there's sufficient interest I can probably port the Lua back to UI4 (there aren't many changes so it shouldn't take long).

My personal feeling is that the separate GUI is a better approach as it provides more flexibility, but others may have different ideas and prefer the closer integration as in UI4 (??)...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on June 06, 2012, 09:44:39 am
I'll say "yes please" to a UI4 version then, everything else is working there and I have no desire to risk breaking a working heating control system by "updating" to UI5.

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 07, 2012, 05:39:21 pm
I've now put a version of dataMine onto apps.mios.com for approval. Assuming MCV are happy with this, it should be in the marketplace in the next day or so...

Installation is a bit more difficult than it needs to be at the moment due to an issue with the current version of Vera software. This is meant to be fixed in the release due out any day now. In the meantime, one file needs to loaded manually, and I'll attach that here once the app is approved.

The picture attached shows the GUI. On the left is the list of channels that dataMine is recording - configuration of channels is the same as UI4 and is done through the plugin properties in UI5. Next to the channel is an icon - either a green tick, yellow warning, or red exclamation. This indicates the state of the channel - green means it's been updated in the last hour, yellow in the last day, and red is older than 24 hours. One warning about this - at the moment, when Lua is restarted, this gets reset and everything goes red (I'll fix this later).

To graph a channel, click on the channel name. This will highlight the channel, and also show an axis icon to allow you to select the primary or secondary axis. Then click the "Update Graph" button. The other button "Clear Channels" clears the selection of channels (it doesn't update the graph though!).

Along the top of the graph is a toolbar - this has buttons to select zoom in/out, display day/week/month, shift left/right, and display a real-time graph. The real-time graph is definately work in progress - it needs some work as the graph looks a bit screwy as it only updates when the data updates, and with the splined graphs, it just looks wrong! By default, pressing the real-time button will display the last hour of data, but you can change that with the buttons.

There's also a set of graph options - this should be pretty self explanatory (press the buttons if you're not sure!).

I think this should provide most of the functionality of the UI4 version, plus some. However, status channels aren't handled quite right at the moment (it's on my todo list and should be easy to add).

I do have some ideas for improvements, but let's see how we get on with this first  :)

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 07, 2012, 08:17:56 pm
Awesome work Chris!

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 08, 2012, 12:48:21 pm
The UI5 version of dataMine appears to have been approved in the marketplace, so here's some installation instructions. Due to the amount of data that dataMine generates, it really needs to use a USB stick, and therefore it's a bit more complicated to install than some plugins (sorry!). I should admit that I haven't tried this yet on a Vera3, but I'm assuming the same instructions I wrote for Vera2 are still applicable. I think the USB stick, and the json library should be installed before the plugin is installed from the marketplace (it's not manditory, but it probably makes sense).

Use WinSCP (or another SSH client) to upload the files to Vera...

The plugin needs the json library. Unfortunately, I can't include this into the app installation as apps.mios.com doesn't support shared files. To install this, go to -:

and download this to the following directory on Vera

Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

Modify (or generate it if it doesn't exist) the "/etc/init.d.custom-user-startup" file to add the line -:

Run the following commands -:

This ensures the USB stick is remapped when Vera restarts. It makes the file executable, and adds the symbolic link to start it on startup.

Now install the app through the marketplace (http://apps.mios.com/plugin.php?id=1088). This installs all the standard Lua device files, and should install the GUI, but due to a timeout issue with Vera (to be fixed soon) you need to load the attached "dataMineWeb.tar.gz" manually into the "/www/dm" directory using WinSCP (or similar). PLEASE NOTE: The attached file needs to be renamed - please REMOVE the .zip off the end so the filename is called "dataMineWeb.tar.gz" (the forum won't allow me to upload a gz file!). This must be done after the app has been installed - the file will already exist, and you should overwrite what's there.  Then click on the "Restart" button on UI5 - when dataMine starts, you should see a notification stating that dataMine is installing the GUI - this will take around 1 minute (at least it did on my Vera2 - it will probably take less time on a Vera3).

Once the dataMine device is loaded, click on the tool icon. You will get the main tab which has a link to the UI, and config tab. Click on the config tab, and select the variables you want to log. All changes to the selected variables should now be logged to files in the /dataMine directory.

Once the next release of UI5 is out I'll update the app to remove all this messing around with the dataMineWeb file, then updates should be nice and simple! If some kind sole wants to write a shell script to automate the USB configuration, I'm sure that would be welcome (I use linux a lot, but am far from an expert in shell scripts).

I'm sure people will let me know if you have problems ;) and I'll try and fix them as soon as possible. I do have a number of improvements planned - it's just a matter of time (I'm also trying to build an extension on the house at the moment, so my better half doesn't like to see me "playing" on the computer!). I hope to update the graphing to add/improve the current features quite soon, but other changes may take a bit more time...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 08, 2012, 01:33:31 pm
Hi Chris,
Installing now.

Question,
for the "/etc/init.d.custom-user-startup" file you have...

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

last time I installed I put in...

mount /dev/sda1 /dataMine &

... this was the only way I could get it to work. Is there a specific reason I need to use your code?

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 08, 2012, 02:34:43 pm
No - I'm not an expert in this area and I think I probably found this in an OpenWRT forum. I'd say use what works for you - of course if you've already done this on your Vera previously, then you don't need to do it again.

I'd really like to get this part of the installation made a bit easier. It's not really my area of expertese, but I think it should be possible to write a shell script to automate this (any volunteers?  :) ).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 08, 2012, 07:25:45 pm
Chris,
Under the Config, all the items are showing properly and they open when clicked... but how do you 'select' which ones you want to use? I don't see a checkbox, or highlight, etc?

thx
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 08, 2012, 08:08:41 pm
Update...

After many clicks, some of the items are responding and highlighting, then providing a drop-down... but only the two I show in my screenshot... all others I tried do nothing.
Title: Re: New dataMine graphing and logging plugin
Post by: elordude on June 08, 2012, 09:38:38 pm
Chris,

What size USB stick do you recommend?
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 09, 2012, 01:58:47 am
Update...

After many clicks, some of the items are responding and highlighting, then providing a drop-down... but only the two I show in my screenshot... all others I tried do nothing.

Also...

When I open the graph web page I get this error...
"Error downloading startup data (dataMine channel configuration)."

Title: Re: New dataMine graphing and logging plugin
Post by: bfromdruten on June 09, 2012, 02:58:30 am
Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

I just emptied the dataMine folder, the plugin will give 0 channels (data & recording), but the configuration tab still shows colours a certain variables. Don't know how to change that.

Using internet explorer 9

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 04:33:15 am
What size USB stick do you recommend?

In reality, it's not a lot of data - with about 6 months data logged, with 38 channels, I've got about 20MB of data.

I bought a really small USB stick (size wize) that only poked out the back of the Vera by a few mm. I think it was 4GB or 8GB.  Looking at my log files, if you're logging light switches, then weekly file size is minimal (<1k), and if you're logging analogue data that changes regularly (I have some humidity sensors that log every 0.1% change!) then I have files that range between 40 kb and 80 kb per week.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 04:41:28 am
Under the Config, all the items are showing properly and they open when clicked... but how do you 'select' which ones you want to use? I don't see a checkbox, or highlight, etc?
I see you've progressed from this, but I'll answer this anyway... This is the same as UI4 - you click on the devices which open the services, click again to get the variables. If you click the variable name, then the variable should highlight in green (this seems to take about a second on my Vera2, so don't click too quickly). If you click again, it will highlight in yellow (multiple clicks will toggle between green and yellow).

Green means the variable is being logged. Yellow means it's been logged in the past. I don't completely remove a variable that you no longer want to log just in case you want to keep the data history.

I see in your attachment that the variables are highlighted in yellow - this means they aren't actually being logged.

In the dataMine device in the UI, you should have two numbers being displayed - total channels, and channels being logged. Total channels is the total number highlighted in yellow or green, and channels being logged are the ones in green.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 04:49:53 am
When I open the graph web page I get this error...
"Error downloading startup data (dataMine channel configuration)."

When the UI starts, it downloads a bunch of data from the Vera - overall Vera configuration, and dataMine configuration (which is the one that is failing). It might be that the plugin isn't running when you start the UI (eg if Lua is restarting, then the UI won't be able to download the data).

If the plugin isn't running for some reason, that may also explain why the config system isn't working properly, since within UI5, the config box communicates with the plugin in a similar way to the dataMine GUI does.

Things I'd look at for starters -:

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 04:56:08 am
Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

The changing of the background colour may take a second or so since it needs to communicate back to the dataMine plugin, save the data, and send an "Ok" to the UI before the colour changes (maybe this is a bad way to do it and I should have instant feedback, but that's how it currently works). This may (or not!) explain what you're seeing?

One of the next things on my ToDo list is to move the config system into the dataMine UI as I ultimately want to add extra configuration for charting purposes. When I move it over, it will be done in a different way so bear with me for now (so long as you can actually get it to work of course).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 09, 2012, 11:42:45 am
Quote from: Chris
    Make sure Lua is running (the "unit busy" statement isn't up in the notification area of UI5)
    Check the log files to see if there's anything in there relating to dataMine.
    Have a look in the /dataMine directory (on the USB stick?) to see if the dataMineConfig.json file is being saved.

It must be running... the /dataMine folder has tons of .backup files and the .json file in it.

Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

The changing of the background colour may take a second or so since it needs to communicate back to the dataMine plugin, save the data, and send an "Ok" to the UI before the colour changes (maybe this is a bad way to do it and I should have instant feedback, but that's how it currently works). This may (or not!) explain what you're seeing?

This is definitely not working properly. I tried for 10 minutes to choose data point items... and only those 2 shown in my screenshot every would highlight/color.  The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.


I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 12:04:23 pm
It must be running... the /dataMine folder has tons of .backup files and the .json file in it.
Ok - every time you select a variable, dataMine saves a backup, so if you've done a lot of clicking, there will be a ton of files  :). Is there anything else in that directory?

The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.
Strange. Is there anything in the log file from dataMine? Until this error goes away, the graph won't work as the error means it can't get the configuration.

I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.
Don't worry - it will be very different. I'm planning on a properties box type system so that I can have the ability to edit a number of different properties easily. I started doing this with a grid, but that was a bit messy and not scalable, so I'm changing this at the moment.

Anyway, back to your problem. Please take a look at the /tmp/log/cmh/LuaUPnP.log file and do a search for dataMine to see if there's anything there. You can use the command "grep dataMine /tmp/log/cmh/LuaUPnP.log" to get the lines with dataMine.
The other question is are you running Firefox, and if so, do you have Firebug? I'd be interested to see what transactions show up in the console with dataMine starts.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 09, 2012, 12:26:10 pm
It must be running... the /dataMine folder has tons of .backup files and the .json file in it.
Ok - every time you select a variable, dataMine saves a backup, so if you've done a lot of clicking, there will be a ton of files  :). Is there anything else in that directory?

The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.
Strange. Is there anything in the log file from dataMine? Until this error goes away, the graph won't work as the error means it can't get the configuration.

I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.
Don't worry - it will be very different. I'm planning on a properties box type system so that I can have the ability to edit a number of different properties easily. I started doing this with a grid, but that was a bit messy and not scalable, so I'm changing this at the moment.

Anyway, back to your problem. Please take a look at the /tmp/log/cmh/LuaUPnP.log file and do a search for dataMine to see if there's anything there. You can use the command "grep dataMine /tmp/log/cmh/LuaUPnP.log" to get the lines with dataMine.
The other question is are you running Firefox, and if so, do you have Firebug? I'd be interested to see what transactions show up in the console with dataMine starts.

Chris

Yeah, tons of backup files... which I manually deleted but that makes a good case for having a cleanup function inside your code ;)

I use FF 14 beta but no Firebug installed... I can if we need to but lets see what you find in the log first.

I have attached the log, it is very big so I removed all lines not pertaining to dataMine...

Quote
/dataMine$ grep dataMine /tmp/log/cmh/LuaUPnP.log
01   06/09/12 8:13:23.156   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:36:48.442   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:36:54.868   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:38:35.238   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:09.268   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:40:10.098   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:40:12.038   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:13.618   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:47.441   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:57.158   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:58.508   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
50   06/09/12 8:41:21.260   luup_log:50: dataMine: Watching: D[3] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Cool] V[CurrentSetpoint] <0x2e839680>
01   06/09/12 8:41:21.298   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:26.568   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:27.255   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:27.843   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
50   06/09/12 8:41:30.509   luup_log:50: dataMine: Watching: D[3] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Cool] V[SetpointTarget] <0x2ee39680>
01   06/09/12 8:41:30.548   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:41:32.781   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:33.807   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:41:45.428   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 9:03:23.040   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>



Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 09, 2012, 01:05:45 pm
Yeah, tons of backup files... which I manually deleted but that makes a good case for having a cleanup function inside your code ;)
Yep - I agree... Another one to add to the list  ;)

The log is useful - I think I know what the problem is, although it is something that I've got code in to fix, so it raises more questions. I've added a field to an internal config structure, and I do have code in to initialise it when dataMine starts, so I'm not sure why this isn't running. Have you used the UI4 version previously?

Can you attach your dataMineConfig.json file. The other thing that might be useful is the part of the log where dataMine starts up. It logs some stuff about starting and finishing config, and it would be interesting to see if there are any errors in there when it starts.

Actually, have you restarted dataMine (ie restarted Lua)? This might explain it - I initialise this variable when dataMine is started, but not when logging... Hmmm - let me have a look at that as it might be the issue...

Thanks
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: jojoflyhigh on June 10, 2012, 04:10:41 pm
Hi Chris,
It seems that you need json.lua file in usr/lib/lua directory like for Google Calendar Switch application.
But your file and GCal file are not using the same file even if it's the same name.
Can you do something for that ?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 10, 2012, 04:45:21 pm
Hi Chris,
It seems that you need json.lua file in usr/lib/lua directory like for Google Calendar Switch application.
But your file and GCal file are not using the same file even if it's the same name.
Can you do something for that ?

Hmmm - I didn't know there was more than 1 json.lua file.  I'll take a look at what the calendar app uses and see if I can harmonise them. I know MCV are looking at incorporating a C JSON library, but it won't be in the next release (unfortunately).

Anyway, I'll take a look and see if I can find a solution. One option is to copy the lib into the dataMine Lua file - not very attractive, but it would make life easier for everyone!

I'm planning an update (probably tomorrow) to fix the problem Aaron found...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 10, 2012, 06:28:16 pm
It seems the GCal plugin is using a newer (but maybe not better?) version of the JSON library. Unfortunately, it's not compatible with decoding the files from dataMine which came from the previous version of the JSON library. I have run the dataMine json files through jsonlint (an online JSON format checker), and it shows they are ok, as are the ones produced by the new library.

I've worked out a work-around that will make it compatible with either format - the old version seems more "compatible" in that it can handle both formats, where the new version wants something done in a specific way. I'll try and put in a fix to convert the dataMine file to the format expected by the new library, and it shouldn't then matter which one you have installed.

It will probably be tomorrow before I get this done though.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 11, 2012, 12:38:39 pm
An updated version is now in the marketplace (V0.92). This updates the Lua plugin to fix two issues that have been found -:

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 11, 2012, 02:46:31 pm
An updated version is now in the marketplace (V0.92). This updates the Lua plugin to fix two issues that have been found -:
  • A variable was not initialised. This caused problems when selecting channels to log.
  • A fix has been added to provide compatibility with 0.9.20 or 0.9.50 of json.lua (0.9.50 is used by the GCal plugin).

Chris

How do we update?  I don't see a way, or notification, in the VERA UI5 to do it?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 11, 2012, 03:32:05 pm
Interesting question - one for which I thought there'd be a really simple answer :)  but.......

I've not used UI5 for long, so I'm not sure to be honest! I thought it was meant to update automagically - ie. when Lua starts, I thought it automatically checked for plugin updates and installed them. In the app configuration I've ticked a box caused auto update. I've already got V0.92 installed, but if you go to Apps | My Apps, what version does it say is installed?

It should be easy (shouldn't it? ;))

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 11, 2012, 04:05:39 pm
.91

Also, if you do a search for data mine, when it is already installed, UI5 shows no results.

once I uninstalled, it shows (v .92) and I can reinstall it.

You might want to ping MCV to see if this is a "feature" or a bug

BTW - I got the same error after trying to open the graph... before clicking 'Reload'

Also, it would be nice to be able to remove Channels from the graph's menu - it creates clutter, especially for people like me who didn't want them in the first place and was just trying to get things tested/working.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 11, 2012, 09:40:45 pm
Plugins are updated automatically. It could take a bit before they get updated.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 11, 2012, 11:26:50 pm
Plugins are updated automatically.

... that is not a desirable feature.
Title: Re: New dataMine graphing and logging plugin
Post by: guessed on June 11, 2012, 11:34:39 pm
Plugins are updated automatically.

... that is not a desirable feature.

Yup, already being tracked for completeness of implementation:
    http://bugs.micasaverde.com/view.php?id=2124
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 14, 2012, 07:44:52 pm
Picked up a couple usb drives as I am running low and installed the datamine plugin today. I must say it works a treat. I now just have to figure out what data I want to start graphing.

Question: What is the difference between sensor graph and status graph?

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 14, 2012, 08:26:00 pm
Picked up a couple usb drives as I am running low and installed the datamine plugin today. I must say it works a treat. I now just have to figure out what data I want to start graphing.

Question: What is the difference between sensor graph and status graph?

- Garrett

To broaden the question a bit... can you give a description of all the controls on the Graph? There is no mouse-over description and just by playing with it, it is difficult to know what they do.

And... when a channel seems to be providing no data, it would be great if that channel was excluded (or marked somehow) so the user does not select that Channel and muddy up his/her graph menu.  There are things like 'Thermostat - CurrentSetPoint'  which I think should be providing data (?) but has a red exclamation point on it in the menu... so, I'm not sure ifthe device is not providing that data, or if there is something wrong and dataMine is not able to use the data, or???
Title: Re: New dataMine graphing and logging plugin
Post by: Gommasch on June 15, 2012, 03:16:06 am
I have installed the Datamine Graphing and Logging Plugin under apps but none of them show a "Graph" page. Is it not enough to add the app under apps or do I have to add some files manually ?
Done on several different MiCasaVerde Lite's with UI5 1.5.346-391 (they all act the same).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 04:09:19 am
I have installed the Datamine Graphing and Logging Plugin under apps but none of them show a "Graph" page. Is it not enough to add the app under apps or do I have to add some files manually ?
Done on several different MiCasaVerde Lite's with UI5 1.5.346-391 (they all act the same).

You need to follow the instructions on post 264 of this thread. There are some files that need to be manually included at this stage, and you also really need to install a USB stick to store the data (not 100% essential, but advisable).

However, there is no "graph" page as such. You need to go to the plugin properties page where it lists the number of channels being logged, and further down that page there's a link. If you click that, you'll end up on another web page which is actually /dm. This has the GUI for dataMine.

Once MCV update the Vera software (which is due any time now) to fix a timeout problem, then I will be able to install the GUI file automatically as well.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on June 15, 2012, 05:11:13 am
Two separate issues I've encountered...  using UI4.  I admittedly haven't read every page of this thread:

Following the readme.txt in the OP, I've uploaded all files to Vera (including the json.lua file), but I haven't found this file:

Quote
Modify the /etc/init.d.custom-user-startup file to add the line -:
   mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &
This ensures the USB stick is remapped when Vera restarts

I have not yet mounted a USB drive, but I wanted to at least check if I could select variables to log/graph.  I'm unable to, so far:

http://i.imgur.com/OxNM0.png

I've followed all steps from the readme.txt file aside from mounting the USB drive.  Where did I go wrong, do you think?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 05:37:31 am
Following the readme.txt in the OP, I've uploaded all files to Vera (including the json.lua file), but I haven't found this file:

Quote
Modify the /etc/init.d.custom-user-startup file to add the line -:
   mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &
This ensures the USB stick is remapped when Vera restarts
/quote]

This file doesn't always exist - if it's not there, just create it and add the lines as per the instructions. I should admit, it's been a while since I looked at the readme, but it may miss some steps associated with remapping the USB on startup. This is covered in the instructions on http://code.mios.com/trac/mios_datamine.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 05:42:21 am
from what I've read in a previous post, this will not work with UI4. It is UI5 only.
The new version I am referring to.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 05:56:36 am
from what I've read in a previous post, this will not work with UI4. It is UI5 only.
The new version I am referring to.

Yes - you can't (currently) use the UI5 version of dataMine on UI4. I'm assuming that @DeltaNu1142 is using the old version as I haven't posted the actual plugin files except on the marketplace, but it is good to clarify this.

The USB part should be exactly the same as it's really OpenWRT dependant, and not UI dependant.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 06:05:39 am
Chris,

It looks like datamine does not record device variables that are not numbers. I am trying to have datamine graph my thermostat using the modestate to see how long my thermostat is operating in what mode and so far datamine is not recording any values. It shows up with a red ! in the graph webpage.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 06:19:28 am
It looks like datamine does not record device variables that are not numbers. I am trying to have datamine graph my thermostat using the modestate to see how long my thermostat is operating in what mode and so far datamine is not recording any values. It shows up with a red ! in the graph webpage.

Can you provide a bit more information? I'm currently logging the Google Weather wind direction, and it plots fine. The logging itself should go straight into the TXT files in the /dataMine directory just as it comes in from the devices. What dataMine does do though, is it creates a lookup table of text -> numbers so that it can graph numbers, but display text in the axis.

I've not seen a red ! in the graph webpage before. I suspect the best thing may be to capture the transaction between the GUI and Vera if you have Firebug installed? If not, (and maybe as well?) then a bit more info is needed for me to debug this.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 06:28:59 am
I have attached a few screenshots. Logging into vera via ssh and look at the dataMine directory, it does not appear the thermostat mode state is being saved. What I mean by modestate is if the thermostat is idle, heating, or cooling. Let me do some more looking.

- Garrett

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 06:46:33 am
Ah - ok, I thought when you said "graph page" that you mean the graph had an exclamantion mark. Ok, so this exclamation mark in the list means there's been no data received in 24 hours, sounds to be consistant with you not seeing any data. I assume that this is actually changing periodically (sorry for the stupid question!).

I'd look in the log and see if there's any thing "interesting" in there - eg an error report saying dataMine has crashed or something (it could crash in the variable notification code for the mode state, and probably still be ok for the others). It may be that there's something not initialised correctly with the lookup array between numbers and names - I just changed that slightly recently to try and accommodate different versions of json.lua, so I may have messed up "slightly"...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 06:59:32 am
Let me look at the logs and I'll post my findings. I actually have my thermostat turned off in the last day (I have the windows open as it is beautiful outside). This could be why.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 07:21:41 am
Let me look at the logs and I'll post my findings. I actually have my thermostat turned off in the last day (I have the windows open as it is beautiful outside). This could be why.

Could be - the exclamation mark just means no data in the past 24 hours - maybe I should increase the times on these (??). Currently you get the orange warning icon after 1 hour, and the red icon after 24 hours (I'm open to suggestions, and I could possibly make it configurable).

You presumably don't live in the UK - summer seems to be passing us by!  :(

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 07:30:45 am
No live in the US and the summer has been pretty good so far (hopefully it wont be a real hot summer!). I will try and record another device and see what happens. I guess it only records the data once the value changes and not at a set interval?

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 07:36:08 am
I guess it only records the data once the value changes and not at a set interval?
Yes - just the changes are recorded. The only thing I update on an interval basis is the real-time graph (which doesn't work properly in the current version of the UI, but is fixed for the next version).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 15, 2012, 07:44:56 am
Thanks for the info and thank you for a great plugin.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on June 15, 2012, 08:16:27 am
This file doesn't always exist - if it's not there, just create it and add the lines as per the instructions. I should admit, it's been a while since I looked at the readme, but it may miss some steps associated with remapping the USB on startup. This is covered in the instructions on http://code.mios.com/trac/mios_datamine.
Thanks, I'll have a look.
from what I've read in a previous post, this will not work with UI4. It is UI5 only. The new version I am referring to.
Yes - you can't (currently) use the UI5 version of dataMine on UI4. I'm assuming that @DeltaNu1142 is using the old version as I haven't posted the actual plugin files except on the marketplace, but it is good to clarify this. The USB part should be exactly the same as it's really OpenWRT dependant, and not UI dependant.
That's why I wanted to clarify that I'm using the files from the ZIP on the OP of this thread.  The screen shots show UI4, so I proceeded with the install; if you can confirm those files are for UI4, I'll be a bit more confident...  thanks!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 15, 2012, 08:22:19 am
That's why I wanted to clarify that I'm using the files from the ZIP on the OP of this thread.  The screen shots show UI4, so I proceeded with the install; if you can confirm those files are for UI4, I'll be a bit more confident...  thanks!
I'm not sure what OP means but if you mean you're using the ZIP on the first post of this thread, then yes, this is for UI4.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on June 15, 2012, 08:24:59 am
I'm not sure what OP means but if you mean you're using the ZIP on the first post of this thread, then yes, this is for UI4.
OP = original post; thanks!
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 24, 2012, 07:29:59 pm
Chris,
I have been seeing a strange occurrence with the logging. Please take a look at the screenshot... notice how the temp drops and jumps back up while the system in on. It shows this most of the time, but not all... I'd be surprised if the temp is actually doing this. I just started seeing this the past day or so... before then, there was no "drop" in the reading ever.

Also, what is the log interval set to... and can we change it per channel?

thx
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 25, 2012, 02:52:15 am
I have been seeing a strange occurrence with the logging. Please take a look at the screenshot... notice how the temp drops and jumps back up while the system in on. It shows this most of the time, but not all... I'd be surprised if the temp is actually doing this. I just started seeing this the past day or so... before then, there was no "drop" in the reading ever.
To me, this looks "normal"... The temperature is dropping by 2 deg F when the cooling kicks in, and it slowly rises again when the cooling stops. The jumps just after cooling kicks in a just rounding issues or something similar - ie the thermostat/thermometer probably only reads to a 1 deg resolution (or at least that's all Vera is showing) and therefore as temperature drops, it will toggle occasionally. I'm pretty sure that if you sat and watched the numbers on Vera, you'd actually see this data.

One point to note though is that the graphs are splines, so the system draws a curved line between each point. If you turn on data points in the options, then you can see where the "real" points are.


Also, what is the log interval set to... and can we change it per channel?
There is no log interval as such. dataMine binds itself to a data change function in Vera - so it only logs data when the data changes value - not periodically based on time. This is the most efficient for logging, and also from a processing point of view as it only wakes up when there is something happening.

Cheers
Chris

Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 25, 2012, 10:19:22 am
gotcha

Do you know if there is a channel for the Trane stat has the total runtime per day? Since it has a  graph feature which can show you the total daily run time for the past 7 days this means...
a) it storea all 7 days of on/off cycle times and calculates it on the fly each time you view it
or
b) it stores only the final total, per day.
... If it stores the total, it would be great to log this.

Also, do you think you could add a feature on the graph to use external data.... I'd like to add outside temperature data and compare it against run times, on peak and off peak - over time.
Title: Re: New dataMine graphing and logging plugin
Post by: cmbeek on June 25, 2012, 02:54:14 pm
@ Chris: Thanks for this amazing plugin!!!

Installation is indeed not plug&play, but once it is running it is a very usefull plugin.
Actually, I think it should be part of the basic configuration of any vera control centre.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 25, 2012, 04:43:45 pm
Do you know if there is a channel for the Trane stat has the total runtime per day?
Sorry - I don't know anything about the Trane thermostat.

Also, do you think you could add a feature on the graph to use external data.... I'd like to add outside temperature data and compare it against run times, on peak and off peak - over time.
dataMine can log any variables from any devices installed in Vera. You can therefore install the Google Weather plugin to get outside weather, and log the temperature from that (or any other thermometer). Is that what you mean?

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 25, 2012, 05:28:49 pm
Also, do you think you could add a feature on the graph to use external data.... I'd like to add outside temperature data and compare it against run times, on peak and off peak - over time.
dataMine can log any variables from any devices installed in Vera. You can therefore install the Google Weather plugin to get outside weather, and log the temperature from that (or any other thermometer). Is that what you mean?

I'll try it out. Thank you... I did not know about the goog weather plugin :) 

The idea is to log the hourly temps and humidity - and compare it against the cycle times of the AC unit during those hours to see if there are any correlations. Thus, how the outside temps/humidity affect the indoor temp rises and cycle times to drop the home indoor temp 2 degrees.

should be interesting!


Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 25, 2012, 07:34:05 pm
From what I gather, the trane graph data is available to be pulled from the thermostat. There is no zwave command to retrieve the data.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on June 26, 2012, 09:51:56 am
Chris, I like the sound of the new Datamine but am reluctant to go to UI5. Are there plans in the pipeline to make the new version work on UI4?

Keep up the great work

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on June 26, 2012, 12:05:11 pm
Chris, I like the sound of the new Datamine but am reluctant to go to UI5. Are there plans in the pipeline to make the new version work on UI4?

Keep up the great work

Nick

Please see this post:

http://forum.micasaverde.com/index.php/topic,7750.msg75507.html#msg75507

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 26, 2012, 02:10:03 pm
Chris, I like the sound of the new Datamine but am reluctant to go to UI5. Are there plans in the pipeline to make the new version work on UI4?

Hi Nick,
Yes, I'll look into this shortly. It shouldn't be a major issue, but I've been spending time fighting with the java libraries I'm using - I seem to have uncovered a number of bugs in it and this is stopping me releasing a version with the configuration embedded in the GUI.

This of course doesn't impact UI4, it's just a matter of time (and work has taken me abroad again for a week or so  >:()

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on June 26, 2012, 02:39:37 pm
I know the feeling, my life has been rather hectic. As long as it's on the way I'll hold on, this time of year Vera is doing very little

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on June 26, 2012, 02:43:17 pm
I know the feeling, my life has been rather hectic. As long as it's on the way I'll hold on, this time of year Vera is doing very little

Yep - I'll sort out the UI4 version. I just want to get a "real" version running first to avoid having to make too many modifications to the Lua files. The UI side will be the same, so if I can get the backend Lua sorted "once", then I can concentrate on a single UI...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Lonestar10 on July 02, 2012, 03:35:24 pm
Chris,

This looks like a terrific plugin.  Unfortunately (sadly?), some of us don't know a thing about installing codes.  Please let us know when you have a "plug-n-play" version available of UI5 or UI4.

Thanks for your hard work!

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 02, 2012, 06:08:31 pm
Chris,

This looks like a terrific plugin.  Unfortunately (sadly?), some of us don't know a thing about installing codes.  Please let us know when you have a "plug-n-play" version available of UI5 or UI4.

Thanks for your hard work!
I agree it's not the easiest plugin to install. Hopefully once the new Vera software is release, it will be easier (I can remove 2 steps out of the process). I'm also trying to automate/simplify the USB installation, and if I can do that, then it should be completely plug and play on UI5. Hopefully this will all be soon........

UI4 will always be partly manual since there just isn't the same functionality.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 04, 2012, 07:38:06 am
I have managed to put all the configuration into dataMine, such that all the messing around loading libraries, and installing USB devices is now removed / simplified. Once the updated Vera software is available, the installation process will be -:


The variable that is configured in dataMine is the location of the USB drive. Once dataMine knows this, it checks to see if the USB drive is mounted, and if it isn't, it mounts it. This should only happen the first time dataMine starts each time Vera is powered on, but it takes no time at all anyway.

Hopefully this will prove to be reliable, and it will certainly simplify installation (which I agree right now is not the easiest). I'm also putting together a PDF explaining this (and anything else I can think of) rather than rely on sifting through the forum (which is getting very long).

The new version will also fix/improve on a few issues with the GUI, so hopefully the new Vera software will be released soon!

Once I get the UI5 version out, I'll look at putting this back into UI4. This will simplify the UI4 installation, however as there's no app installer for UI4, it will always be a manual process - I can hopefully just make it easier...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 04, 2012, 11:14:07 am
Chris,
Really good to hear - your changes will make it easier for the masses to use and that is always a good thing!

I really like what I've been using so far, it has helped me fine tune my air conditioning usage.

Are there changes to the graphing and such you will be putting in?  Here's some things I think would be useful to most people..

save graph profiles for ease of recovery - save the channels selected, with orientation, time/date range, etc so we can quickly pull it back up
datapoint value markers - show values on the graph
set a graph range - directly set the day/time range. ths current buttons and arrows are a bit tedious ;)

on graph analysis measurements/data... this one is a bit more in depth.
-- options to auto display items like: min/max values, peak-to-peak & trough-to-trough times, etc
-- custom user markers (one or more sets of markers on single channel) can be placed in the graph, and measurements are take for time and/or value difference for eac h set

save data options... ability to save a date/time range of data from within the graph.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 04, 2012, 11:26:44 am
save graph profiles for ease of recovery - save the channels selected, with orientation, time/date range, etc so we can quickly pull it back up
datapoint value markers - show values on the graph
set a graph range - directly set the day/time range. ths current buttons and arrows are a bit tedious ;)

on graph analysis measurements/data... this one is a bit more in depth.
-- options to auto display items like: min/max values, peak-to-peak & trough-to-trough times, etc
-- custom user markers (one or more sets of markers on single channel) can be placed in the graph, and measurements are take for time and/or value difference for eac h set

save data options... ability to save a date/time range of data from within the graph.
Yep - saving graphs is something I intend to put in. It's currently hidden, but there is another pane in the same area as the channel config and options for listing saved graphs.

Other areas I'm looking at are -:

I'll start a list  ;)

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 05, 2012, 04:25:32 pm
Since there's now a new version of Vera code available, I can now release the latest dataMine (it will probably take a day or so for MCV to approve).

This version may work if you haven't upgraded Vera, but you may also find that Vera gets into a loop where it can't install the plugin (it depends on your internet speed). If you've got a good internet connection, you'll be fine - if it's rubish (like mine), then you will need to have upgraded Vera before installing this!

If you've installed the previous version, I would recommend deleting the /www/dm directory before starting the install of the new version. This isn't 100% necessary - it will still work, but it's cleaner as I've changed the structure of the GUI to make it more extensible...

To install -:

Once you've installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I'm not 100% sure. I've been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I'm told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.

I've tested this on a newly installed Vera, so it should work fine, but if you have any issues with the installation, please let me know. I've tried to streamline it as much as possible, but there's always the possibility I've missed something.

I've attached an overview PDF with installation instructions that hopefully helps to answer some questions - it's kind of work in progress, but as I've had a couple of people PM me asking for something like this, I thought I'd put it into a PDF rather than rely on the forums.

If anyone has any questions, let me know and I'll try and answer them.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 05, 2012, 05:51:44 pm
I updated the firmware... it wiped the dataMine channels I had selected?

all my other plugins/settings look untouched.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 05, 2012, 06:10:31 pm
I updated the firmware... it wiped the dataMine channels I had selected?

all my other plugins/settings look untouched.
Strange - I had no issues when I updated the firmware. Now, I am running the new dataMine, which doesn't rely on Vera configuration to mount the USB, so if the firmware upgrade has changed something there, then the new dataMine will (hopefully) solve this.

It's unlikely the channels are "wiped". The configuration is stored in a file on the USB stick which (hopefully) the Vera upgrade shouldn't have affected, so it's possible that something has become "temporarily unhinged".

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 05, 2012, 06:39:25 pm
Chris, can you post the new code? I'd like to try to install today... I leave town Sunday for a week.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 04:07:14 am
Chris, can you post the new code? I'd like to try to install today... I leave town Sunday for a week.
I've attached the Lua file, but I hope that MCV will be able to approve this today (assuming they don't find any issues with it!).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on July 06, 2012, 04:34:38 am
I approved version 0.94. It installed fine for me on 1.5.407 and it looked like it worked correctly.
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on July 06, 2012, 04:51:53 am
I approved version 0.94. It installed fine for me on 1.5.407 and it looked like it worked correctly.

Installed fine on 1.5.408 as well. Working fine.
Thanks again Chris.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 05:26:26 am
I approved version 0.94. It installed fine for me on 1.5.407 and it looked like it worked correctly.
Perfect - thanks Florin.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 06, 2012, 09:52:29 am
I approved version 0.94. It installed fine for me on 1.5.407 and it looked like it worked correctly.

my dataMine is still showing at .92 - autoupdate is enabled

shouldn't it check for updates at login each time?
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on July 06, 2012, 09:53:44 am
shouldn't it check for updates at login each time?

No, it checks once every 24 hours.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 06, 2012, 10:14:26 am
shouldn't it check for updates at login each time?

No, it checks once every 24 hours.

I'd like to suggest a feature request:

Under APPS sections... "Check Now for Updates" button

Having a set 24 hour period makes sense but there needs to be a way to force updates
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 11:12:16 am
Having a set 24 hour period makes sense but there needs to be a way to force updates

I completely agree with this - automatic updates are fine, but there needs to be a manual update button. This is possibly even more necessary if/when the auto update feature is disabled (which I think is going to be an option in an upcoming release?).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 06, 2012, 11:14:54 am
I can confirm that mine has updated to .94. I believe if you hit the reload button it should check for updates on the plugins. It can not see my config or saved graph data. Looking at the advanced tab, everything appears to be ok. But is shows zero for total channels and channels. Going to the website pops up an error saying it can not find the data files.

- Garrett

Error in log:

50      07/06/12 11:23:11.403   luup_log:125: dataMine: Initialising dataMine System <0x2b817680>
50      07/06/12 11:23:11.467   luup_log:125: dataMine: Mounting dataMine storage (/dataMine) <0x2b817680>
50      07/06/12 11:23:11.635   luup_log:125: dataMine: Mount point error: /dev/sda1::/dataMine <0x2b817680>
Title: Re: New dataMine graphing and logging plugin
Post by: Trotsky40 on July 06, 2012, 11:27:04 am
@Chris

Will this work concurrently with using the USB stick for the system logs?

Everything appears to be working and the only difference I can see is the mountType (mine shows the Linux file system EXT3 and your example shows the Windows VFAT). I won't be able to check if it is actaully collecting and storing until late tonight.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 06, 2012, 11:38:16 am
Just wanted to post an update. I looked at the lua dataMine code and did some diagnosing. Found that the variable SetMountPoint needed to be set to the device of the usb drive. This field was empty. Since my usb drive is located at /dev/sda1, I simply set SetMountPoint to /dev/sda1 and reloaded the lua engine and all appears to be working.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 11:45:03 am
Just wanted to post an update. I looked at the lua dataMine code and did some diagnosing. Found that the variable SetMountPoint needed to be set to the device of the usb drive. This field was empty. Since my usb drive is located at /dev/sda1, I simply set SetMountPoint to /dev/sda1 and reloaded the lua engine and all appears to be working.

Yes - this is as per the instructions  ;)

I guess you already had a drive mounted, but the idea here is to eliminate the need to mess around with the custom_startup file etc to mount the USB drive. This is now all done by the plugin (as you've seen).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 11:50:17 am
Will this work concurrently with using the USB stick for the system logs?
No - this mounts a USB drive directly to the /dataMine mount point. I don't know where the log USB function mounts to (maybe MCV can comment). If it is mounted on startup, then it may be possible to point the dataMine configuration into the same drive. I couldn't get the logging to USB to work way back when I started writing dataMine....

Everything appears to be working and the only difference I can see is the mountType (mine shows the Linux file system EXT3 and your example shows the Windows VFAT). I won't be able to check if it is actaully collecting and storing until late tonight.
Yep - this is just the drive format. I formatted mine on Win7 after failing to get the log function to work on Vera. EXT3 is a Linux format.

To be honest - I don't really know what will happen if you use the USB logging function on the same drive. From the previous thread, I guess this is all on a VeraLite, so there's no option to change that...

Let me know how it goes and I can make changes/suggestions as required. ;)

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Trotsky40 on July 06, 2012, 12:15:19 pm
Will this work concurrently with using the USB stick for the system logs?
No - this mounts a USB drive directly to the /dataMine mount point. I don't know where the log USB function mounts to (maybe MCV can comment). If it is mounted on startup, then it may be possible to point the dataMine configuration into the same drive. I couldn't get the logging to USB to work way back when I started writing dataMine....

Everything appears to be working and the only difference I can see is the mountType (mine shows the Linux file system EXT3 and your example shows the Windows VFAT). I won't be able to check if it is actaully collecting and storing until late tonight.
Yep - this is just the drive format. I formatted mine on Win7 after failing to get the log function to work on Vera. EXT3 is a Linux format.

To be honest - I don't really know what will happen if you use the USB logging function on the same drive. From the previous thread, I guess this is all on a VeraLite, so there's no option to change that...

Let me know how it goes and I can make changes/suggestions as required. ;)

Cheers
Chris

I am happy to be the guinea pig.

On a related note, I am guessing the graphing page only works on the same LAN. I am trying it remotely and am getting a "404 not found" error (this why I won't be able to check until later tonight). Just making sure this is correct and not a side effect of using the USB for double duty.
Title: Re: New dataMine graphing and logging plugin
Post by: mcvflorin on July 06, 2012, 12:25:27 pm
I completely agree with this - automatic updates are fine, but there needs to be a manual update button. This is possibly even more necessary if/when the auto update feature is disabled (which I think is going to be an option in an upcoming release?).

Auto-update can be disabled both globally by the plugin author (on apps.mios.com) and locally by the user (on the plugin's control panel).

I agree with the option for manually updating a plugin. The backend supports this by implementing the update_plugin request, it's the UI that's missing the option.

Code: [Select]
http://Vera_IP:3480/data_request?id=update_plugin&Plugin=Plugin_ID
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 12:26:01 pm
I am happy to be the guinea pig.
8)

I'm thinking that we can probably work with the log USB mount once we know where it mounts - you should be able to see this by simply typing "mount" from SSH. We should be able to configure dataMine to put its directory into the same structure reasonably easily, although I may need to make a small mod to the plugin...

On a related note, I am guessing the graphing page only works on the same LAN. I am trying it remotely and am getting a "404 not found" error (this why I won't be able to check until later tonight). Just making sure this is correct and not a side effect of using the USB for double duty.
It should work across lans - the same as UI5 does at least. It won't work outside a firewall - MCV does some reverse link magic to avoid NAT issues in the firewall by havingan outgoing connection initiated by Vera... If you can connect to UI5 without going through cp.mios.com, then you should be able to connect to the dataMine GUI.

I don't think the USB issue would be related to this though - the Lua plugin and GUI are reasonably independant.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 06, 2012, 12:46:24 pm
Chris,

You might want to make the instructions posted either on the first post or in another section so they are visible. I did not read the last post back in regards to the instructions. The instructions are pretty much scattered through out the thread.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 12:49:38 pm
You might want to make the instructions posted either on the first post or in another section so they are visible. I did not read the last post back in regards to the instructions. The instructions are pretty much scattered through out the thread.

Fair point - That's why I created a PDF set of instructions, so that I could start to capture the info. Of course, that's now hidden in 25 pages as well...  ::)

I'll try and update the first post.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 06, 2012, 12:58:55 pm
Thanks Chris. I have no issue with getting my hands dirty and trying to figure out how to get the plugin working. I am just looking out for the inexperienced users.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Trotsky40 on July 06, 2012, 04:16:11 pm
I am happy to be the guinea pig.
8)

I'm thinking that we can probably work with the log USB mount once we know where it mounts - you should be able to see this by simply typing "mount" from SSH. We should be able to configure dataMine to put its directory into the same structure reasonably easily, although I may need to make a small mod to the plugin...


It looks like they both mounted to the same location. dataMine is working perfectly and the system logs appear to working as well.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 06, 2012, 04:21:57 pm
I am happy to be the guinea pig.
8)

I'm thinking that we can probably work with the log USB mount once we know where it mounts - you should be able to see this by simply typing "mount" from SSH. We should be able to configure dataMine to put its directory into the same structure reasonably easily, although I may need to make a small mod to the plugin...


It looks like they both mounted to the same location. dataMine is working perfectly and the system logs appear to working as well.

This is dangerous! You should not have the same device mounted twice as it will cause data corruption.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Trotsky40 on July 06, 2012, 04:43:22 pm

This is dangerous! You should not have the same device mounted twice as it will cause data corruption.

- Garrett

I guess I don't understand how the Linux file system works. I guess I was thinking "dev/sda1" was the drive and "/datamine/" and "/tmp..." were folders on the drive.

(On a side note, I did hit the command twice so the top part and bottom part are just duplicates of the same drive)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 04:49:45 pm
It looks like they both mounted to the same location. dataMine is working perfectly and the system logs appear to working as well.

Well, I'm surprised that can happen, but looking at Garretts note, it sounds "bad".

The easiest way around this is put the dataMine directory under /tmp/log/cmh and then stop dataMine mounting the USB, but logically that's a bit crap. Otherwise we might need another USB drive (somehow!).


I guess I don't understand how the Linux file system works. I guess I was thinking "dev/sda1" was the drive and "/datamine/" and "/tmp..." were folders on the drive.
Unfortunately not. You mount a file system into a folder, but it's a complete file system. I'm just surprised that Linux allows multiple mounts of the same FS to different mount points.


Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Trotsky40 on July 06, 2012, 05:21:50 pm
I think I will move the system logs back to the Vera and leave the dataMine on the USB. Ultimately, sounds like the DM will take up more memory.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 06, 2012, 06:06:37 pm
Chris,
I let the Vera update but now I get this error when opening the graph...


"Error downloading startup data (dataMine channel configuration)."
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 06, 2012, 06:15:28 pm
Chris,
I let the Vera update but now I get this error when opening the graph...


"Error downloading startup data (dataMine channel configuration)."

I found the problem... my SetMountPoint was empty. After putting it in, I no longer get the error.

One thing I just noticed.... in the Config, the channels don't say what room they are associated with. Thus, if you have 5 rooms, with 'Lights' you have no idea which 'Lights' are in what rooms.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 06, 2012, 06:15:55 pm
I let the Vera update but now I get this error when opening the graph...

"Error downloading startup data (dataMine channel configuration)."
Most likely the Lua plugin isn't running. Did you add the SetMountPoint variable to the configuration? Take a look in the log and see if there's anything there...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Will H on July 07, 2012, 10:59:54 am
Chris,
Great plugin.  I have it working except where it comes to adding/removing channels or changing the graph type.  I can only add channels (for those added, graphing works) and only if I have drop down for the room assignment for the dataMine module open (took me a while to stumble on that).  If I go back into the config, channels that are loaded (and showing in the graph) are not highlighted.  Also, if I click on one (turns green) and click on it again, nothing happens (shouldn't it turn yellow per the PDF directions?).  Maybe I have a bit of a dirty install (read below) with left over, manually uploaded files from the old install method?

Some background:
I just got my Vera Lite a couple nights ago and immdiately hooked it to my two honeywell thermostats.  I was hoping to get some data logging (as the Trane z-wave thermostat has some kind of good logging) and your plugin was luckily at the top of list because of your recent update.  Unfortunately, I clicked the Help tab on the plugin install window which took me to the old install directions (code.mios.com/trac/mios_datamine).  I already had a USB drive for Vera logs (Setup>Logs), I uploaded the six files in the old directions (e.g. L_DataMine1.lua) and I set the SetMountPoint but I couldn't get channels added.  I uninstalled your plugin, disabled Vera logging to USB, then followed the new directions: install plugin, format the drive Windows, plug in the drive, configure setpointmount (confirmed path in ssh) and rebooted Vera to restart LUA (just to be sure).

Will
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on July 08, 2012, 04:16:04 am
Hi

I have data in an rrd dB (which has been capturing energy usage and room temperatures for quite a while now ) on another server on my home network.

Any ideas how I can maybe make that accessible to datamine?

//192.168.1.111/Usr/rrdtool.rrd
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 08, 2012, 05:54:25 am
Great plugin.  I have it working except where it comes to adding/removing channels or changing the graph type.  I can only add channels (for those added, graphing works) and only if I have drop down for the room assignment for the dataMine module open (took me a while to stumble on that).
I'm not sure what you mean by "drop down for the room assignment"? dataMine should depend on any other settings in UI5.

If I go back into the config, channels that are loaded (and showing in the graph) are not highlighted.  Also, if I click on one (turns green) and click on it again, nothing happens (shouldn't it turn yellow per the PDF directions?).
Just to confirm - if you click on a variable, it turns green, clicking again doesn't change anything (??), and if you close the config and open it again, then this isn't green any more?
However, if you go to the GUI, then all the channels that you've selected are showing in the list and are graphing correctly?

The GUI and the config use the same data for showing channels, so if the GUI is showing data, then the config should also be showing things in green, so these two points confict (which is why I'd like to confirm I understand what you see).

It's possible that there's a mixed up install - I would have expected the install from apps.mios.com to have overwritten any files loaded separately though.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 08, 2012, 06:02:57 am
I have data in an rrd dB (which has been capturing energy usage and room temperatures for quite a while now ) on another server on my home network.

Any ideas how I can maybe make that accessible to datamine?

What do you mean "accessible"? You could probably convert the data over to the dataMine file format reasonably easily (dataMine just uses CSV files), but if you mean is it possible for dataMine to read the rrd data, then that's a different story.

If you want to convert existing data into a format that dataMine can read, I can tell you the file name/format in more detail...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on July 08, 2012, 09:25:36 am
..... if you mean is it possible for dataMine to read the rrd data, then that's a different story.

Does it have a happy ending? The external file could potentially be a csv or XML on this external server (same subnet etc.) ?

If you want to convert existing data into a format that dataMine can read, I can tell you the file name/format in more detail...

There seems to be ways to do rrd to csv so that might help me/us?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 08, 2012, 09:37:50 am
Does it have a happy ending? The external file could potentially be a csv or XML on this external server (same subnet etc.) ?

dataMine does store it's data as a CSV file - actually, a number of CSV files (one per week). dataMine only records data when it changes - of I remember correctly, rrd records data at a specific interval.  The dataMine GUI doesn't read the CSV files directly - the data gets served up via the plugin which can sub-sample if a large period is asked for, or can serve up processed data.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: NovaIceman on July 08, 2012, 11:58:35 pm
I am using UI4 and I believe i have followed all the install directions correctly, can you help with this error.

Quote
50   07/08/12 20:52:05.852   luup_log:48: dataMine: Reinitialising configuration structure <0x402>
50   07/08/12 20:52:05.857   luup_log:48: dataMine: Startup complete <0x402>
50   07/08/12 20:52:06.081   luup_log:49: dataMine: Initialising Data Mining System <0x402>
50   07/08/12 20:52:06.083   luup_log:49: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>

thanks
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 09, 2012, 03:27:53 am
I am using UI4 and I believe i have followed all the install directions correctly, can you help with this error.

Quote
50   07/08/12 20:52:05.852   luup_log:48: dataMine: Reinitialising configuration structure <0x402>
50   07/08/12 20:52:05.857   luup_log:48: dataMine: Startup complete <0x402>
50   07/08/12 20:52:06.081   luup_log:49: dataMine: Initialising Data Mining System <0x402>
50   07/08/12 20:52:06.083   luup_log:49: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>

These entries are from two separate copies of dataMine - the device numbers are different (48 and 49), so you have two copies of the device installed somehow.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: JOD on July 09, 2012, 10:19:13 am
@Chris,
Awesome plugin. Kudos to you.

Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 09, 2012, 11:18:52 am
@Chris,
Awesome plugin. Kudos to you.

Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.

Unfortunately this won't work  :(  MCV use a reverse link to connect to theirs server to avoid NAT issues in firewalls and DNS problems.  Remote access isn't impossible to configure, but it requires setting up a bunch of stuff that's not so simple (Dynamic DNS and NAT in your firewall)...

Chris

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: JOD on July 09, 2012, 03:34:27 pm
Oh well.....
I can review the .txt file if I really need to check on something, just not as much fun as watching the graph.
Title: Re: New dataMine graphing and logging plugin
Post by: huib on July 14, 2012, 04:23:26 am
@chris,


i have got your plugin working and is helpfull

can you advise how to delete one (or two) itmes to be logged

if i select them the line will tunr green
if i de select they turn yellow

but the line is still visible on the graph

so my question is how delete the line in the graph to clear up the screen

thanks

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 14, 2012, 06:23:28 am
if i select them the line will tunr green
if i de select they turn yellow

but the line is still visible on the graph

so my question is how delete the line in the graph to clear up the screen

Unfortunately, this isn't possible at the moment - I'll add this shortly. The reason I originally didn't do this is I wanted to avoid accidentally deleting the data, but I can work around this and clearly it's a necessary feature.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 14, 2012, 11:12:37 am
Chris,
I came home after a 5 day work trip and found all Channels had been wiped from the config. This is what happened after the update. Except, this time, all the settings under 'Advanced' were there.

So, I used the built-in uninstall from the Apps tab. Then I looked at the LUA files and found there were still two dataMine XMLs and the dataMine folder is still under <root>
... it looks like maybe the uninstall is not really removing things?

One thought... I never fully reversed the initial manual install... I just installed the new code from the installer once it was posted. Is there anything that might be conflicting?

I reinstalled (using Vera's Apps tab) and reset the channels... so I'll let you know what happens.

Also, I noticed one channel that seems to have an issue. Under "Device: Front Door Lock" it shows all the normal items plus... "Service: urn:upnp-org:serviceId:HVAC_FanOperatingMode1"


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 14, 2012, 02:26:15 pm
So, I used the built-in uninstall from the Apps tab. Then I looked at the LUA files and found there were still two dataMine XMLs and the dataMine folder is still under <root>
... it looks like maybe the uninstall is not really removing things?
I don't know about the XML files - that's all part of how Vera does things, but the dataMine directory will not be removed. Firslty, when an app is removed, it doesn't run an uninstall function in the apps Lua, so there's no way or doing an app specific uninstall. Also, it's not something I would typically do as the data in the dataMine directory should still be of use even if the app gets uninstalled.

It is however strange that you're loosing your setup. I've never had any problem and I wonder if there's some sort of issue with your USB? The configuration is stored in the dataMineConfig.json file in the /dataMine directory, so it would be interesting to look in there and see what's there. It should also be backed up each time it changes, so you should be able to restore the configuration.


Also, I noticed one channel that seems to have an issue. Under "Device: Front Door Lock" it shows all the normal items plus... "Service: urn:upnp-org:serviceId:HVAC_FanOperatingMode1"
While this could be an issue with dataMine, it may also be that this service is, strangely, under this device. I have seen the occasional "strange" services under devices that you think shouldn't have such a service. For example, I've notices that with UI5, all devices have a "VideoURL" variable, which is a bit weird for devices like a temperature sensor!  You should be able to see this under the Advanced tab.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: bfromdruten on July 14, 2012, 02:33:46 pm
Chris,
Aarons message now makes something clear. Last couple of months I have had the same. Suddenly all channels disappeared and no data. I think I have had this 2 or 3 times. Last couple of days no problems, but currently I'm running with default dataMine folder configuration where as previously I changed the folder location. Maybe this helps.

Thanks,
Bernard
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 14, 2012, 02:42:50 pm
Hi Bernard,
Thanks - I can only think that there's some issue with mapping of the USB drive. There's nothing in dataMine that deletes files, so the issue must be somewhere else. I think (hopefully) that the new version of dataMine, which does the USB mapping internally, should avoid some of these issue. Have you had any issues siince the new version was installed?

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 14, 2012, 02:54:45 pm
my system reboots every night, so I suspect if it was a USB mounting issue it would happen every night... but it does not. And, I also have the Vera logs going to the USB drive... they seem fine.

Here's what is in my json file...

Quote
{"Variables":[{"Type":0,"Device":3,"Service":"urn:upnp-org:serviceId:TemperatureSensor1","FirstRec":1342280443,"Archive":"Thermostat - CurrentTemperature","Name":"Thermostat - CurrentTemperature","LastVal":"77","Logging":1,"Variable":"CurrentTemperature","LastRec":1342286562,"Lookup":[]},{"Type":1,"Device":4,"Service":"urn:micasaverde-com:serviceId:DoorLock1","FirstRec":1342286605,"Archive":"Front Door Lock - Status","Name":"Front Door Lock - Status","LastVal":0,"Logging":1,"Variable":"Status","LastRec":0,"Lookup":[]},{"Type":0,"Device":55,"Service":"urn:upnp-org:serviceId:TemperatureSensor1","FirstRec":1342280740,"Archive":"Temperature - CurrentTemperature","Name":"Temperature - CurrentTemperature","LastVal":"82","Logging":1,"Variable":"CurrentTemperature","LastRec":1342280740,"Lookup":[]},{"Type":0,"Device":58,"Service":"urn:micasaverde-com:serviceId:HumiditySensor1","FirstRec":1342280740,"Archive":"Humidity - CurrentLevel","Name":"Humidity - CurrentLevel","LastVal":"70","Logging":1,"Variable":"CurrentLevel","LastRec":1342284340,"Lookup":[]},{"Type":1,"Device":3,"Service":"urn:micasaverde-com:serviceId:HVAC_OperatingState1","Alpha":1,"FirstRec":1342280443,"Archive":"Thermostat - ModeState","Name":"Thermostat - ModeState","LastVal":"Idle","Logging":1,"Variable":"ModeState","LastRec":1342284922,"Lookup":{"Idle":2,"Cooling":1}}]}
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 14, 2012, 03:13:35 pm
my system reboots every night, so I suspect if it was a USB mounting issue it would happen every night... but it does not. And, I also have the Vera logs going to the USB drive... they seem fine.

Here's what is in my json file...
Thanks Aaron,
The json looks fine, but I guess this is from after your reinitialisation?

When your Vera reboots, what actually happens? Does it completely restart? I'd tend to agree that if it does, then if there was a problem remounting the drive, it should happen every night.

However, I can't see any reason, or way, that dataMine can remove these files. It doesn't actually delete any data, so if all data is being removed, and the dataMine directory is completely empty, then I think the issue must be somewhere else.

If you're saving all your log files, I don't know if there any way to get all the dataMine log entries before an after the event? I don't know if they can tell me much, but if it was easy, it might be interesting. If dataMine starts, and doesn't find the config file, then it logs "Unable to open config file for read", and this should be followed by "Reinitialising configuration structure". Maybe by looking for these strings we might be able to find a trend that may help?

Sorry this isn't as helpful as you'd like...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 14, 2012, 03:23:31 pm
Lets give it some time and see what happens - since I just reinstalled.

Also, on that HVAC channel being in my Lock... it is actually in both places on dataMine. I never saw it under Lock before... I think this is a 'new' occurrence.

any idea when the new Vera firmware will fix the USB mount issue?  I think they said it would be ready like 2 weeks ago ;)
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on July 16, 2012, 02:09:50 pm
The plug is working on Ui5 last firmware 408.

I can add device but I m not more able to remove the variable to the graph. If i click it come yellow, bit I don t see the flag in the box, so I cannot also not remove it.

Note: wonderful plug in

Thank you for your help
regards,
Matteo
Title: Re: New dataMine graphing and logging plugin
Post by: huib on July 16, 2012, 04:02:18 pm
@chris
like indicated also by aaron i loose all data when i reboot the system.
it looks to me that the settings of the plug are cleaned and wipeout during the rebooting
the data written on the usb remains

rgds
huib
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 16, 2012, 05:26:57 pm
like indicated also by aaron i loose all data when i reboot the system.
it looks to me that the settings of the plug are cleaned and wipeout during the rebooting
the data written on the usb remains

The plugin only reads the data from the USB - it's not stored anywhere else. So, the settings aren't being "cleaned or wiped out", it may however be that dataMine can't read them when it starts. The question must be how the reboot process is working, and in what order the USB is being mounted (ie before or after dataMine starts).

When you say "the data written on the usb remains" - what do you mean? How have you checked this (on a Windows/Mac/Linux computer, or via Veras SSH interface?).

What version of dataMine are you using? Assuming it's the latest, is it configured with the SetMountPoint variable? It would also be worth looking at the "mountLocation" variable (in the Advanced tab) - what does this say?

Hopefully the above questions will help me find out what's up...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 16, 2012, 05:30:03 pm
I can add device but I m not more able to remove the variable to the graph. If i click it come yellow, bit I don t see the flag in the box, so I cannot also not remove it.
Hi Matteo,
I'm really sorry, but I don't really understand what you're doing. What "flag in the box" do you mean, and what do you mean by "I cannot also not remove it".

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 16, 2012, 08:07:08 pm
like indicated also by aaron i loose all data when i reboot the system.
it looks to me that the settings of the plug are cleaned and wipeout during the rebooting
the data written on the usb remains

The plugin only reads the data from the USB - it's not stored anywhere else. So, the settings aren't being "cleaned or wiped out", it may however be that dataMine can't read them when it starts. The question must be how the reboot process is working, and in what order the USB is being mounted (ie before or after dataMine starts).

When you say "the data written on the usb remains" - what do you mean? How have you checked this (on a Windows/Mac/Linux computer, or via Veras SSH interface?).

What version of dataMine are you using? Assuming it's the latest, is it configured with the SetMountPoint variable? It would also be worth looking at the "mountLocation" variable (in the Advanced tab) - what does this say?

Hopefully the above questions will help me find out what's up...

Chris

Since my post of this happening, and me setting it back up... it has been fine. As since I reboot every night I doubt the reboot could be causing the problem.

I'm on the latest firmware and version of dataMine. All the Advanced info is correct and data is logging / graphing.
Title: Re: New dataMine graphing and logging plugin
Post by: bfromdruten on July 17, 2012, 05:04:54 pm
Guys, also no problems after last version and moving my dataMine mount to the standard folder. Can it be this problem is related to a plugin upgrade instead of a reboot? I was thinking if you don't use the default setting the software upgrade would reset them. Don't know however if this is true. At least...currently running ok.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 17, 2012, 05:15:17 pm
Guys, also no problems after last version and moving my dataMine mount to the standard folder. Can it be this problem is related to a plugin upgrade instead of a reboot? I was thinking if you don't use the default setting the software upgrade would reset them. Don't know however if this is true. At least...currently running ok.
That sounds like good news...

The upgrade shouldn't reset the configuration. The directory/mapping is stored in a variable which shouldn't get reset, and all other configuration is stored in the json file in the dataMine directory.

Anyway, it sounds promising, so fingers crossed that we're sorted...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: teonebello on July 18, 2012, 02:50:30 pm
I can add device but I m not more able to remove the variable to the graph. If i click it come yellow, bit I don t see the flag in the box, so I cannot also not remove it.
Hi Matteo,
I'm really sorry, but I don't really understand what you're doing. What "flag in the box" do you mean, and what do you mean by "I cannot also not remove it".

Cheers
Chris

How can I remove a variable from the graph?

Thank you
Matteo
Title: Re: New dataMine graphing and logging plugin
Post by: Jacob on July 18, 2012, 03:31:05 pm
Hi Chris,

I appreciate your work and with great pleasure I monitor my house and it's surroundings with the use of your plug-in's datamine and OW-server.
I make use of an IMac and a IPad with SQremoteHD

Do you think it is possible to extend the dataMine plug-in with full screen user predefined trends without the selection box.
It might be convenient to be able to call these from the SQremote web display's

In addition, a bit off topic I would like to draw your attention to the 1-wire weather instuments
http://www.hobby-boards.com/store/products/Anemometer-ADS.html
http://www.hobby-boards.com/store/products/Rain-Gauge-w%7B47%7DCounter.html
Would these show their data when connected to the OW-Server or can you add them to the plug-in?

Regards,
Jacob
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 18, 2012, 04:12:50 pm
How can I remove a variable from the graph?
Sorry - you can't at the moment. I will add this soon.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 18, 2012, 04:26:19 pm
Do you think it is possible to extend the dataMine plug-in with full screen user predefined trends without the selection box.
It might be convenient to be able to call these from the SQremote web display's
If I understand you correctly, then it would be possible to add functionality that allowed displaying a graph without the selection box. There are a couple of ways that this could be done, but I'm not sure that it would necessarily work with SQRemote. I've got SQRemote on an IPod Touch, but I've not tried to display dataMine with it - have you tried this?

However, the downside is that if you want to do this remotely, then that's a different issue since dataMine can't easily get its data outside your local network.

Regarding the OW questions - I've seen the HobbyBoards stuff and keep thinking about installing a weather system, but haven't to date. The OWServer will likely "work" with them, but it would require some work to make it read the correct data. This could probably be done reasonably easily...
If you have any other questions on this, it might be better to move the conversation over to the OWServer thread (http://forum.micasaverde.com/index.php/topic,8381.0.html).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Jacob on July 18, 2012, 05:33:39 pm
If I understand you correctly, then it would be possible to add functionality that allowed displaying a graph without the selection box. There are a couple of ways that this could be done, but I'm not sure that it would necessarily work with SQRemote. I've got SQRemote on an IPod Touch, but I've not tried to display dataMine with it - have you tried this?
I imagine to configure and store specific trends that can be retreived from the Vera e.g. by means of a url
In this way one can support and clarify the normal data in SQremote

Indeed this works both on my Iphone and Ipad and you are correct only within the personal network however I concider this no issue

The small display size and difficulty to make the selections makes the datamine trends on these machines not very convenient to work with that's why the preconfigured trends
Jacob
Title: Re: New dataMine graphing and logging plugin
Post by: chixxi on July 20, 2012, 12:01:53 pm
I use you plugin on a vera 3 and everything just runs absolutely perfect, no problems at all! Just wanna say THANK YOU (!!) for this amazing plugin!
Title: Re: New dataMine graphing and logging plugin
Post by: Gommasch on July 21, 2012, 04:02:00 pm
When I try to view my graphs from a remote location I get the following error:

Not Found
The requested URL /dm/index.html was not found on this server.

Is it not possible to view the graphs that way ?
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 21, 2012, 04:21:43 pm
This has been answered a few pages back:

http://forum.micasaverde.com/index.php?topic=7750.msg78874.msg#78874

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 22, 2012, 12:59:03 pm
I was looking forward to trying this plugin, but I'm seeing the following error in the notification area at the top of UI5:

dataMine[81] : Mount point error: /dev/scsi/host0/bus0/target0/lun0/part1::

I did enter the SetMountPoint info as instructed (see attachment).  The whole text that is partly cutoff is "/dev/scsi/host0/bus0/target0/lun0/part1"

Here is what the SSH session shows:

Quote
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.3M      6.3M         0 100% /rom
tmpfs                    14.9M    316.0K     14.6M   2% /tmp
/dev/mtdblock/4           8.9M      7.3M      1.6M  82% /overlay
mini_fo:/overlay          6.3M      6.3M         0 100% /
/dev/scsi/host0/bus0/target0/lun0/part1
                        503.9M     21.8M    456.5M   5% /tmp/log/cmh
/dev/loop/0               4.6M      4.6M         0 100% /mios
/dev/scsi/host0/bus0/target0/lun0/part1
                        503.9M     21.8M    456.5M   5% /dataMine



The USB device appears to be working and the location description appears to be accurate.  Any ideas why I'm getting this error?  Thanks very much.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 22, 2012, 02:21:39 pm
Here is what the SSH session shows:

Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.3M      6.3M         0 100% /rom
tmpfs                    14.9M    316.0K     14.6M   2% /tmp
/dev/mtdblock/4           8.9M      7.3M      1.6M  82% /overlay
mini_fo:/overlay          6.3M      6.3M         0 100% /
/dev/scsi/host0/bus0/target0/lun0/part1
                        503.9M     21.8M    456.5M   5% /tmp/log/cmh
/dev/loop/0               4.6M      4.6M         0 100% /mios
/dev/scsi/host0/bus0/target0/lun0/part1
                        503.9M     21.8M    456.5M   5% /dataMine


The problem is that the same drive is mounted twice - once for dataMine, and once for the logs. This is actually "quite bad" as it can cause some unpredictable behavour. I would be tempted to either use another USB stick in the other socket (if possible), or disable USB logging, or disable dataMine...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 22, 2012, 02:51:11 pm
Chris,

Thanks very much for the quick response.  I guess I was under the mistaken assumption that I needed to enable USB logging for Vera to recognize the device.  I have disabled USB logging and confirmed through SSH:

Quote
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.3M      6.3M         0 100% /rom
tmpfs                    14.9M      5.0M      9.9M  34% /tmp
/dev/mtdblock/4           8.9M      7.2M      1.6M  82% /overlay
mini_fo:/overlay          6.3M      6.3M         0 100% /
/dev/loop/0               4.6M      4.6M         0 100% /mios
/dev/scsi/host0/bus0/target0/lun0/part1
                        503.9M     21.3M    457.0M   4% /dataMine

Unfortunately, UI5 is showing the same error (after reboot):  "dataMine[81] : Mount point error: /dev/scsi/host0/bus0/target0/lun0/part1::"

Should I try to start over?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 22, 2012, 05:24:12 pm
Unfortunately, UI5 is showing the same error (after reboot):  "dataMine[81] : Mount point error: /dev/scsi/host0/bus0/target0/lun0/part1::"
The error should display the value of the "SetMountPoint" variable after the :: so this is the issue (not sure why though!). Can you take a look in the log and see anything from dataMine - basically anything after the "Initialising dataMine System" entry.

You could try removing the app and reinstalling, but I'm not really sure this will help.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 22, 2012, 06:27:56 pm
This is what is in the log:

Code: [Select]
50 07/22/12 16:22:22.624 luup_log:80: dataMine: Initialising dataMine System <0x402>
06 07/22/12 16:22:22.627 Device_Variable::m_szValue_set device: 80 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x930150/NONE duplicate:1 <0x402>

AND

Code: [Select]
03 07/22/12 16:22:24.595 JobHandler_LuaUPnP::Run ready 36 devices installation 21495 missing /etc/cmh-lu//I_DataMine1.xml  version *1.5.408*  allow_evt: 1 allow_ud: 1 upnp 1 netpnp: 1 reprovision: 0 changed: 0 pMem 0xa8c000 start 7860224 diff: 3198976 <0x400>
06 07/22/12 16:22:24.985 Device_Variable::m_szValue_set device: 80 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: /dev/scsi/host0/bus0/target0/lun0/part1 now: /dev/scsi/host0/bus0/target0/lun0/part1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
06 07/22/12 16:22:24.986 Device_Variable::m_szValue_set device: 80 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: ext3 now: ext3 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
02 07/22/12 16:22:25.140 ZWaveJobHandler::SetDeviceStatus XXX2 0/0/0/0 <0x803>
50 07/22/12 16:22:25.921 luup_log:80: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>
06 07/22/12 16:22:25.921 Device_Variable::m_szValue_set device: 80 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x930150/NONE duplicate:1 <0x402>
06 07/22/12 16:22:25.922 Device_Variable::m_szValue_set device: 80 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
50 07/22/12 16:22:25.923 luup_log:80: dataMine: Reinitialising configuration structure <0x402>
50 07/22/12 16:22:25.967 luup_log:80: dataMine: Startup complete <0x402>
50 07/22/12 16:22:25.972 luup_log:81: dataMine: Initialising dataMine System <0x402>
06 07/22/12 16:22:25.975 Device_Variable::m_szValue_set device: 81 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x930150/NONE duplicate:1 <0x402>
06 07/22/12 16:22:25.976 Device_Variable::m_szValue_set device: 81 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
06 07/22/12 16:22:26.851 Device_Variable::m_szValue_set device: 81 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: /dev/scsi/host0/bus0/target0/lun0/part1 now: /dev/scsi/host0/bus0/target0/lun0/part1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
06 07/22/12 16:22:26.852 Device_Variable::m_szValue_set device: 81 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: ext3 now: ext3 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
50 07/22/12 16:22:28.784 luup_log:81: dataMine: Mounting dataMine storage () __LEAK__ this:139264 start:139264 to 0xad2000 <0x402>
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 22, 2012, 08:46:18 pm
It may be working now, at least I'm no longer getting an error message.  I uninstalled the app, manually removed the DataMine *.xml and related files via SSH, and reformatted the USB drive - a couple of times.  The USB drive would not mount without enabling Vera's USB logging; I don't think that it got formatted by Linux until I did that.  In any case, no current error messages.

Edit:  Unfortunately, still does not appear to be working as I'm getting zero channels.
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 23, 2012, 11:22:02 am
I have it working now.  I believe I was confused by the old instructions and the new instructions, and what needed to be done and not done.  After FAT32 formatting the USB drive and plugging it into Vera, it needed to be mounted.  I followed the instructions here (http://code.mios.com/trac/mios_datamine) where it says the following:

    mkdir /dataMine
    mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine

After that, I was able to set the mount point variable.

I was also confused about using the tool and setting a channel.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 23, 2012, 11:53:00 am
I'm glad it's working now. However, you shouldn't have needed to mount the drive - the new version should sort this out for you/ All you should have to do is format the USB drive (on Windows is fine), then install the app and set the SetMountPoint variable. DataMine will then do the "mount" command for you when it starts. You will still get an error the first time dataMine starts (ie before you set the "SetMountPoint" variable), but after that, it should sort it out for you without messing around with manual mounting etc.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: kkl on July 23, 2012, 03:33:10 pm
Quote
you shouldn't have needed to mount the drive

Is there a timing issue?  The current instructions say to install the app before inserting the USB drive.  When I did that, Vera could see the drive (dmesg command), but it never got mounted.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 23, 2012, 03:52:14 pm
Quote
you shouldn't have needed to mount the drive

Is there a timing issue?  The current instructions say to install the app before inserting the USB drive.  When I did that, Vera could see the drive (dmesg command), but it never got mounted.

Maybe. What I'd intended is that until the SetMountPoint variable was configured, the app shouldn't do anything in which case it shouldn't matter what order things were done so long as the mount point was set last. However, this isn't 100% true and I suspect there may be an issue in this area - I'll look at modifying the startup code to prevent this.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 26, 2012, 02:45:29 pm
Chris,
my channels are blank again... how can we fix this?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 27, 2012, 07:38:13 am
Chris,
my channels are blank again... how can we fix this?
The first question is to look in the /dataMine directory and see what's in the dataMine Config.json file. Is it empty? If not, please can you email it to me (PM me I guess). All configuration is stored in this file, and I have seen it being corrupted in the past, so it would be good to try and understand this.

The next thing to do is to use a backup configuration. dataMine saves backups every time the configuration changes - these are stored in the /dataMine directory and you should just have to rename the latest backup to dataMine Config.json.  This should get you moving again without having to reconfigure dataMine.

The big question is what's happening - if you can look at the dataMine Config.json as soon as you notice this happening - ie before doing any reconfiguration - that may be of use (although dataMine may overwrite the file if it's corrupt). Looking in the log file to see if there's anything logged in there would also be good.  I'll take a look at the code and see if I can add some more logging around the startup so that if/when it does happen, at least we can start to find out why.

Cheers
Chri
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 27, 2012, 11:00:08 am
the dataMine folder is nearly empty, see attached JPG.

I'm uninstalling and removing all files/folders, LUA, etc.... then reinstalling.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 27, 2012, 11:14:54 am
the dataMine folder is nearly empty, see attached JPG.

I'm uninstalling and removing all files/folders, LUA, etc.... then reinstalling.

Something very strange is happening then, and I don't think it has anything to do with dataMine directly. dataMine doesn't delete any files, so something has deleted everything from your USB drive. I don't really know what could cause this.

One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 27, 2012, 11:48:29 am
One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

I'm not manually copying anything to the dataMine folder.

Yes, I do have the Vera logs also saving to the USB drive where \dataMine is ... but this should not matter, unless Vera has a very strange bug - since the only some of the dataMine files go missing.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 27, 2012, 11:54:26 am
One thing that's interesting is that I see the log files appearing in here. How is this happening, or are you manually copying them over? You don't have USB log files enabled and pointing to the same USB stick do you?

I'm not manually copying anything to the dataMine folder.

Yes, I do have the Vera logs also saving to the USB drive where \dataMine is ... but this should not matter, unless Vera has a very strange bug - since the only some of the dataMine files go missing.

This definately does matter - it is not a good idea and can cause strange things to happen. This will effectively mean that you have 2 mount point for the same USB drive, and this will cause unpredictable results (as you're seeing). There have been some other posts on this issue. Looking at the image you posted, there are no other logs, and the log file that is shown is only 700kb, which means it was generated quite recently, so I suspect that this is also going "astray".

You should either disable USB logging, or install a separate USB stick for use with dataMine. This is almost certainly the cause of your problems.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 27, 2012, 12:28:53 pm
I don't understand the problem... the drive should be mounted once, but the OS... then the app(s) can access it.  Once the OS mounts it, the app(s) should see it just like any other drive.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 27, 2012, 12:39:21 pm
I don't understand the problem... the drive should be mounted once, but the OS... then the app(s) can access it.  Once the OS mounts it, the app(s) should see it just like any other drive.
No - both dataMine and Vera USB logging are mounting the drive. Effectively, you now have the same file system mounted into two places, which can cause "bad things" to happen. dataMine mounts to the /dataMine directory, and USB logging mounts to /tmp/logs (or maybe /tmp/logs/cmh - I don't have this enabled, so I'm not sure). It's therefore not easy to have dataMine and logging using the same drive (although it's also not impossible, but the directory structure would have to change).

I recommend you run SSH, or the WinSCP terminal, and type "mount". If there are two mount points for the same drive, as I suspect there is as both Vera logging and dataMine will be mounting it, then you need to change something to eliminate this.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 27, 2012, 12:59:16 pm
here's my mount points...

rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sda1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime,errors=continue,barrier=0,data=ordered)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /dataMine type ext3 (rw,relatime,errors=continue,barrier=0,data=ordered)

... so there are two mount points with /dev/sda1 -- I don't understand why this is an issue... why can't two apps use the same mount point, but different folders in that mount point?  Windows can do this, no problem.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 27, 2012, 01:13:37 pm
... so there are two mount points with /dev/sda1 -- I don't understand why this is an issue... why can't two apps use the same mount point, but different folders in that mount point?  Windows can do this, no problem.

Not really - Windows has a completely different concept. When mounting a drive under Windows, you simply refer to the drive (A: or whatever). The drive becomes a single entity.  Under Linux, you create a directory as the mount point, and then mount the drive to that directory. You've now got the same drive mounted to two places. Under the Windows concept, it's like having the same USB drive mounted as drive A:, and drive B: (which under Windows, you can't do).

I'm not a linux expert, but from what I've read, this produces unstable results. You end up with the Linux kernal trying to manage the USB stick as though it was two drives, and it's not - "bad stuff" then happens. I think most linux systems prevent this (although I've not tried on my Linux box, for obvious reasons), but OpenWRT doesn't.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 27, 2012, 02:16:17 pm
You can not have two mount points pointing to the same device. This will cause data corruption.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 30, 2012, 09:11:06 pm
You can not have two mount points pointing to the same device. This will cause data corruption.

I'm not sure the data is corrupting ... the dataMine channels just disappeared again and I looked in the Vera log and found...

Quote
2012-07-30_15:00:40 [is_usb_plugged] WARNING: Found UNPARTITIONED USB STICK Connected
2012-07-30_15:00:41 [erase_part_schema] WARNING: We'll RAW ERASE disc: /dev/sda, having: 984784 sectors

It looks like a problem upon reboot - like Vera has a problem with the drive... a vera bug?

BUT then dataMine must have automatically remounted and created the files it needed since at 3:01pm is the creation date of mount_tests.

So, I'm turning off Vera logging to USB to see if there is an issue... either way, I think this needs to be fixed/addressed. We need to have a way to use the USB drive for more that one thing... like logging from two different apps.  I suspect Linux can do this. Maybe dataMine can use an existing mount point that Vera uses instead of trying to mount its own?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 31, 2012, 04:43:49 am
Hi Aaron,
As per the previous comments about this, you can't run both logging and dataMine on the same mount point as things currently stand as this means the drive will be mounted twice. This is known the cause problems. I suspect this log text came from Veras USB logging functionality - it's detected data corruption in that file system, and it's reformatted the drive. When dataMine starts, it reinitialises the configuration files (with nothing being logged obviously).

It's not possible to have 2 mount points to the same drive. As I think I've said previously, the only option would be to put the dataMine directory under the log directory, which isn't exactly intuitive as it would be /tmp/log/cmh/dataMine. This would solve the problem as I would then not need to mount the USB, but would be a bit of a strange place to put the directory. You'd then be totally reliant on Veras USB logging for dataMine to work. If this gets turned off, dataMine will stop working. I can however easily add the option to do this.

The other option is to use a second USB stick for dataMine. In my opinion, this is cleaner, but does of course require a new USB drive.

What do you think?

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on July 31, 2012, 10:08:31 am
I think providing the option to log dataMine using Vera's log mount point is best... even though it might not be intuitive, the more I
'm learning about Linux, that would be par for the course... since most things about Linux's file structure is in no way intuitive.

Another reason in favor of this is many of us (including myself) use Insteon, and we already have the other USB port in use for that.

Thank you!
Title: Re: New dataMine graphing and logging plugin
Post by: AbNa on August 09, 2012, 07:31:24 am
Hi,
I seem to have a similar problem on one of my units, the channels are cleared if i reload the engine. It seems the usb does not mount correctly, I have set the correct mountpoint but it does not seem to work. The usb stick is formatted on windows(FAT32). USB logging is not enabled.

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)

Any suggestions?
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on August 14, 2012, 11:00:29 am
@Chris
Any new updates to test?



Hi,
I seem to have a similar problem on one of my units, the channels are cleared if i reload the engine. It seems the usb does not mount correctly, I have set the correct mountpoint but it does not seem to work. The usb stick is formatted on windows(FAT32). USB logging is not enabled.

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)

Any suggestions?

This fixed my problem when I had something similar...

uninstall dataMine
remove usb drive from Vera
reformat your USB drive on your PC
unplug power from Vera
put USB drive back in same USB slot in Vera
plug Vera's power back in
reinstall dataMine




Title: Re: New dataMine graphing and logging plugin
Post by: Chris on August 19, 2012, 03:53:55 am
Sorry for the slow replies. I've been working abroad for the past couple of weeks without much access to Internet  :(

I tried the plugin on my other unit and everything worked fine, except the missing DataMine icon on both units. Both units are Vera3(.408)
I don't understand what you mean by this - what DataMine icon is missing? If you're talking about the plugin icon, then if that's not there how are you configuring the plugin?

Any new updates to test?
Not yet - sorry. I blame my employer... I'm at home for the next week so if the other boss doesn't have too much for me to do at home, I'll try and get a new version out that can work with the USB logging.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on August 19, 2012, 10:51:50 am
Sorry for the slow replies. I've been working abroad for the past couple of weeks
That sounds like fun... but my wife is against me being with other broads. Oh wait, maybe you meant overseas?  :P

Any new updates to test?
Not yet - sorry. I blame my employer... I'm at home for the next week so if the other boss doesn't have too much for me to do at home, I'll try and get a new version out that can work with the USB logging.
Well there's your problem... letting work get in the way!  ;)

Seriously though,
I stopped using the USB for Vera logging and have no had the problem since. So as much as it pains me bag on Linux  ;D the problem is Linux. If you can find a way for dataMine to use the already mounted USB drive, that would be perfect.

Also, a while back you mentioned adding some more features with the graphs?

Title: Re: New dataMine graphing and logging plugin
Post by: bfromdruten on August 19, 2012, 11:19:16 am
Aaron, I have had the same problems. My solution is to create two partitions on the usb stick. One partition for usb logging, one for datamine. I used 'EaseUS partition' to get the stick partitioned. Works fine since.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 02, 2012, 02:08:22 pm
Chris,
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

I removed the mount for Vera, weeks before so only dataMine is using the USB. - dataMine worked fine for a while.

Are there any updates coming, I need to wipe the install and retry but wanted to check with you first.

thx
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 02, 2012, 03:46:20 pm
Chris,
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

I removed the mount for Vera, weeks before so only dataMine is using the USB. - dataMine worked fine for a while.

Are there any updates coming, I need to wipe the install and retry but wanted to check with you first.

thx

Hi Chris,
Having the same problem as Aaron. A red ! next to my channels. Also I'm having trouble selecting different channels. The channels are reset and I cannot unselect channels.

BTW I have a Veralight and USB logging is off
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 02, 2012, 04:56:04 pm
I'm having a problem where the last few weeks it is not logging 4 of the 5 triggers. They have red ! next to them.

Hi Aaron,
What do you mean by "triggers" - do you just mean "variables being logged"?

The red ! means that there have been no updates to the variable in the last day (or since Lua restarted if it was soon). For analogue variables (temperatures, humidity, etc), this rarely happens. The yellow ! means that there hasn't been anything between 1 hour and 1 day - this is reasonably common (most of my variables show this).

dataMine uses a callback feature in Vera, so it just gets notified when the variables change.  The most likely reason for variables not being logged therefore is if for some reason the channels have been disabled. Can you take a look at the log - when dataMine starts it logs all the variables that it has asked Vera for a callback. If the variables are being logged, then dataMine really should receive the callbacks unless Vera/Lua is doing something strange (very unlikely).

I have added some code to support the USB logging, but I haven't had the chance to test it out yet (just a bit busy with other things at the moment unfortunately).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 02, 2012, 05:20:40 pm
Yes, I meant Channels - not triggers.

They are enabled (green) in the config. The Vera is rebooted each night so everything should be 'fresh' (memory, etc).

I get the Red ! on 4 channels - see attached jpg

All the .log files have dates of 7/30 on them... and the LuaUPnP.log is over 1GB!

The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 02, 2012, 05:29:29 pm
I'm having trouble selecting different channels. The channels are reset and I cannot unselect channels.

Hi @rarev,
What do you mean by "the channels are reset" - I would think this means they aren't selected, but the next part says you can't unselect channels.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 02, 2012, 05:53:05 pm
The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.
Stupid question, but I assume that all these channels are actually updating in Vera? I see one of the channels is a lock status, so I assume this is updating when you change the state?
Can you look at the log to see what channels dataMine says it's logging?
Title: Re: New dataMine graphing and logging plugin
Post by: mjcohnmd on September 09, 2012, 08:40:35 pm
I haven't read all the posts so I apologize if this question has already been addressed.

Great plugin.  I'm using it to monitor my six Wayne-Dalton WDTC-20 thermostats.   The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

Thanks,  --JC
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 09, 2012, 10:28:50 pm
The Channel .txt files all show last dates of 8/26. 8/29, 8/30... except the one that is still working, which has today's date.
Stupid question, but I assume that all these channels are actually updating in Vera? I see one of the channels is a lock status, so I assume this is updating when you change the state?
Can you look at the log to see what channels dataMine says it's logging?

Yes they are showing data in Vera. two channels are from google weather which is definitely getting the correct, and updated info.
I deleted all the logs and reset the channels... now one of the channels that was not graphing is, and all the others are not. Something is going hanky
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 11, 2012, 10:33:09 am
Hi Chris,
Sorry for not being clear. What I meant was that the channels seemed to disappear. So I selected 5 channels and the virtual DataMine device showed "5". When I checked a few hours later the counter was 0. A reinstall fixed somehow, so I'm afraid I can't reproduce it.

As for the exclamation marks,  I seem to have the same problem as Aaron.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 11, 2012, 11:50:43 am
Sorry for not being clear. What I meant was that the channels seemed to disappear. So I selected 5 channels and the virtual DataMine device showed "5". When I checked a few hours later the counter was 0. A reinstall fixed somehow, so I'm afraid I can't reproduce it.

I had this problem... I think it was that I had mounted the Vera logs to USB... thus it was 'sharing' a mount point with dataMine... about every week or so my channels would go bye-bye. After removing the Vera logs, this stopped and they have not disappeared since. One solution provided was to partition the USB drive and mount to different partitions... but Chris said he would make it so we don't need to us a mount point... I'm waiting for this :)

As for the exclamation marks,  I seem to have the same problem as Aaron.

Still happening for me.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 11, 2012, 12:50:12 pm
Thanks guys,
Unfortunately, I really need more information. It seems something is awry, but god knows what. Unfortunately we really need to look in the log file. There are two things that we need to look for -:

First is the startup - when dataMine starts, it logs all the variables that it's actually logging
Code: [Select]
50 09/11/12 17:27:13.328 luup_log:87: dataMine: Initialising dataMine System <0x2b635680>
50 09/11/12 17:27:13.373 luup_log:87: dataMine: Mounting dataMine storage (/dev/sda1) <0x2b635680>
50 09/11/12 17:27:15.192 luup_log:87: dataMine: Watching: D[5] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x2b635680>
50 09/11/12 17:27:15.193 luup_log:87: dataMine: Watching: D[11] S[urn:cd-jackson-com:serviceId:Irrigation1] V[WaterButtVolume] <0x2b635680>
50 09/11/12 17:27:15.214 luup_log:87: dataMine: Startup complete <0x2b635680>

Next is if there are any problems that dataMine encounters when it's logging data.  Assuming the startup shows that all the variables you think are being logged, really are beinmg logged, then Vera will call dataMine when they change (this almost has to happen - if it doesn't, it's a Vera problem). When dataMine receives this callback, it tries to correlate the data with the variables being logged, and find the file to write the data into. This code can log a couple of errors.

Lastly, is if there's some sort of other error, the Vera will log it. These are more difficult to find, so I'll ignore them for now....

To get the log file entries that dataMine logs, you need to run the following in WinSCP (or some similar browser) -:
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
The other question is, is there anything "special" about the variables that aren't working? Are they character based (eg a thermostat mode, or wind direction), or do they have funny names with special characters (ie not alphanumeric, but say colons or /[]\ type characters).

I'll see if I can find a simpler way to get the logged data, but in the meantime any information that you can provide would be great... 

Cheers
Chris

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 11, 2012, 12:54:19 pm
The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

I'll look at adding two options - one to filter "out of range" data (I'll provide the ability to set these ranges). The other is the ability to export the data as CSV - this is something I've been meaning to add anyway... I'll try and add this in the next few weeks (I'm currently a bit busy with work and getting heating into the new extension before winter sets in).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 11, 2012, 01:08:27 pm
The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

I'll look at adding two options - one to filter "out of range" data (I'll provide the ability to set these ranges). The other is the ability to export the data as CSV - this is something I've been meaning to add anyway... I'll try and add this in the next few weeks (I'm currently a bit busy with work and getting heating into the new extension before winter sets in).

Cheers
Chris

Just go to Home Depot and get a portable heater... you have work to do ;-)
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 12, 2012, 04:35:27 am
Sorry for not being clear. What I meant was that the channels seemed to disappear. So I selected 5 channels and the virtual DataMine device showed "5". When I checked a few hours later the counter was 0. A reinstall fixed somehow, so I'm afraid I can't reproduce it.

I had this problem... I think it was that I had mounted the Vera logs to USB... thus it was 'sharing' a mount point with dataMine... about every week or so my channels would go bye-bye. After removing the Vera logs, this stopped and they have not disappeared since. One solution provided was to partition the USB drive and mount to different partitions... but Chris said he would make it so we don't need to us a mount point... I'm waiting for this :)

The problem with the disappearing channels was definitely a problem with the USB mount. So that was my bad. Setting the mount point to /dev/sda in stead of /dev/sda1 on vera3lite. This one is fixed.
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 12, 2012, 06:06:14 am
Somehow destroyed it this time. There's no way to select a channel...
I uninstalled DataMine. Formatted the USB-drive (1Gb) to FAT32. Installed DataMine again. Plugged in the USB-drive and entered the DataMine MountPoint to /dev/sda1. MountType is EXT4 and I cannot change this. (changing this to VFAT or EXT3, will make no difference, since hitting SAVE in UI5 will reset it to EXT4)

When I open DataMine to select channels nothing happens. The number of selected/logged channels will remain 0. ARRRG!
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 12, 2012, 07:38:21 am
It seems that the browser is causing some trouble.
@Aaron By any chance you are using Windows 8 Internet Explorer?
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on September 12, 2012, 07:41:19 am
Internet Explorer has been known to cause all sorts of issues with Vera's Interface. It is recommended to use another browser like firefox, chrome, safari, etc.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 12, 2012, 10:07:10 am
Using Chrome solved most issues. But still only exclamation marks on the graphing page
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on September 12, 2012, 10:43:02 am
Check the graph data. I have a few exclamation marks but the graph data is current.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 12, 2012, 10:53:55 am
I tried resetting by removing datamine and formatting the USB drive and now I can't even select channels anymore. After setting SetMountPoint, mountLocation and mountType to "/dev/sda1" (2x) and vfat resp. and hitting save in ui5, mountLocation and mountType are empty again. So at the moment DataMine can't even reach my USB drive.  :-[
Title: Re: New dataMine graphing and logging plugin
Post by: ipstas on September 12, 2012, 12:58:29 pm
rarev,

I would suggest to log in through ssh and check out there.
When you insert the stick and run dmesg, it should show you someting like

root@MiOS_30002917:/tmp# dmesg  | grep sda
sd 0:0:0:0: [sda] 1949696 512-byte logical blocks: (998 MB/952 MiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
EXT3-fs (sda1): using internal journal
EXT3-fs (sda1): mounted filesystem with ordered data mode

So mine has /dev/sda1 and /dev/sda2
At the moment I am using /dev/sda1 for the usb logging and /dev/sda2 for datamine

to be sure /dev/sda2 is mounted to /dataMine you would need to update
vi /etc/init.d/custom-user-startup
mount /dev/sda2 /dataMine &

make sure, the /dev/sda2 have filesystem. You can do it:
mkfs.vfat /dev/sda2
then mount
mount /dev/sda2 /dataMine

I don't recommend the other method of mounting which was mentioned before, /dev/sda is standard in any unix and just easier to do.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 12, 2012, 06:00:04 pm
I don't recommend the other method of mounting which was mentioned before, /dev/sda is standard in any unix and just easier to do.

@ipstas - what do you mean by this? You seem to know a little about linux (more than me anyway!) so if you see anything that I'm doing incorrectly, or that could be done better, please let me know.

At the moment, dataMine automatically runs the mount /dev/sda2 /dataMine command (the device being based on config of course).

Thanks
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: ipstas on September 12, 2012, 06:14:40 pm
Chris,

I am from linux side ;)
In http://code.mios.com/trac/mios_datamine it mentioned to mount like that:
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine, it maybe is better to fix it.

I don't know how datamine does mounting, on my unit /dev/sda2 did not have filesystem, probably because I have usb logging and Vera3 formatted the stick itself. So I have created filesystem first and then mounted manually. And just in case created /etc/init.d/ script to mount it at the restart.

Title: Re: New dataMine graphing and logging plugin
Post by: mjcohnmd on September 12, 2012, 11:40:57 pm
The thermostats seem to send erroneous outlying data fairly often (usually over 180 deg F, once or so per day per thermostat).  A setting to ignore outlying data (?>50% from mean) for the graphs would be nice. Or a tool for exporting the data to manually edit (or maybe teach me how to get directly at the raw data)

I'll look at adding two options - one to filter "out of range" data (I'll provide the ability to set these ranges). The other is the ability to export the data as CSV - this is something I've been meaning to add anyway... I'll try and add this in the next few weeks (I'm currently a bit busy with work and getting heating into the new extension before winter sets in).

Cheers
Chris

Just go to Home Depot and get a portable heater... you have work to do ;-)

Thank you...I appreciate your efforts.

--JC

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 13, 2012, 04:33:52 pm
Chris,

I am from linux side ;)
In http://code.mios.com/trac/mios_datamine it mentioned to mount like that:
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine, it maybe is better to fix it.

I don't know how datamine does mounting, on my unit /dev/sda2 did not have filesystem, probably because I have usb logging and Vera3 formatted the stick itself. So I have created filesystem first and then mounted manually. And just in case created /etc/init.d/ script to mount it at the restart.
Thanks - the "original" mount data I found on an OpenWRT forum for mounting USB sticks - it worked, and I didn't know better. The current method simply uses a "mount <mp> <dir>" where mp and dir are set in the configuration. This eliminated the need to mess around with anything in init.d, or custom files or anything. This seems to work ok (I think). These "original" instructions are for the UI4 version - the UI5 version has the functionality to mount the drive in the Lua file.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 14, 2012, 04:08:13 am
I found the problem was in formatting the usb. Still don't know how exactly it should be formatted, but this is what I did and it worked:

It just works
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 14, 2012, 01:49:21 pm
I've had no problems using Windows to format a USB - I've done it a number of times on different Vera boxes and never had an issue. It's interesting though that by doing it this way you get the mount point information "for free".

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 18, 2012, 06:32:16 am
not to take anything away from the documentaion provided which is really quite excellent, there seems to just be a gap for those of us on Veralites and also the use of a hub.

Which leads me nicely onto sharing that I am using a Veralite, but with the added complexity of a hub in the middle.

As directed I formatted a new USB memory key and it looks to have create a few things in my /dev/ directory, I can see entries for a sda, sda1 and 1-1.3. Then after a reboot I also now have ttyUSB, ptmx and tyyS0 - (I'm using winscp BTW to look into the directory)

If I ry the guidance above from ipstas

root@MiOS_3XXXXXXXX:~# dmesg | grep sda
sd 0:0:0:0: [sda] 7819264 512-byte logical blocks: (4.00 GB/3.72 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 03 41 00 00
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI removable disk
root@MiOS_3XXXXXXXX:~#

So I have put /dev/sda1 into both SetMountPoint and MountLoation of the device, and logging in as root and checking Mount - i seem to have the following

/dev/sda1 on /dataMine type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp4                                                                             37,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

It looks like I can add channels and it records them - but nothing seems to come up on the graphs (6 selected, 6 recording) no data being seen = every graph shows the time across the bottom and runs a line from a -9999989.01 value ?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 18, 2012, 12:55:03 pm
To find the drive, I think the following works -:
Using WinSCP, click on the little Console / "Open Terminal" button and type "fdisk" with the USB unplugged. Then plug in the USB stick, and type "fdisk" again. You should (I hope!) see another drive installed.

Quote
It looks like I can add channels and it records them - but nothing seems to come up on the graphs (6 selected, 6 recording) no data being seen = every graph shows the time across the bottom and runs a line from a -9999989.01 value ?
From WinSCP, take a look in the /dataMine directory. This is where the files are stored and it would be worth opening one of them to see if there's any data. Note that dataMine only logs changes in value, so depending on what type of data you're logging, this may not happen too often. It doesn't log data periodically... If you're logging a light switch for example, it will only log something when you actually change the switch state.  Until there's more than 1 entry logged, the graphs don't really work.

Let me know if that helps...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 18, 2012, 02:43:48 pm
Thanks again Chris,

I'm sure you're sick of all the praise you've been getting for this plugin by now ;) but boy does this plugin look bloody good !!!

As a newbie, a few hours in, i have some questions for you if that is OK.

1) Out of 7 variables it says it should be recording, looking in the /dataMine/ directory, the names of only 3 are listed. The ones that are missing look to be those that are reporting the 9999989.01 values on the graph, yet 2 of them are showing a green tick to suggest they're being updated?

UPDATE 1 - I think the might have found the cause for 2 of them, as a couple of device name have an '/' in them, e.g. RFX Power ELEC2/56898. Plus I could see in the logs that datamine was trying to write to it but it was unable to open the file (not surprising as it's not there) plus I also couldn't create a version of it manually with the exact same name.

UPDATE 2 - This links to question 7 I think, as I have renamed one of the devices that had '/' in the name, which after a refresh is now visible under its new name on the Config tab and surprisingly still has the required variable highlights in green, However the graph entry has not been updated with the new name and looking in the logs it's still saying that datamine is unable to write to a file with the old nam (RFX Power ELEC2/56898....txt) etc .

2) Under the Advanced tab - LastUpdate is set to zero, should that  change with each write for a recording?

3) Under the Config tab - when selecting a variable there is no clock icon visible/available as far as I can see, only an empty square?

4) there are a lot of .backup files (17 to be exact ) in the /dataMine/ directory, is that to be expected?

UPDATE - Ok, I see now, every change in selection of variables that i want to be recorded or not etc creates a back up of the dataMinejson file

5) There is no device icon appearing for the dataMine device in the UI, is that to be expected too?

UPDATE - Looking in the JSON file that was installed from the app store, I can see the following icon listed, "flashicon":"icons\/Data_Mine.swf"., yet nothing appears

6) Is there a way to delete a graph for a variable I selected by mistake?

UPDATE - It seems the only way to do this is to restore a backup file that did not have it in, which sadly removes everything else added subsequently, are there still plans to have a field in the UI to remove/hide old graphs and variable references? ..

7) What happens if I need to rename a device, will it lose it's connection with its associated  log file in the /dataMine/ directory ?

UPDATE - As mentioned in question 1 above, it seems even if I try to change the name in the dataminjson file it is not reflected. All it looks like I can do is restore an old back up?.

8.) I have two temperature logging files, each with the same start date/time, yet depending on the order I select them to be 'graphed' the info is presented in two very different ways. Why?

I will continue to try various things and add updates , but if anyone can help me, your support would be appreciated.
Title: Re: New dataMine graphing and logging plugin
Post by: rarev on September 19, 2012, 05:24:23 am

6) Is there a way to delete a graph for a variable I selected by mistake?


Couldn't find this either. Guess it's not there (yet). Would like to see this option. But deleting a graph can be done by unselecting the channel. I'd like to see the possibility to remove the channel to unclutter my list of channels.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 10:25:06 am
Quote
1) Out of 7 variables it says it should be recording, looking in the /dataMine/ directory, the names of only 3 are listed. The ones that are missing look to be those that are reporting the 9999989.01 values on the graph, yet 2 of them are showing a green tick to suggest they're being updated?
UPDATE 1 - I think the might have found the cause for 2 of them, as a couple of device name have an '/' in them, e.g. RFX Power ELEC2/56898. Plus I could see in the logs that datamine was trying to write to it but it was unable to open the file (not surprising as it's not there) plus I also couldn't create a version of it manually with the exact same name.
UPDATE 2 - This links to question 7 I think, as I have renamed one of the devices that had '/' in the name, which after a refresh is now visible under its new name on the Config tab and surprisingly still has the required variable highlights in green, However the graph entry has not been updated with the new name and looking in the logs it's still saying that datamine is unable to write to a file with the old nam (RFX Power ELEC2/56898....txt) etc .
Ok - the '/' in the filename is an issue that I have fixed and I'll get this update released in the next day or so. As per the answer to question 7 below, the file names and display names are different. Also, datamine won't change any of its configuration when you change the device configuration (ie if you edit the device name after starting dataMine logging). DataMine doesn't use the names of devices to operate internally - it uses the device numbers which is why when you change the name it still works, but the filename is still wrong.

Quote
2) Under the Advanced tab - LastUpdate is set to zero, should that  change with each write for a recording?
No idea - this is an internal Vera variable I think - it's not a dataMine specific/controlled variable.

Quote
3) Under the Config tab - when selecting a variable there is no clock icon visible/available as far as I can see, only an empty square?
Ah - thanks. Yes, this is probably a hangover from the UI4 version where the dataMine GUI was integrated in UI4. Now this reference is broken. My ultimate plan is to remove this from the plugin and have the configuration in the dataMine GUI, but at the moment I'm fighting with bugs in the java lib I'm using...

Quote
4) there are a lot of .backup files (17 to be exact ) in the /dataMine/ directory, is that to be expected?
UPDATE - Ok, I see now, every change in selection of variables that i want to be recorded or not etc creates a back up of the dataMinejson file
Yes - you can safely delete the old ones. I will add a clean funtion to delete old backups,

Quote
5) There is no device icon appearing for the dataMine device in the UI, is that to be expected too?
UPDATE - Looking in the JSON file that was installed from the app store, I can see the following icon listed, "flashicon":"icons\/Data_Mine.swf"., yet nothing appears
This should be taken care of by the Vera plugin manager. I'll double check this...

Quote
6) Is there a way to delete a graph for a variable I selected by mistake?
No - not at the moment, but this is planned. Note that since you seem to be playing in the dataMine.json file, you can delete stuff in there. A good pointer - if you do edit the json file, I would recommend a json formatter so that you can read it better. I personally use jsonlint.com.

Quote
7) What happens if I need to rename a device, will it lose it's connection with its associated  log file in the /dataMine/ directory ?
UPDATE - As mentioned in question 1 above, it seems even if I try to change the name in the dataminjson file it is not reflected..
The config file (dataMine.json) contains separate references for 1) log file names, 2) the name displayed in the GUI, and 3) the actual link to the device. You can change the display name without impacting on the operation of dataMine.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 19, 2012, 10:37:41 am
Thanks so much Chris, if you get chance I had added a question 8. which is a wired one?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 10:53:55 am
Thanks so much Chris, if you get chance I had added a question 8. which is a wired one?

8.) I have two temperature logging files, each with the same start date/time, yet depending on the order I select them to be 'graphed' the info is presented in two very different ways. Why?
Can you provide an example of what you mean? Maybe upload a couple of screenshots? The data is passed into a graphing library, so it may be something to do with the way that interprets the data, but a couple of screenshots would help to explain the problem.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 19, 2012, 11:08:05 am
Can you provide an example of what you mean? Maybe upload a couple of screenshots? The data is passed into a graphing library, so it may be something to do with the way that interprets the data, but a couple of screenshots would help to explain the problem.

Sure here you go.

Both have the same date/time entry for the first reading, yet the graph renders them differently and places them far apart.

Additional questions if you have time?

a) - If I want to transfer all my current cost energy usage data, that's stored in an RRD, do I do that in one hit, or did i read i needs to create different files ?
b) - what does the [R2228] represent at the end.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 11:46:16 am
Hmmm - interesting. I'll need to get back to you on what I think is happening with the graphs! It's certainly not immediately obvious....

Quote
a) - If I want to transfer all my current cost energy usage data, that's stored in an RRD, do I do that in one hit, or did i read i needs to create different files ?
Yes, you need different files. The files are broken into weekly files just to avoid a single massive file and having to scan through to access the data.

Quote
b) - what does the [R2228] represent at the end.
The R means Raw, and the 2228 is the week number. In theory, R can be something else (the code isn't hooked in, but you can generate hourly and daily files as well - the idea being to speed up the generation of long timespan graphs). If you're familiar with unix time, then the 2228 comes from the current unix time divided by 604800 (the number of seconds in a week).
So, if you're familiar with messing with RRD, then I suspect you won't find it too difficult to convert the data if you wanted...  Note that RRD samples/logs data periodically where dataMine just logs when there are changes, so you will see some differences in the graphs.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 12:04:38 pm
So, I put your data into my system, made a couple of bogus entries in the config file, and it seems to graph just fine... Are you familiar with FireBug for logging data to/from the browser? If so, it would be really interesting to see what the request and response is...

I also don't see a difference depending on which way around I select the variables - it's all a bit strange. I'm running a different version here, but I don't think that should impact on this issue...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 01:14:28 pm
An updated version of the plugin should be hitting your Vera in the next day or two. This is only a change to the plugin, not the GUI.  It has the following changes -:

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 19, 2012, 01:18:43 pm
Thanks Chris

Were they 'graphing' OK before you added the bogus entries?

Are you familiar with FireBug for logging data to/from the browser? If so, it would be really interesting to see what the request and response is...

Sorry no I'm not, is it straight forward to use?

What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 01:29:38 pm
Were they 'graphing' OK before you added the bogus entries?
I needed the bogus config entries for it to show up in the channels list, otherwise I can't graph them...

Sorry no I'm not, is it straight forward to use?
If you are using FireFox, then it's reasonably easy for what I need if you're vaguelly tech savvy. Once it's installed, it adds a button to FireFox and there's a console that can show all the traffic between the browser and the server. It would be possible to capture the request and response with this. IE may have a similar feature as you can hit F12 to enter developper tools area, but I don't tend to use it...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 19, 2012, 01:52:36 pm
What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?

Any thoughts on the above?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 19, 2012, 02:12:12 pm
What's strange is that both .csvs 'graph' fine on their own, it is only when they are combined ?

Any thoughts on the above?
Nope - unfortunately not (although I can agree with you - it's strange! ;))
It would be good to get a cope of what the server is producing. The other thing I did think of earlier is if there's something in your config file that's a bit wierd, but I really can't think of anything that could cause this.

One more thing you could look at. Graph the data both ways, then do a grep on the log file - from WinSCP, go to the terminal and type "grep dataMine /tmp/log/cmh/LuaUPnP.log". This does log data about requests coming in - I don't think it will tell me much, but you never know (and at the moment I just don't have a clue).

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 19, 2012, 02:32:11 pm
Cheers Chris.

Another strange observation with the graphs for you..

If I select a variable that has been logging the longest (Ben's Temp) first and then add another one that has not been logging as long (Jane's temp) it plots the new/additional line graph starting further down. (seems logical)

Ok now clearing the graph and I did this..

If I select a variable that was not logging the longest (Jane's Temp) and then add another one that's been logging longer, it a) does not extend the graph back to factor in the longer period, and b)  it actually plots the older one further down?

Weird?
Title: datamine graphic
Post by: save on September 20, 2012, 04:43:15 am
Hello,

I have installed datamine aplication.  I attache the graphic apperars using a AEON labs power meter. What kind of units are -999999 in the Y axis?
The meter runs Ok and it should appear units in Watt as 200 or similar...



Regards
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 20, 2012, 05:37:43 pm
Hi @save

One thing I'm learning is that everything to do with the data used by this plugin, pretty much revolves around the data captured and stored in the /dataMine/ directory on your Vera.

Can you use WinSCP or alternative to confirm that there are a) corresponding named text files for each variable you're logging in there and b) if so, do they have data stored within them

I had the very same issue and it was because the files were not here
Title: Re: New dataMine graphing and logging plugin
Post by: save on September 21, 2012, 11:12:09 am
Thanks

How must I use WinSCP, must I connect Vera USB port to the computer?

Regards
Title: Re: datamine graphic
Post by: Chris on September 21, 2012, 01:46:41 pm
I have installed datamine aplication.  I attache the graphic apperars using a AEON labs power meter. What kind of units are -999999 in the Y axis?
The meter runs Ok and it should appear units in Watt as 200 or similar...
dataMine appears to be only getting data for one channel - the one with the green tick. Note that dataMine only logs changes to channels - I would expect a energy monitor to change quite often, but mine can be static for a while if nothing is changing (and it also depends on how the HEM is configured to send data).

Now that you've had dataMine running for a day or so, has anything changed? Does it still do the same thing? What happens if you only graph the channel(s) that has a green tick?

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 21, 2012, 05:15:15 pm
Hi Chris,

When you have time, please could you let me know the naming convention I need to use for the .csv files that will store the 8 months worth of Current Cost EnviR energy data I'd like to extract from an RRD table and use with you app.

* How do you think DataMine will cope with such a large amount of data (as you know energy monitors can produce a lot)?
* Should I try to strip out all the duplicate values as i guess theoretically they're worthless?
* Are there any plans to manage/graph data thats well over a month old, so people can see the history more easily?
* Have you thought about adding an option to compare previous captured periods?

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 21, 2012, 05:33:45 pm
* How do you think DataMine will cope with such a large amount of data (as you know energy monitors can produce a lot)?
It should be fine. I have been logging my energy monitor for the past 11 months and it works just fine. The energy monitor isn't my highest data producer - some of my other sensors (the light sensor is the highest producer) produce up to 120k files per week.
dataMine does sub-sample data to produce graphs with about 700 entries (from memory) and keep the amount of data transfer down, and the speed up. Longer term data files (sort of history files) are coded, but aren't generated automatically (it's something I need to finish).

* Should I try to strip out all the duplicate values as i guess theoretically they're worthless?
Up to you, but I probably would reduce the data to changes only (which is what Vera/dataMine will produce).

* Are there any plans to manage/graph data thats well over a month old, so people can see the history more easily?
Yes - it's actually coded to produce hourly/daily history files which speed up production of longer term graphs. There are some issues with generating this as Vera has a habbit of resetting itself (that said, my Vera 3 seems a LOT better than my Vera 2). This just makes it a bit more difficult - not impossible though. I'm currently trying to get some issues sorted out with the GUI, then I will work on this. The raw files are the important ones though - the other files can be generated any-time.

* Have you thought about adding an option to compare previous captured periods?
It's not something I've thought of, but should be possible. I'm trying to get the GUI into better shape so that I can add more features - the current js library I'm using has issues and I'm currently looking at another library as the existing supplier isn't fixing the issues quite as fast as I'd like.  :(

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 21, 2012, 05:40:15 pm
Thanks Chris,

Last one - Do I just paste the old values in the current live file?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 21, 2012, 05:52:05 pm
Last one - Do I just paste the old values in the current live file?
The files have to be generated with a separate file each "week". That's the [R2228] type bit on the end of the file - it's just the current unit time divided by the number of seconds in a week (604800).
This just makes it quicker for the plugin to search through the data - a single file would take "forever".

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 21, 2012, 06:11:04 pm
Sorry Chris, I'm lost now (my lack of Linux experience),

If I'm looking to create say 30 weekly files, I should aim to pick the first unix date time entry in each weekly file and then divide that value by 604800, and then just put an 'R' at the beginning? So 1337265267 becomes "Name Of Variable [R2211]"
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on September 22, 2012, 06:00:38 am
Sorry Chris, I'm lost now (my lack of Linux experience)
It's not really a linux issue - (so called) unix time is used reasonably universally...

If I'm looking to create say 30 weekly files, I should aim to pick the first unix date time entry in each weekly file and then divide that value by 604800, and then just put an 'R' at the beginning? So 1337265267 becomes "Name Of Variable [R2211]"

Basically, yes.
Somewhere I actually have a C program to do this as I used to store data in a MySQL database in an older system I had. Anyway, assuming your data is in chronological order, then you want to do the following -:
Code: [Select]
for each time/variable pair do {
    if value != lastvalue then {
        lastvalue = value
        week = (int)(time / 608400)
        filename = variable name [Rweek].txt
        open file(filename)
        write time,value<CR>
        close file
    }
}

I hope my psuedo-code makes sense  ;) So long as the data is in the right order, you should end up with a bunch of files with your data that should be compatible with dataMine.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on October 10, 2012, 02:36:04 am
Thanks

How must I use WinSCP, must I connect Vera USB port to the computer?

Regards

@save - Sorry, It doesn't look like your question was answered.

With WinSCP, both the PC running it and your Vera need to be on the network, next you need to create a connection to your Vera using it's IP and the root name and password credentials which you can find here... http://wiki.micasaverde.com/index.php/Logon_Vera_SSH#Can.27t_find_the_root_password.3F

Once inside navigate to the dataMine folder and you can see the logs in there.
Title: Re: New dataMine graphing and logging plugin
Post by: slajgaj on October 16, 2012, 05:31:48 am
Hi Chris!

I disabled any device logging, but the variable will still be listed in the graph page dropdown.
How to clear/delete from the graph page list? I don't like to see, because wrong data variable.

Best R.
SL
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on October 16, 2012, 05:40:28 am
Chris has not implemented deleting of values that are not being recorded in the current datamine version. The reason for this was to archive values for historic reasons. The only way to whip these values is to modify the file that these values were recorded in. Most likely to make sure that your other data is not lost is to just ignore the older values.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 16, 2012, 12:35:29 pm
I will change the way this works with the next GUI version. The original concept was to leave the old channels listed for historical reasons, but there's probably a better way to handle this... The new GUI allows me to add more functionality, so I'll look at handling historical data differently if it's needed...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on October 16, 2012, 01:29:00 pm
I will change the way this works with the next GUI version. The original concept was to leave the old channels listed for historical reasons, but there's probably a better way to handle this... The new GUI allows me to add more functionality, so I'll look at handling historical data differently if it's needed...

Chris

Chris,
I suggest creating a pane in the left side called 'disabled channels'. Then allow users to disable a channel which is then automatically moved to that pane (which can be collapsed to save space). This way, it is still available when/if needed but does not get in the way when it is not.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 16, 2012, 01:37:32 pm
Chris,
I suggest creating a pane in the left side called 'disabled channels'. Then allow users to disable a channel which is then automatically moved to that pane (which can be collapsed to save space). This way, it is still available when/if needed but does not get in the way when it is not.
Hi Aaron,
Yes - I had a similar thought. There are two potential issues - one is channels that have been disabled. They are relatively easy since they still exist. The other one is channels for devices that have actually been removed (orphans?). I'm thinking of putting something on the config pane, but I need to have a think about it once I've got the new version out there...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on October 16, 2012, 02:44:12 pm
Chris,
I suggest creating a pane in the left side called 'disabled channels'. Then allow users to disable a channel which is then automatically moved to that pane (which can be collapsed to save space). This way, it is still available when/if needed but does not get in the way when it is not.
Hi Aaron,
Yes - I had a similar thought. There are two potential issues - one is channels that have been disabled. They are relatively easy since they still exist. The other one is channels for devices that have actually been removed (orphans?). I'm thinking of putting something on the config pane, but I need to have a think about it once I've got the new version out there...

Cheers
Chris

use indicator icons on the list (next to the channel name) to show it no longer exists... like a small ghost or something ;)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 16, 2012, 04:34:18 pm
use indicator icons on the list (next to the channel name) to show it no longer exists... like a small ghost or something ;)
Good idea... Not sure I've got the best ghost icon, but it's a useful feature as I found I've got a couple of variables that are from an old (deleted) Google weather plugin  :)

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on October 19, 2012, 04:41:41 pm
Chris

Coming back to Vera after a summer off (it's main task is to run my heating) I am wondering if you ever ported the new version of Datamine to UI4?

I am on UI4 here and reluctant to upgrade as it's reliable and the heating control scenes all work well.

I've not got datamine running at the moment as when I had to restore vera from a backup in the spring it didn't come back.

Now the heating is in use I want to get it running again

Many thanks

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 20, 2012, 11:36:39 am
Coming back to Vera after a summer off (it's main task is to run my heating) I am wondering if you ever ported the new version of Datamine to UI4?
Hi nick,
No, I've not ported it back to UI4 (yet). I'm currently doing some reasonably major mods and when that's in a reasonably done state (in the next few days), I'll take a look at UI4 again. It shouldn't be a major mod - it's mainly just the init stuff that's different but most of my "Vera time" has been spent fighting with the java library...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on October 20, 2012, 12:55:27 pm
I am on UI4 here and reluctant to upgrade as it's reliable and the heating control scenes all work well.

Nick,
I suggest taking a look at updating to UI5. Unless there is something in UI5 that you know breaks your existing functionality I don't see any reason not to move to it, and several reasons to do so.

One is that UI6 will likely be out Q1 2013 and is expected to have major updates to core areas like Scenes. I don't suspect any plugin owner will want to support code for UI4, UI5 and UI6. Plus, who knows if you will be able to update from UI4 directly to UI6, but I doubt MCV will support a 2 version direct update. Even major OS companies usually don't do that.

Anyway, my $0.02
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on October 21, 2012, 05:03:28 am
I am on UI4 here and reluctant to upgrade as it's reliable and the heating control scenes all work well.

Nick,
I suggest taking a look at updating to UI5. Unless there is something in UI5 that you know breaks your existing functionality I don't see any reason not to move to it, and several reasons to do so.

One is that UI6 will likely be out Q1 2013 and is expected to have major updates to core areas like Scenes. I don't suspect any plugin owner will want to support code for UI4, UI5 and UI6. Plus, who knows if you will be able to update from UI4 directly to UI6, but I doubt MCV will support a 2 version direct update. Even major OS companies usually don't do that.

Anyway, my $0.02

Where did you receive this information?

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on October 21, 2012, 01:44:32 pm
Thanks for the thoughts folks. My intent has been to stay with UI4 because it works, my house stays the right temperature and energy consumption is reduced at the same time. To me Vera is a tool and not a hobby and hence I don't want to risk breaking what works for me.

I have quite a few scenes with Lua code in them and no idea what will port to UI5 (or 6) without breaking.

The same argument goes for 6 as 5, if it's not broken don't fix it.

Can anyone tell me what I will gain by stepping up to UI5 that I don't have at the moment apart from the updated version of Datamine??

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on October 21, 2012, 02:00:22 pm
Thanks for the thoughts folks. My intent has been to stay with UI4 because it works, my house stays the right temperature and energy consumption is reduced at the same time. To me Vera is a tool and not a hobby and hence I don't want to risk breaking what works for me.

I have quite a few scenes with Lua code in them and no idea what will port to UI5 (or 6) without breaking.

The same argument goes for 6 as 5, if it's not broken don't fix it.

Can anyone tell me what I will gain by stepping up to UI5 that I don't have at the moment apart from the updated version of Datamine??

Nick

Nick, there are several threads discussing UI4 / 5 so I suggest searching on them and doing some exploring since we are getting too off topic going down this road.

My $0.02 with supporting (or not) UI4... As with most coders here, this is a hobby given limited time. I (and I think most users) prefer dataMine be enhanced to provide more features than back ported to a UI that is EOL and cannot be ran current or future Vera hardware. Adding features to dataMine is better for the community while it makes Vera a better 'tool'.

anyway... Chris has some cool stuff coming in dataMine - I'm looking forward to it.
Title: Re: New dataMine graphing and logging plugin
Post by: nickrwym on October 21, 2012, 02:03:28 pm
Yes, agreed, hence my enquiry to Chris was "whether he intended to support IU4 or not", if it is to be supported I will stay there, if it is not on the list of things to do that my be the trigger for me to update, I didn't intend this to be a "shall I update" thread hijack, it just happened for which I apologise

Nick
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 21, 2012, 06:37:08 pm
Yes, agreed, hence my enquiry to Chris was "whether he intended to support IU4 or not", if it is to be supported I will stay there, if it is not on the list of things to do that my be the trigger for me to update, I didn't intend this to be a "shall I update" thread hijack, it just happened for which I apologise

"Supported" is an interesting word..... I intend to produce a UI4 version with the usual UI4 caveats (installation not as easy as UI5 etc). I need to remind myself about the differences between UI4 and UI5 - from memory, it's mostly in the init code - if I can easily make both versions maintainable, then I'll "support" UI4, otherwise it may be a one off version...

However, the current intention is to make a few mods now (currently ongoing) and then most of the updates are hopefully in the GUI. This will be compatible with both UI4 and UI5 since it's effectively outside of the Vera UI now.

We'll see how it goes...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: tyfoon on October 25, 2012, 01:06:43 pm
I'm trying to use datamine for energy logging of the mains. Unfortunately my energy logger (NorthQ) only captures cumulative KWHReading data. If I make a graph of this I only get a line trending upwards of course.

Is there any way to configure DataMine to on 'understand'  this? So  it subtracts the data from the previous period (resulting in x kwh for per period y) so I can see the power usage over time?
Title: Re: New dataMine graphing and logging plugin
Post by: JeroenZvZ on October 25, 2012, 04:08:11 pm
I'm trying to use datamine for energy logging of the mains. Unfortunately my energy logger (NorthQ) only captures cumulative KWHReading data. If I make a graph of this I only get a line trending upwards of course.

Is there any way to configure DataMine to on 'understand'  this? So  it subtracts the data from the previous period (resulting in x kwh for per period y) so I can see the power usage over time?
*me too*
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 26, 2012, 05:03:47 am
I'm trying to use datamine for energy logging of the mains. Unfortunately my energy logger (NorthQ) only captures cumulative KWHReading data. If I make a graph of this I only get a line trending upwards of course.

Is there any way to configure DataMine to on 'understand'  this? So  it subtracts the data from the previous period (resulting in x kwh for per period y) so I can see the power usage over time?
Currently dataMine doesn't do this, however I do intend to add something that may do what you're after. The next area I'm playing with is energy monitoring, and in order to do this I will be looking at statistics, and usage over 1 hour, 1 day etc periods. This may do what you want, but I suspect that you're after something with a finer resolution (?).

It would not be hard to write a plugin to do this. I don't know anything about the NorthQ, but you could monitor the KWh variable, and when it changes work out the average power since the last reading. This would however result in the power being calculated retospectively - ie it would show the power now is (say) 500W, where actually the power between the last two data updates was 500W. This probably isn't a problem - just something to be aware of...

While this could be added to dataMine quite easily, I think it is probably best handled as a separate plugin. I could probably write a plugin quite quickly if no-one else wanted to - it sounds like there's probably a reasonable amount of interest in this?

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: tyfoon on October 26, 2012, 04:04:22 pm

Currently dataMine doesn't do this, however I do intend to add something that may do what you're after. The next area I'm playing with is energy monitoring, and in order to do this I will be looking at statistics, and usage over 1 hour, 1 day etc periods. This may do what you want, but I suspect that you're after something with a finer resolution (?).

It would not be hard to write a plugin to do this. I don't know anything about the NorthQ, but you could monitor the KWh variable, and when it changes work out the average power since the last reading. This would however result in the power being calculated retospectively - ie it would show the power now is (say) 500W, where actually the power between the last two data updates was 500W. This probably isn't a problem - just something to be aware of...

While this could be added to dataMine quite easily, I think it is probably best handled as a separate plugin. I could probably write a plugin quite quickly if no-one else wanted to - it sounds like there's probably a reasonable amount of interest in this?

Resolution of a hour would be fine I guess, 15 min better (NorthQ passes info every 15min). For me the objective is to make a start of influencing the power usage during the day/week. I will use separate z wave devices to monitor specific device use.

Would be great if you could make something!!
Title: Re: New dataMine graphing and logging plugin
Post by: JeroenZvZ on October 28, 2012, 07:14:52 am
Chris,

It would be great if you could help us out on this one.

A separate plugin would be great  for it would give me an opportunity to learn how a (simple?) plugin is made and experiment a little. I read a couple of topics about plugin development but don't know how to start on programming a plugin. I would really appreciate it if you could come up with one for the NorthQ.

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 28, 2012, 05:17:27 pm
I'll take a look at it tomorrow. I don't think it will take very long (an hour or two). I'll post a separate thread in the plugin folder when I've got something to play with (since it's not strictly a dataMine issue).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: JeroenZvZ on October 29, 2012, 05:22:47 pm

Thanks Chris,

Looking forward to see what you come up with.

I've some book knowledge on programming and hope to learn from the source files how to setup a plugin. I've read up a bit on Lua but need a kickstarter. I think this would help me a lot.

Jeroen
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on October 29, 2012, 05:30:57 pm
Hi Joroen,
I've written the plugin - the easiest way to distribute, and for people to install the plugin was through the app store. So, it's currently awaiting MCV to approve it...

See http://forum.micasaverde.com/index.php/topic,12350.0.html and let me know how it goes...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Francescobacchini on November 09, 2012, 07:37:51 am
Hi! I'm very happy of having finally your plugin working. Great Job, but... I can't access to it from outside (https://fwd2.mios.com/dm/index.html) Is it a problem of my setup ?
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on November 09, 2012, 07:50:06 am
This has been discussed before in other posts of dataMine. Access outside of your network (especially from Mios's remote servers) is not possible.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: maaxmelbourne on December 11, 2012, 09:04:05 pm
Greetings

Have one strange issue. All is working fine, but I try to log

a) inside temperature from the SQ Blaster
b) outside temperatire from Wunderground API

Both works well on the dashboard and SQ remote, but the logging throws both logs into one file, i.e both data sets in one graphical line. All other logs work without problem.

Any suggestion would be appreciated
Ax
Title: Re: New dataMine graphing and logging plugin
Post by: maaxmelbourne on December 11, 2012, 10:18:57 pm
OK, solution!

It appears that the logger does not take room or device id into account when generating the log files. Though, I shut down vera, pulled the USB stick, format and back into vera.

Datamine at this stage does not yet record.

Make sure that the channels you want to log have different names (again, different room does not matter!), though rename them in the dashboard

In my case: Temperature (1) -> Inside Temperature & Temperature (2) -> Outside Temperature

Go back to DM setup and configure channels for recording.

Voila, all good (and as I have just started logging, no big loss in data, only the last 24hrs which is not a drama)

Have fun, and by the way, a fantastic application!!
Ax
Title: Re: New dataMine graphing and logging plugin
Post by: maaxmelbourne on December 12, 2012, 03:24:51 am
Hi Chris

May I ask you to share some info about

a) the unit setting in the Variables Properties Section of the configuration tab?
b) what "drives" the y-axis scale? Some variables if graphed generate 2 unit scales, other just one which disturbs the graph

Example on b) is again the Wunderground API which logs beautifully, but when I graph wind direction and wind speed, the scale is translated to East/North/EastNorthEast etc overwriting the scale for wind from 1 to 10 (graph 1) or logging the direction full screen (graph 2)

Thanks for your feedback!!
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 12, 2012, 06:17:34 am
It appears that the logger does not take room or device id into account when generating the log files. Though, I shut down vera, pulled the USB stick, format and back into vera.

Datamine at this stage does not yet record.

Make sure that the channels you want to log have different names (again, different room does not matter!), though rename them in the dashboard
Yes, this is an issue that came up recently... DataMine uses the device name, and variable name to generate the filename. So, if you have the same device name for multiple devices, then this will generate the same filename. I started looking at putting a check into the software to stop this happening, but it's not completed yet as I've been away from home for the past week.

a) the unit setting in the Variables Properties Section of the configuration tab?
What is the question? Currently, this isn't used, but you can put W for Watts, C (or F) for temperature etc. Currently, dataMine uses the data type to put different channels onto the different axis, but I was considering changing this to use the "unit" setting as to me this makes more sense since sometimes different data types can still have the same data (if that makes sense!).

b) what "drives" the y-axis scale? Some variables if graphed generate 2 unit scales, other just one which disturbs the graph
I see your issue, but I'm not sure if I can change this. This is decided within the graphing library that I use - I don't think there's (easy) control over this I'm afraid.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: maaxmelbourne on December 12, 2012, 08:23:15 pm
Hi Chris

Thanks for the feedback. I thought that I may be able to force different axis by using the unit variable. Though, you plan sounds good.

Keep up the good work & Merry Christmas from Australia!
Ax
Title: Re: New dataMine graphing and logging plugin
Post by: skropko on December 21, 2012, 11:23:40 am
Chris,

I've been using dataMine for the last half year as it has been very helpful.  Thank you for the continued improvements.  One observation with the latest version.  I can no longer view my 'Thermostat - Mode State' data.  It appears to be logging ok in the dataMine directory but will not draw the graph when selected.  This had previously worked.  'Thermostat - Current Temperature' and 'Thermostat - Current Setpoint' continue to work and will draw the graph when selected.

Here is the contents of the dataMineConfig file...

{"Graphs":[],"Variables":[

{"Type":0,"Device":14,"Service":"urn:upnp-org:serviceId:TemperatureSensor1","Logging":1,"DataType":1,"Archive":"1 Thermostat - CurrentTemperature","LastRec":0,"Name":"Thermostat - CurrentTemperature","LastVal":0,"FirstRec":0,"Variable":"CurrentTemperature","Id":1,"Lookup":[]},

{"Type":0,"Device":14,"Service":"urn:micasaverde-com:serviceId:HVAC_OperatingState1","Logging":1,"DataType":0,"Archive":"2 Thermostat - ModeState","LastRec":0,"Name":"Thermostat - ModeState","LastVal":0,"FirstRec":0,"Variable":"ModeState","Id":2,"Lookup":[]},

{"Type":0,"Device":14,"Service":"urn:upnp-org:serviceId:TemperatureSetpoint1_Heat","Logging":1,"DataType":7,"Archive":"3 Thermostat - CurrentSetpoint","LastRec":0,"Name":"Thermostat - CurrentSetpoint","LastVal":0,"FirstRec":0,"Variable":"CurrentSetpoint","Id":3,"Lookup":[]}

],"nextId":4}



ID 1 and 3 work... 2 does not.

Here is the contents of the 2 Thermostat - ModeState [R2242].txt file...

1356106080,Idle
1356106600,Heating


I originally had 18 data points being logged.  I noticed some of them stopped drawing the graph in the latest update.  I removed dataMine (uninstalled the pluging and all the data in the dataMine folder) and did a clean install and added the 3 datapoints above but still have the issue.  Let me know your thoughts.

Thanks,
-Scott
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 21, 2012, 12:44:06 pm
Hi Scott,
I suspect a couple of issues... Firstly, I inadvertently commented out the code to draw graphs that have text data rather than numbers. I think it still should have graphed (??), but not printed the data correctly, but I'm not completely sure about that. The second issue is that as you've re-installed dataMine, you've lost some config data - the Lookup[] field in the config file is a conversion to numbers, and as this is now empty (due to the re-install) this will completely stop the graphing I think.

Now for the good news - I've fixed the initial problem, and the Lookup[] field will regenerate automatically as your thermostat mode changes over the next day or so...

Depending on how much time I get over the next day or so, I'll try and get a new version ready for MCV to approve on Monday (assuming they aren't on holiday!). Hopefully this will solve the problem - if it doesn't, let me know and I'll look into it again.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: skropko on December 21, 2012, 12:58:54 pm
Thanks Chris for the quick response.  I will keep an eye on the lookup field to see if it regenerates the values in the config.  I'll also let you know on the code update once you find the time and the updated version is approved and installed.  Happy Holidays. -Scott
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on December 23, 2012, 02:11:16 pm
I've install the new datamine plugin on a vera3 following the overview doc and setup my usb drive. My /dataMine folder has a dataMineConfig.json file in it now, yet In the dataMine plugin I only have the dataMine and Advanced tabs I don't see a graph and config tab. I've tried restarting the vera3 multiple times and removed and reinstalled the plugin. Is there something else I need to do or a file I need to manually download to the vera3 ?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 23, 2012, 02:16:11 pm
Hi,
I need to update the overview document. This has changed now, and the config tab has been removed from the plugin itself. Now, you need to go to the GUI (there's a link off the plugin), and there is a "Configuration" tab. Select the variable you want to log, and change the "LoggingEnabled" setting to "true".
I hope this helps - please let me know if you have any more problems...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on December 23, 2012, 05:48:22 pm
Ok thanks I got it working now, the other thing that I realized that you have to do is after you enable logging you have to click off of that row and click save at top in order to enable it.
Looks good and works well so far, I remember using this back on ui4 and my vera2, it's working much better on ui5 and a vera3.
Title: Re: New dataMine graphing and logging plugin
Post by: jpitts on December 28, 2012, 11:23:54 pm
Is there any easy way on a sensor to show how many times it triggered for the day, I have a lot of door/motion sensors, that show 0 or 1 on the graph, but I'd really like to just know how many times it's been triggered for the day,week,month etc.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 29, 2012, 05:37:21 am
No - at the moment you can't do this, however hour/day/month statistics is the next thing I want to add. Hopefully this will do what you want.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on December 29, 2012, 06:45:16 pm
Hi Chris

DM is working well over here - thanks for that.

Was wondering if you have thought of allowing an offset to be applied to the variables. In particular I have some temperature measuring devices and they are clearly out of whack. One is too high and the other is too low. An offset could be associated with the measuring device itself but this may be asking too much, given all the other request being made of the Vera OS. Regardless, I suspect that it's best to record the raw data and correct it at will later but there's room for debate on that one. Here's an example where a graph offset has been put in place:

http://board.homeseer.com/showthread.php?t=154432

Also would be possible to turn off the spline interpolation per variable graphed and what are the options for "GraphType" currently?


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 30, 2012, 05:30:49 am
Was wondering if you have thought of allowing an offset to be applied to the variables. In particular I have some temperature measuring devices and they are clearly out of whack. One is too high and the other is too low. An offset could be associated with the measuring device itself but this may be asking too much, given all the other request being made of the Vera OS.
I tend the agree with you - this probably ought to be done at the measuring device, but this will never happen! I'll take a look at this - it's easy to do, but it will have an impact on display speed as obviously I'd need to go through all data points to add the offset...

Also would be possible to turn off the spline interpolation per variable graphed and what are the options for "GraphType" currently?
Graphtypes are -:
0   Default (spline)
1   Line
2   Scatter
3   Area

Setting the type to line will turn off spline and will also ensure data values are maintained between points. By this, I mean that if a power value goes to 0 at midnight, and then goes to 100W at 6am, the graph will show 0 between midnight and 6am rather than a diagonal line from 0 to 100...

I've actually updated the PDF guide with this and had intended to upload it to the forum yesterday, but it's crept over the 512k filesize limit so I need to work out what to do with it...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: mfp on December 30, 2012, 08:57:36 am
Chris,

Great plugin!

Saw you had trouble with the file limit for the PDF Guide, here are some ideas:
1) Compress the pictures some more to get under the file limit.
2) Split the document in a userguide and installation guide.
3) Put more info on http://code.mios.com/trac/mios_datamine.

By the way is it possible to make a user option to display both the Config and Graph from within Vera instead on a separate tab in the browser?

Mark
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on December 30, 2012, 12:20:25 pm
Saw you had trouble with the file limit for the PDF Guide, here are some ideas:
1) Compress the pictures some more to get under the file limit.
2) Split the document in a userguide and installation guide.
3) Put more info on http://code.mios.com/trac/mios_datamine.
1) I tried, but couldn't get it below the 512k limit. 3) is my default - not actually to add more data to the web page as that's not as easy as writing a document and publishing it as a PDF (the Trak wiki software used for code.mios.com is a bit clunky) - but I could put the PDF on the webpage and link to it from the forum. 2) I like though and I might see if this works...

By the way is it possible to make a user option to display both the Config and Graph from within Vera instead on a separate tab in the browser?
Not really - this is kinda what I had in UI4. The graph was limited to the small properties window, and doing much in a config screen was a bit difficult due to real-estate limitations. I initially tried to work with this when I moved to UI5, but it just wasn't happening. While there are a few limitations with having a separate page, I think in general the benefits outweigh them.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin calibration ?
Post by: bucko on January 08, 2013, 09:05:34 pm
Hi Chris,
I have a couple of temp sensors that come built in to a Zwave plug in socket controller. The problem is that they are way off accuracy probably due to the internal heat of the controller. So I want to add an error correction to the datamine channel to adjust for this sensor error. So basically channel 2 (temp sensor) is sensor temp (-12), or whatever number I need to get this sensor closer to the actual room temp.

How can I do this?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 09, 2013, 04:50:13 am
I have a couple of temp sensors that come built in to a Zwave plug in socket controller. The problem is that they are way off accuracy probably due to the internal heat of the controller. So I want to add an error correction to the datamine channel to adjust for this sensor error. So basically channel 2 (temp sensor) is sensor temp (-12), or whatever number I need to get this sensor closer to the actual room temp.
Hi
This will be available in the next version which is currently awaiting approval by MCV. You will be able to set an offset per channel...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: nathanhall on January 17, 2013, 08:36:59 pm
I just installed dataMine and (mostly) got it working but I do have a few questions as I'm very excited to start digging into this.

The dashboard doesn't appear to display all three graphs for me - it only displays the energy consumption donut and the whole-house gauge but not what looks to be a plot over time of energy usage in the manual (figure 10, page 9).   

Also - is there a way to combine variables into a single graph plot?  Say for example I want to combine several lights in one room into a single plot - is that possible?  Alternatively, is it possible to plot all energy use over time in the "chart" tab (e.g. the total wattage used over time in a single variable to show total energy usage)?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 18, 2013, 05:27:05 am
The dashboard doesn't appear to display all three graphs for me - it only displays the energy consumption donut and the whole-house gauge but not what looks to be a plot over time of energy usage in the manual (figure 10, page 9).   
The third graph should (currently) display the "whole house" data for the past 24 hours. So, it should select the energy monitor with cat-1 (if you have more than 1, then I think it will select the first one).  Currently, I've noticed that this displays a wrong graph anyway, and this graph is really a placeholder for the next stage where I hope to have usage trends displayed here. So, I don't know that I'd get too worried about this at the moment. Oh - one other thing - this third graph is only in the latest version, so if you're a version behind, then you'll just get the other 2.

Also - is there a way to combine variables into a single graph plot?  Say for example I want to combine several lights in one room into a single plot - is that possible?  Alternatively, is it possible to plot all energy use over time in the "chart" tab (e.g. the total wattage used over time in a single variable to show total energy usage)?
This isn't something that can currently be done. I did think about making the donut graph selectable between categories and rooms, but I haven't done this. I'm happy to think about this once I've finished getting the history stuff integrated reliably (it's not so simple having to account for random Vera/Lua restarts when you're post processing data). The history may support some of what you're after in that it will derive usage stats on an hour/daily basis (and maybe month), and I then hope to support combining these in different ways (ie by category, room...). But this isn't currently planned for the raw data...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: mrtrilby on January 18, 2013, 12:49:23 pm
Hello, I've been running the dataMine plugin for a while and it's fantastic, but it now seems to have stopped working. The interface just reports "Loading dataMine web application for Vera" but never actually loads. Judging by the files on the USB stick, the plug-in is still running. I thought about uninstalling and re-installing the plug-in to fix it, but I don't see it listed in the "Install Apps" page any more, so I'm a bit worried about not being able to get it back again.

Is its removal from the Vera Apps a temporary thing? Is there a simple way to fix the web application?

Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 18, 2013, 01:14:26 pm
What version of dataMine are you using? The later versions shouldn't sit there with this screen - if they are having trouble, they should report an error instead...

When dataMine starts, it tries to download a bunch of information from Vera - some direct from Vera, and some from the dataMine plugin. That's what's happening when you get this message. The new version will report where it's failing, so I'm wondering if you're using an old version?

Normally this is indicative of the plugin not working, but if the txt files continue to update, then it's probably not this....

I would suggest -:

You could try reinstalling - so long as you don't scrape the USB stick, everything should still be there when you restart (to be 100% sure, take a backup of the /dataMine directory by copying it to your PC). Personally, I'd like to try and understand the error if possible rather than just reinstalling so I can fix it if there's an issue, but if you want to try a reinstall, then up to you.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Poe1 on January 23, 2013, 07:45:57 am
Hi Chris,

First, thanks for the app - I'm a data junkie so to me, what's a home automation system without the ability to log data?!

Anyway, your datamine plugin was working good for me for a few weeks, but now I just get the message "Downloading Graph Data" (see attachment) on the screen and nothing ever updates. This message will stay up for a while (couple minutes?) and then go away and just leave a blank graph.

I'm using plugin version 0.965 running vera thru Chrome and Datamine thru Firefox.  Any ideas what I could do to fix it?

Thanks,

-Casey
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 23, 2013, 08:02:33 am
Hi
Do you have the graph type set to 1? If so, this may cause the problem if there's a lot of data (given the data is an energy variable).
Does it work for other channels?

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Poe1 on January 23, 2013, 08:34:54 am
Yep, it was set to 1 - and there's a lot of data with the energy variable.

I changed it to zero and it's working now!!

Thanks!

-Casey
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 23, 2013, 08:49:40 am
Good stuff. :)

In the next version this won't just hang like this - instead it will return with the data it has within a timeout period and provide an error message. I need to work out a better way to handle this data...

Cheers
Chris
Title: Graphing times triggered
Post by: brian.k on January 24, 2013, 10:40:47 am
Is there any easy way on a sensor to show how many times it triggered for the day, I have a lot of door/motion sensors, that show 0 or 1 on the graph, but I'd really like to just know how many times it's been triggered for the day,week,month etc.

jpitts (and Chris and the rest of the faithful contributors).  Can't this issue be solved outside the plugin with a stored variable that you increment each time the device(s) are triggered?  Maybe even having separate incremented variables for the zero and one states?  That wouldn't give you date/time, but counts.  You could even schedule a reset of the counter variable(s) on a daily/weekly/other basis.

I haven't tried it, just thinking out loud.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 24, 2013, 12:28:08 pm
Is there any easy way on a sensor to show how many times it triggered for the day, I have a lot of door/motion sensors, that show 0 or 1 on the graph, but I'd really like to just know how many times it's been triggered for the day,week,month etc.

jpitts (and Chris and the rest of the faithful contributors).  Can't this issue be solved outside the plugin with a stored variable that you increment each time the device(s) are triggered?  Maybe even having separate incremented variables for the zero and one states?  That wouldn't give you date/time, but counts.  You could even schedule a reset of the counter variable(s) on a daily/weekly/other basis.

I haven't tried it, just thinking out loud.
Yes, it could be done in a number of ways, but I think in this case, it's probably best (easiest) if done in dataMine, and I think it fits with what I am doing. Statistics is something I'm working on at the moment - initially there will be a basic set of stats available (min/max/average/delta - per hour/day), but once I get this working, I can add more reasonably easily (I hope). The question will be how I squeeze this into the GUI as there may then need to be a way to select different type of stats per channel....

Oh the fun to come.......  ;)

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: brian.k on January 24, 2013, 05:16:00 pm
Chris.  Thanks for the reply.  I'm still wrestling with the problem many others have asked about... trying to figure out if there is a way for me to grab periodic snapshots of the graphs and be able to access them via remote MIOS.  The ugly solution would be to run a local copy with a webcam and no screensaver.   :( No way I'll settle for that.  Beyond that, I'm playing with VPN/DMZ and other options since the graphs are so cool and vital and there are some Vera's I administer that I can only access remotely during business hours.

Hope your development goes well.  - Brian
Title: Re: New dataMine graphing and logging plugin
Post by: 480vac on January 29, 2013, 10:39:30 pm
Am I missing something?  I can't find datamine in the install apps page - is it no longer available?   I was going to update from 0.962 but can't figure out how to do that...   Sorry if this is a newbie question, but I don't think it is.

Geoff
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 30, 2013, 05:23:26 am
Am I missing something?  I can't find datamine in the install apps page - is it no longer available?
dataMine is still available in the app store - http://apps.mios.com/plugin.php?id=1088 (http://apps.mios.com/plugin.php?id=1088).

I was going to update from 0.962 but can't figure out how to do that...
There's a post here http://forum.micasaverde.com/index.php/topic,12877.msg95089.html#msg95089 (http://forum.micasaverde.com/index.php/topic,12877.msg95089.html#msg95089) that describes how to manually update. Unfortunately it's not as straight forward or intuitive as it should be  :(.
In future, I think it ought to update automagically - maybe!

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 31, 2013, 08:56:11 am
I've added a few features to dataMine. There have been a few fixes/additions under the hood, but the main changes are -:
  • Multiple channels are now graphable - currently up to 4 channels are supported.
  • Graphs are now in local time
  • A legend has been added to the graph. This can be disabled using a button on the toolbar
  • You can disable mouse tracking using a button on the toolbar

The system allows you to select multiple channels and place them onto the left or right Y axis to allow for channels with different scales. This is shown in the attachement. When you click on the "Select Channels" button, there are four fields of interest on each line. First, there's an arrow button. Pressing this will instantly graph this channel with no further action required. Next is the variable name. Clicking on this selects the channel, and enables two further buttons. The first button on the right is a "go" button - it will display all variables currently selected. The second button allows you to select two the left or right axis. This is extremely useful if you have different types of data (say, temperature and humidity) which would otherwise not graph well on the same scale.

It may take a little bit of getting used to, but it's a lot more powerful than a simple dropdown list! I'm  open to suggestions on improving this if it's deemed to be too complex.

If you're upgrading, note that there are a number of new PNG files to support the new buttons and features of the software, so please ensure you upload these as per the installation instructions. The LUA, JSON, XML and JS files also need to be updated via the "MiOS Developers" button.

A wiki has also been started at http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine).

Very old post... very new quote... very much in need of more than 4 selecatble channels. Could that be possible? In the latest datamine stil 4 channels are selectable...
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on January 31, 2013, 12:50:33 pm
Very old post... very new quote... very much in need of more than 4 selecatble channels. Could that be possible? In the latest datamine stil 4 channels are selectable...
Yes - this is possible. However, there are constraints - you will be able to graph more channels, but if each channel has a lot of data, then it may cause problems so it will be up to the user to understand this. There is a timeout in the next version, so the "problems" won't be major - it will just mean that if you hit the limit, some channel data may be truncated and you'll get a warning...

I probably wont change this in the next release, so maybe another couple of weeks...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on February 01, 2013, 01:22:04 am
Hi Chris, did you mean it is currently possible to have more than 4 channels?  Or in next release?  And if already possible,  how?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 01, 2013, 09:57:35 am
Hi Chris, did you mean it is currently possible to have more than 4 channels?  Or in next release?  And if already possible,  how?
Sorry for my poor explanation. I meant it's possible for me to do, but it's not in the current version. There's a new version just approved today, and it's not in that either  :( sorry - I'll try and get it into the next version though...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on February 01, 2013, 10:53:20 am
Oh, read here... sorry already posted in section for version 0.966 also... it's a pity! I hope i can select 5 channels quickly!

Thanks in advance!
Title: Re: New dataMine graphing and logging plugin
Post by: cmbeek on February 01, 2013, 03:47:11 pm
Every time my vera reloads all my logging configuration is deleted.....
Any idea what causes this behaviour?
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 06, 2013, 01:01:44 pm
Every time my vera reloads all my logging configuration is deleted.....
Any idea what causes this behaviour?

I've had the same problem with every version I've used.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 06, 2013, 01:19:57 pm
Every time my vera reloads all my logging configuration is deleted.....
Any idea what causes this behaviour?

I've had the same problem with every version I've used.

This is probably caused by USB mounting (or not).  Please can you advise what Vera you have (2, 3, Lite), how many USB sticks you have, and if possible, please send the logs (from WinSCP, click on the Terminal button, and type "cat /var/log/cmh/LuaUPnP.log | grep dataMine"). The logs are most important, as without this it's difficult to see what's going on.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 06, 2013, 01:45:24 pm
Every time my vera reloads all my logging configuration is deleted.....
Any idea what causes this behaviour?

I've had the same problem with every version I've used.

This is probably caused by USB mounting (or not).  Please can you advise what Vera you have (2, 3, Lite), how many USB sticks you have, and if possible, please send the logs (from WinSCP, click on the Terminal button, and type "cat /var/log/cmh/LuaUPnP.log | grep dataMine"). The logs are most important, as without this it's difficult to see what's going on.

Chris

I have not used 6 months.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 08, 2013, 11:21:30 am
Chris,
I uninstalled the old version and installed the new, from Vera's App tab. I put in the channels 3 times now... they disappear every day? I do not have a USB drive installed. The last version was working fine.

Is there anything I need to config in the dataMine settings after installing?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 08, 2013, 11:31:59 am
Chris,
I uninstalled the old version and installed the new, from Vera's App tab. I put in the channels 3 times now... they disappear every day? I do not have a USB drive installed. The last version was working fine.

Is there anything I need to config in the dataMine settings after installing?
Hi Aaron,
You need to set up the USB mount (or not), so yes, there are things that you need to configure in dataMine even if you're not using a USB stick.

Please can you post your settings (maybe a screen dump of the advanced page), and any dataMine log entries. I'll take a look and see if I can see anything.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 08, 2013, 05:25:40 pm
I forgot about the fields, for some reason I was thinking it defaulted to a setting for no USB. I'm adding a USB.

two feature suggestions...

1) GUI warning if a user attempts to setup channels without having the Vera settings config'd  :)

2) it would be cool if there was an option to store the config (channels, etc) on the USB too... this might help those of us whos config is wiped with every update of dataMine.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 09, 2013, 03:50:35 am
1) GUI warning if a user attempts to setup channels without having the Vera settings config'd  :)
If you haven't configured at least some of the dataMine config, then the plugin shouldn't start, and you shouldn't be able to set up the variables... Maybe I need to scrape down my old Vera 2 and try a new install just to check this out as I haven't done this for a few months...

2) it would be cool if there was an option to store the config (channels, etc) on the USB too... this might help those of us whos config is wiped with every update of dataMine.
The channel configuration is stored on the USB - all dataMine configuration, with the exception of the USB setup and a few other things line this, are stored in the dataMineConfig.json file in the /dataMine directory which should be on the USB... This is why I' surprised that your config is being wiped - it almost has to mean something is scraping down the USB (or something like that).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 09, 2013, 10:46:13 am
Chris,
Yes, it was trying to load the GUI before I had input anything in the Advanced field. I have it logging now on USB.

One seemingly new issue:
In the past I had mentioned some strangeness where I was seeing odd things with channels... like the parent Thermostat having a child Door Lock. That is still there, but now I'm also seeing (see screenshot)...

Low Temp, High Temp, Humidity ... should all be under World Weather just like WindSpeed, etc.
Temperature ... is both under Thermostat and made its own parent. I verified the 'Service' under Properties is identical.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on February 09, 2013, 12:22:12 pm
Aaron,

Low Temp, High Temp, and Humidity are under their respective child devices that the World Weather Plugin creates. The service id's are not part of the main device.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 09, 2013, 12:53:27 pm
I see... this is quite confusing :)

I noticed many of the Category labels show Undefined. Maybe that can be more descriptive as to explain what the item is (Event, App, Parent Device, etc)?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 09, 2013, 01:12:58 pm
I see... this is quite confusing :)
It can be, and without wanting to confuse you more, you may find things turn up in more than one place  ???. I have an ST814 temp/humidity sensor - you will find the temperature and humidity under the parent device, and under the children it creates...


I noticed many of the Category labels show Undefined. Maybe that can be more descriptive as to explain what the item is (Event, App, Parent Device, etc)?
Unfortunately, I can't really just make it up  ;) Undefined, is the category as provided by Vera. This is simply showing how the variable is set up in Vera. Often with plugins, the category isn't set, so this is what you get.

All items in this list are "devices" - a device can have an app behind it, or it can be directly from an in-built Vera device. However, they are all still devices, and there's no real way to differentiate - other than with the category. Sorry... If anyone has any ideas on how better to handle this, I'm happy to look at it - I actually think that "category", as defined by Vera, isn't as useful as it could be anyway (but that's another story  ;)).

Cheers
Chris

Title: Re: New dataMine graphing and logging plugin
Post by: gengen on February 09, 2013, 06:36:38 pm
I also ran into the "all graphs deleted" problem if not logging to a USB stick. Defaults for USB or non-USB should be set up in the GUI without having to go into the command-line and do all sorts of setup.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 09, 2013, 07:31:19 pm
... Defaults for USB or non-USB should be set up in the GUI without having to go into the command-line and do all sorts of setup.
It seems the Vera has very basic install methods/abilities so I don't think the Vera allows for the install of apps to interact with the user... to asking things like "Would you like to use USB?". I think the way Chris has it today is fine - it would be nice to have the dataMine web GUI check for the proper settings an let the user know if settings are missing. This would solve the problem IMO.

btw... nothing needs to be setup using command line - it is all in the Vera UI
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on February 09, 2013, 09:33:31 pm
Hello Chris,

I want to try out your plugin. I assume I need to go to apps.mios to download it. But where do I find a guide for installing correctly to my Vera 3? I want to use a USB stick in the upper slot and also install another USB stick in the lower slot for Vera USB logging.

What is is method to do this so that I am assured that datamine logs to the upper slot and Vera logging only logs to the lower slot? I am worried that the mounts will get screwed up with each other.

Do you have an updated install docs somewhere I can use?

Thanks, Jack
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 10, 2013, 12:33:09 am
Hello Chris,

I want to try out your plugin. I assume I need to go to apps.mios to download it. But where do I find a guide for installing correctly to my Vera 3? I want to use a USB stick in the upper slot and also install another USB stick in the lower slot for Vera USB logging.

What is is method to do this so that I am assured that datamine logs to the upper slot and Vera logging only logs to the lower slot? I am worried that the mounts will get screwed up with each other.

Do you have an updated install docs somewhere I can use?

Thanks, Jack


the first and only bolded post title in this forum... "dataMine manual"
Title: Re: New dataMine graphing and logging plugin
Post by: svaleb on February 10, 2013, 03:41:14 am
Hi Cris
This is great work, thanks.
I have been following the development fore some time. At first I tried DM with the same stick as vera, but as foretold it went wrong.
Then there has been some confusion about how to maunt a stick into the othervera slot...and I halted.
Happy to see there is a solution now. I see there is new documentation.

@bucko
Thanks for putting the question direktly "can it bee done, or what".
Regards
           CE
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 10, 2013, 04:31:34 am
Defaults for USB or non-USB should be set up in the GUI without having to go into the command-line and do all sorts of setup.
This is a good idea. Following some discussion with @hek regarding setup, I was also starting to think along these lines. As @Aaron said, Vera doesn't offer much in the way of setup support, but I can probably put something into the GUI...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 10, 2013, 04:40:27 am
I want to use a USB stick in the upper slot and also install another USB stick in the lower slot for Vera USB logging.

What is is method to do this so that I am assured that datamine logs to the upper slot and Vera logging only logs to the lower slot? I am worried that the mounts will get screwed up with each other.
A very valid concern. @hek recently pointed me to a feature of USB where a unique ID is put on the USB stick. DataMine can now find the mount point based on this "UUID", and from further reading on the topic, it seems this is (nearly!) the only 100% guaranteed way to ensure mounting works properly under Linux (ie the mountpoint is the same each time).

As suggested by @gengen, I will look at providing something in the GUI to ease setup, however if you are familiar with the WinSCP program, then you can find the UUID by running the "blkid" program.  You just need to find out which ID is relevant to which USB stick, and put this into the configuration SetMountUUID. I highly recommend this configuration method...

I don't know how Vera mounts the USB stick for the logging, so I'm afraid I can't be sure that it won't get confused. If it also uses UUID, then I believe there won't be a problem.

Do you have an updated install docs somewhere I can use?
I try and keep the docs updated - they can be found here -:
http://forum.micasaverde.com/index.php/topic,13057.0.html (http://forum.micasaverde.com/index.php/topic,13057.0.html)

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on February 10, 2013, 09:02:45 pm
@CHris

Thank you for the quick response. You are really doing a good job working through the various issues and ideas with your plugin development. Again, great job here.

I came to the conclusion that Vera logging to USB seems to work if it is the only USB thing running. If I reboot my Vera 3 with USB logging, it shows CHKdisk errors, then tries to reformat the USB stick, fails again, etc. This is while ONLY using logging and Datamine uninstalled!

Once I try to add a datamine stick and set it up, USB goes bonkers, and once it fails to read it's own USB stick, it simply takes over the datamine stick and reformats it for logging. Then of course datamine fails. I am not a programmer, but it looks like Vera logging has some issues and should be looked into by MCV.

That said, I stopped USB logging, rebooted my Vera, re installed dataMine, and put in a 250 mb USB stick in the Vera UPPER slot. I ran blkid and got this ..../root$ blkid
/dev/sda: UUID="60BA-4C9C"

Also in my Vera log it reads.....
50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>
50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>

Isn't is reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Can you help to explain this please?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 11, 2013, 07:07:46 am
Hi @bucko,
Firstly, thanks for your thanks  :)

A bit of background first - I suspect you probably know most/all of this, but it may be useful for others...

I recently became aware of a feature of USB where the device has what's called a UUID - Universally Unique ID. I'm not completely sure how unique it is since it seems to change when you reformat the drive, but it will still identify a drive without ambiguity...

From what I've read, this is almost the only way to guarantee that a drive gets mounted correctly. My original thought when I started DM was that different USB sockets in Vera have different mount points, and to some degree this seemed to hold true, but not completely. Certainly, if you have a hub connected, each time you power cycle Vera, the USB stick will end up mounting to a different point - seemingly randomly (at least with my limited understanding!).

So, I've implemented the mount system that uses the UUID (which you appear to be using), and in the next version this will become the default system for dataMine mounting USB sticks. The way it works is that dataMine runs a program called "blkid" which gives you the UUID, and the associated mount point. From this, I can find the correct mount, and mount the drive...

I don't know how Vera mounts the drive for USB logging - I might try and look through the files to see if I can work it out. However, with what I now know (as above) the mount point can change when you have multiple USB sticks, so I'm not completely surprised by what you've seen (I may have had a different view a couple of weeks back!).

My hope is that by using the UUID feature within dataMine, that at least dataMine should mount correctly every time. However, unless Vera is also using this, I'm not sure if we can guarantee that Vera won't go and use the dataMine USB stick for logging - effectively mounting it twice, and we know "bad stuff" happens if you do this.

So, on to your question (sorry it's taken a while to get there ;)).
The mount point doesn't link to a physical USB port as I previously thought - it can change each time Vera starts...

Have the problems you've seen been happening since you have been using the UUID feature, or was the corrupted disk (etc) before changing to this?

I'll see if I can find out how Vera mounts the log USB drive - this may help...

I hope that helps at least a little - I understand this isn't as simple as one would otherwise hope!

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: svaleb on February 11, 2013, 07:20:55 am
Hi Chris

My vera formatted the usbstick for vera loggging to 512  it vas a 2gb stick and I cant reformat it to 2gb.
I dont know if there has been something wrong with it from the start.
But if vera formats a 2gb stick for its own use to 500thomthing I will be very careful to put in another stick untill I am sure that it wont happen agin.
Others who have had this eksperience ???
I wil wait ontil you have solved the problem, you are doing a great job, THANK YOU.
Regards
           CE
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 11, 2013, 11:29:05 am
My vera formatted the usbstick for vera loggging to 512  it vas a 2gb stick and I cant reformat it to 2gb.
I dont know if there has been something wrong with it from the start.
But if vera formats a 2gb stick for its own use to 500thomthing I will be very careful to put in another stick untill I am sure that it wont happen agin.
Others who have had this eksperience ???
I wil wait ontil you have solved the problem, you are doing a great job, THANK YOU.
Hi @svaleb,
I think you are getting confused - this is nothing to do with dataMine - the Vera logging is an internal Vera function and not something that I am doing. The wiki information here http://wiki.micasaverde.com/index.php/Store_logs_USB (http://wiki.micasaverde.com/index.php/Store_logs_USB) does indeed say that it formats up for 512M, but I would have thought that you could still reformat it....

For dataMine, you do not need to use the Vera USB logging. dataMine uses a pre-formatted USB drive, so you don't need to worry about this problem.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: svaleb on February 11, 2013, 01:30:54 pm
Hi Chris

No, I am awere that it is not DM.
But if I put in the other stick vera might bee "confused" and format that one too.
And it was not a 2gb stick but an 8gb stick, and I cant reformat it so...bad luck.
I have seen DM work and it is great, now it is how to store it.
Thanks for your answer looking forward to a solution.
Regards
         CE
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on February 11, 2013, 01:41:18 pm
You should be able to recover the full size of the usb drive. Either in linux or in windows. In windows you'll need to go into the disk management console. For linux, you can use fdisk.

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 11, 2013, 01:45:45 pm
Hi Chris

No, I am awere that it is not DM.
But if I put in the other stick vera might bee "confused" and format that one too.
And it was not a 2gb stick but an 8gb stick, and I cant reformat it so...bad luck.
I have seen DM work and it is great, now it is how to store it.
Thanks for your answer looking forward to a solution.
Ok - I don't think Vera will reformat it - not unless you go into the USB logging section in UI5 and ask it to reinstall USB logging.  You should be fine to put in another stick.

Take a look at the other post I've just put in the dataMine folder, which presents my best guess as to what's happening in Vera. I think if you avoid using Vera to format the USB stick that dataMine uses, it should work ok.

The next version of DM has an easier install in the GUI that allows selection of the USB UUID and should ease configuration...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: svaleb on February 11, 2013, 02:06:56 pm
Thanks Garretwp and Chris
Yes, I let vera format the first stick and I tryed to reformat it in windowes, but could not do it. Vera has it back in for logging now.
The other 8gb has been formatted in windowes and wil be redy for the next version of DM.
Thanks again both of you !
Regards
       CE
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on February 11, 2013, 08:46:00 pm
@Chris,

On my last post I mentioned I had this report in WinScp with the blkid command.


50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>
50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>


I went to the dataMine folder by wINsCP'ing into my Vera and noticed that the data is writing to this folder and not to my USB stick in the upper Vera USB slot. DM advance tab is set with my UUID of my stick ok.  "mount point = 0" and "manual mount point = (blank) and "mount location" = /dev/sda

Isn't this reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Why is mine reporting /dev/sda while my USB is physically plugged into the upper slot? Should I go ahead and change DM mount location to /dev/sda1 even though the stick is in the upper slot? I'm a bit confused by this.

Right now I turned off Vera USB logging and only have  my DM stick on my Vera (upper slot) because Vera logging was corrupting my DM stick. I think maybe it has something to do with the fact my Vera slots seem to be reversed.

On my Vera it appears my upper slot is /dev/sda and the lower slot is /dev/sdb according to blkid. I just verified this again by plugging in my stick to Vera lower and blkid reports it as /dev/sdb. BTW, there is no "1" shown at the end from the blkid command result.

What do you make of that?

Title: Re: New dataMine graphing and logging plugin
Post by: svaleb on February 12, 2013, 01:01:46 am
Yes, I have found out the same as bucko.
CE
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 12, 2013, 04:52:21 am
Quote
On my last post I mentioned I had this report in WinScp with the blkid command.50   02/11/13 9:12:26.831   luup_log:109: dataMine: UUID = /dev/sda: UUID="60BA-4C9C" <0x2b56b680>50   02/11/13 9:12:26.849   luup_log:109: dataMine: Found UUID '60BA-4C9C' at '/dev/sda' <0x2b56b680>
This looks fine.  

Quote
I went to the dataMine folder by wINsCP'ing into my Vera and noticed that the data is writing to this folder and not to my USB stick in the upper Vera USB slot.
??? I don't understand this? dataMine DOES write into the /dataMine directory - this is correct. The dataMine directory IS the USB stick - that's how drive mounting under Linux works. The USB mounting maps itself into the file system - you don't get a separate drive letter or whatever like you do under Windows.  

Quote
DM advance tab is set with my UUID of my stick ok.  "mount point = 0" and "manual mount point = (blank) and "mount location" = /dev/sda
Again, this looks fine.  

Quote
Isn't this reporting a stick plugged into the lower USB slot? Your chart says upper should report /dev/sdb1. Why is mine reporting /dev/sda while my USB is physically plugged into the upper slot? Should I go ahead and change DM mount location to /dev/sda1 even though the stick is in the upper slot? I'm a bit confused by this.
No - as I've said elsewhere, there appears to be no way of mapping a physical USB slot to a mount point such as /dev/sda1 under Linux. Others have seen that the device mount points change - especially with USB logging. From the reading I've done on this, the only way to ensure you map correctly is to use either the UUID (which is preferred) or the LABEL. I've written another post on this  http://forum.micasaverde.com/index.php/topic,13634.0.html (http://forum.micasaverde.com/index.php/topic,13634.0.html).  

Quote
Right now I turned off Vera USB logging and only have  my DM stick on my Vera (upper slot) because Vera logging was corrupting my DM stick. I think maybe it has something to do with the fact my Vera slots seem to be reversed. 
The slots aren't "reversed" - they just don't relate directly to the mount points. How did you format the drives used by dataMine and the USB logging? IF you used the Vera USB logging to format both of them, then I think this is likely to cause problems. Again, take a look at the post I wrote on USB logging http://forum.micasaverde.com/index.php/topic,13634.0.html (http://forum.micasaverde.com/index.php/topic,13634.0.html)

  
Quote
On my Vera it appears my upper slot is /dev/sda and the lower slot is /dev/sdb according to blkid. I just verified this again by plugging in my stick to Vera lower and blkid reports it as /dev/sdb. BTW, there is no "1" shown at the end from the blkid command result.
Again, dataMine doesn't use the /dev/sd** to mount drives - it uses the UUID which is the preferred method. Also, I don't think USB logging uses the /dev/sd** either - I believe it's using the LABEL definition that you may also see in the blkid command output.  

Quote
What do you make of that?
I hope the above explains what I believe is happening. I don't profess to be an expert in Linux by any means! If others have better knowledge, I'm willing to be updated (please!). However, from everything I've read, and from discussions with other users, I believe the above to be correct. 

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on February 12, 2013, 09:08:21 pm
@Chris

The slots aren't "reversed" - they just don't relate directly to the mount points. How did you format the drives used by dataMine and the USB logging?

I formatted the stick on my PC as a fat 32 . datamine advanced shows it as a vfat as well. Under the directory it reads /tmp/log/cmh/datamine

 dataMine is working ok. But I did WinScp into my vera and looked into the directory location (tmp). It shows new files written in there. Is this ok or should ALL files be written only to the USB stick? How can I tell if files are being written to the USB stick? Do I just unplug it and look at it with my PC?

WinScp blkid does show my stick there and it's UUID is the same I put into datamine advanced tab.

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 13, 2013, 04:25:46 am
dataMine is working ok. But I did WinScp into my vera and looked into the directory location (tmp). It shows new files written in there. Is this ok or should ALL files be written only to the USB stick?
Where abouts in tmp? I see you have your dataMine directory mapped to /tmp/log/cmh/datamine, so all dataMine files should end up there. There is a small exception to this, and that is I write a temporary file into the /tmp directory, but it should be deleted before you see it...

How can I tell if files are being written to the USB stick? Do I just unplug it and look at it with my PC?
Yes, I think the only way to be 100% sure the files are going onto the USB stick is to plug it into your PC and take a look. The "problem" is that the way this works is you create a directory in the file system, and you mount the USB into that directory. So, with or without the USB, the data will be written into that directory when you look at the listing with WinSCP (or anything else). If the USB isn't mounted correctly, then the files just go into the standard file system, if it's ok, then they go onto the USB. As far as I know, there's no way to see this directly other than to unplug the USB (any Linux experts feel free to comment).


WinScp blkid does show my stick there and it's UUID is the same I put into datamine advanced tab.
I suspect it's mounting working ok, but you do have a slightly non-standard configuration with the directory in /tmp. This is normally only suggested if you're using Veras USB logging and dataMine on the same stick...  I think you said earlier that USB logging is disabled now, but if you previously had USB logging enabled, and dataMine also mounting the USB to this directory, then this may cause problems. I'm not 100% sure what would happen - it might be ok, but you would end up with the USB logging mounting to /tmp/log/cmh, and then dataMine mounting to a subdirectory of this. I don't know enough about Unix/Linux file systems to know if this is allowed, but it seems "dodgy".

Another problem you may also be having with this configuration. I think that the /tmp/log/cmh is normally mounted to a RAMdisk (I'm not 100% sure about this, but I think it's the case). In this case, you would be mounting into a volatile directory - again, I don't know enough to say if this is good, bad, or otherwise....  Sorry.

I would suggest that you change the dataMine directory back to "/dataMine" - this would avoid any potential problems that may occur with the above uncertainties. If you are trying to do something "special", then let me know what you want to achieve and I'd be happy to give you my thoughts...

Mounting of file systems under Unix/Linux isn't as straight forward as it is under Windows - it's probably a lot more powerful, but with that comes a lot more potential for confusion! Keeping things simple is definately best  :).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on February 15, 2013, 12:22:41 am
I would suggest that you change the dataMine directory back to "/dataMine" - this would avoid any potential problems that may occur with the above uncertainties.

I checked the USB stick and it is logging files there ok. I went ahead and changed the directory to /datamine and redid my channels.

Thank you
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 20, 2013, 10:46:48 am
Chris,
the updated interface it very nice, very functional - nice work!

2 Questions...

1) dataMine shows my Trane thermostat with ton of channels, many more than the User manual shows is Zwave addressable.  I cannot tell what they are/do. Some seem to have data being sent but I have no way to interpret and since many of the items are named nearly identical it is hard to know what does what.  Is there any way to get more info on what these are?  Here is the manual's info...

Zwave Command Class Comments
COMMAND_CLASS_THERMOSTAT_MODE The operating mode of the TZ43 thermostat
COMMAND_CLASS_THERMOSTAT_OPERATING
_STATE
Returns the operating state of the HVAC system
COMMAND_CLASS_THERMOSTAT_SETPOINT The TZ43 Setpoint (heat or cool)
COMMAND_CLASS_THERMOSTAT_FAN_MODE The Fan Mode
COMMAND_CLASS_THERMOSTAT_FAN_STATE The HVAC system mechanical fan state
COMMAND_CLASS_SENSOR_MULTILEVEL Returns the temperature as displayed by the TZ43. Note this
temperature is the TZ43?s working temperature and includes any
averaging sensors
COMMAND_CLASS_VERSION Returns various Zwave version information
COMMAND_CLASS_CLOCK Used to set the TZ43?s real-time clock
COMMAND_CLASS_MANUFACTURER_SPECIFIC Returns Manufacture and Product information
COMMAND_CLASS_CONFIGURATION Used to read/write internal TZ43 variables. The majority of the
variables are accessable via the on-screen menu, the
CONFIGURATION command class is used to remotely set the
variables. It is recommended that you have a thourough
understanding of the variables before making any changes. See
variable table below
COMMAND_CLASS_LOCK Used to implement the Display Lock, locking the thermostat keys


2) in the old interface we could change a specific channels line display to be status, thus only showing it as absolute on/off. Where is the setting for that? my thermostat 'mode' shows as a ramp, it should only be on or off.

thx
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 20, 2013, 12:27:41 pm
1) dataMine shows my Trane thermostat with ton of channels, many more than the User manual shows is Zwave addressable.  I cannot tell what they are/do. Some seem to have data being sent but I have no way to interpret and since many of the items are named nearly identical it is hard to know what does what.  Is there any way to get more info on what these are?  Here is the manual's info...
Not that I know of. Also, what dataMine shows are the variables mapped to MCVs UPnP services - that's not necessarily the same as the device classes. MCV have additional variables that define how Vera uses the data, over and above those that are returned from the device...

2) in the old interface we could change a specific channels line display to be status, thus only showing it as absolute on/off. Where is the setting for that? my thermostat 'mode' shows as a ramp, it should only be on or off.
Yes - you can change the graph type. Changing to type 1 will do what you want I think. This is configured in the variable configuration - "GraphType".

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 01:05:41 pm
Chris,
I'm having a new problem, not seen before. I have attached a ZIP of ALL FILES in the dataMine folder... which btw is hidden? When I used WinSCP it does not list it in the tree but if I type it in manually it will enter the folder just fine and see the file fine. Not sure if this is what you intended? The dataMine folder is on the USB and Vera logging is not using USB.

You'll see that dataMine stop showing new data. I did a ZWave heal last night but that did not change anything. My Zwave network has not changed at all and the heal shows the Thermostat with a full 5 stars.

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 01:47:59 pm
Hi Aaron,
The dataMine folder shouldn't be hidden - I don't know why that would be... My understanding of Linux is that the only way to make a directory hidden is to put a "." as the first character. Clearly this isn't the case if you can still access the directory, so maybe it's a WinSCP issue?

I don't know why things would have stopped logging and I really need to see the log files to be able to have any idea. The easiest way to get this is to type the following url -:

<vera IP>/port_3480/data_request?id=lr_dmCtrl&control=debug

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 01:58:54 pm
when I use your URL I get a blank screen with this msg... "Handler failed"

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 02:05:53 pm
Hmmm - not sure if that's indicative of the other problem... Can you get the logs direct from WinSCP then?
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 02:28:06 pm
What logs - I don't know when you want me to look?

FYI - The attached zip is the entire dataMine folder.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 02:31:54 pm
Ok - sorry, I thought you'd sent me logs in the past...

From WinSCP, click on the TERMINAL button (little black button with HOM on it). Type the following in and hit enter.

cat /var/log/cmh/LuaUPnP.log | grep dataMine

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 02:43:01 pm
Ok - sorry, I thought you'd sent me logs in the past...

From WinSCP, click on the TERMINAL button (little black button with HOM on it). Type the following in and hit enter.

cat /var/log/cmh/LuaUPnP.log | grep dataMine

Cheers
Chris

Out of space? It is a 4GB USB. BTW, dataMine is not hidden.

maybe it is not on the USB?

SetMountPoint & mountLocation = /dev/sda1
SetDataDirectory - /dataMine

Code: [Select]
/dataMine$ cat /var/log/cmh/LuaUPnP.log | grep dataMine
50 02/23/13 12:31:11.842 luup_log:117: dataMine: Logging error logging variable 76::urn:upnp-org:serviceId:TemperatureSensor1::CurrentTemperature <0x2c74f680>
50 02/23/13 12:31:11.842 luup_log:117: dataMine: Unable to open file for write /dataMine/4 Temperature - CurrentTemperature [R2251].txt <0x2c74f680>
50 02/23/13 12:31:11.842 luup_log:117: dataMine: Error: '/dataMine/4 Temperature - CurrentTemperature [R2251].txt: No space left on device' <0x2c74f680>
50 02/23/13 12:31:11.846 luup_log:117: dataMine: Logging error logging variable 79::urn:micasaverde-com:serviceId:HumiditySensor1::CurrentLevel <0x2c74f680>
50 02/23/13 12:31:11.846 luup_log:117: dataMine: Unable to open file for write /dataMine/5 Humidity - CurrentLevel [R2251].txt <0x2c74f680>
50 02/23/13 12:31:11.846 luup_log:117: dataMine: Error: '/dataMine/5 Humidity - CurrentLevel [R2251].txt: No space left on device' <0x2c74f680>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 03:03:02 pm
I think this came up a week or two back. There's a lot of messages on the web (ie google search) about Linux reporting no space left when it's clearly wrong.

One option is that the root directory is full. You only have about 120 files, so it should be ok. FAT file systems only allow (I think) 512 files in the root directory. Normally though, FAT-32 is used these days, and that doesn't have such a limitation... If you type "mount" into WinSCP terminal, it will tell you what file system is used. vfat is (I believe) FAT32, and this will be ok - if it just says fat, then this might be an issue...

You could try deleting all the .backup files and see if that helps (the next release has a function to do this automatically).  I'll have a bit of a dig on the net...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 03:07:16 pm
Aaron, can you post a bit more of the logfile (ie earlier) so I can see (hopefully) why the debug link I sent you earlier didn't work. Or run the link again, and the dump the log again...

Thanks
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 03:18:15 pm
debug link still throws failure

the log file I provided was everything it gave me. I reboot at 4am every day (to ensure stability) so maybe that is why the log is small.

I deleted the backup files - maybe you can make a clean-up function in your code? Have it save X days but delete the rest automatically to save space?

mount shows...
/dev/sda1 on /dataMine type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

also...
/dataMine$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 4352      4352         0 100% /rom
tmpfs                    63728      6948     56780  11% /tmp
tmpfs                      512         0       512   0% /dev
/dev/mtdblock7           11264      3180      8084  28% /overlay
overlayfs:/overlay       11264      3180      8084  28% /
/dev/mtdblock8            4352      4352         0 100% /mios
/dev/sda1               524640      1040    523600   0% /dataMine


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 23, 2013, 03:24:33 pm
Logs get rotated occasionally, so it will only go back a short time. If you run the debug link, then use WinSCP to get the log, that ought to show something (although grep may be case sensitive, so you may also need to try a capital D in dataMine).

Yes- the next version of dataMine (hopefully available in a day or two) has a function that deletes old backups, and just keeps the last 10 files.

Given you seem to have FAT32 (vfat), I don't think that deleting the files will help in this case...  :(

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 23, 2013, 03:28:00 pm
/dataMine$ cat /var/log/cmh/LuaUPnP.log | grep DataMine

Code: [Select]
09 02/23/13 12:49:07.037 JobHandler_LuaUPnP::Run device 117 dataMine graphing and logging room 15 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xcedc60 upnp: 0 <0x2ad49000>
06 02/23/13 12:49:09.565 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 9 now: 0 #hooks: 0 upnp: 0 v:0xe194d0/NONE duplicate:0 <0x2b67d680>
06 02/23/13 12:49:09.566 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 6 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b67d680>
06 02/23/13 12:49:09.592 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: /dev/sda1 now: /dev/sda1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b67d680>
06 02/23/13 12:49:09.592 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: vfat now: vfat #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b67d680>
06 02/23/13 12:49:09.882 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 9 #hooks: 0 upnp: 0 v:0xe194d0/NONE duplicate:0 <0x2b67d680>
06 02/23/13 12:49:09.883 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 6 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b67d680>
06 02/23/13 12:49:09.911 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 9 now: 9 #hooks: 0 upnp: 0 v:0xe194d0/NONE duplicate:1 <0x2b67d680>
06 02/23/13 12:49:09.912 Device_Variable::m_szValue_set device: 117 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 6 now: 6 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b67d680>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 24, 2013, 04:57:03 am
Thanks Aaron,
unfortunately this doesn't show anything useful...

Did deleting the files help? I suspect not, but there's always hope...  If that doesn't work, then I'd suggest copying the files onto your PC, reformatting the stick and copying the files back again. I don't know why this should be necessary, but it should solve the issue...

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 24, 2013, 10:42:26 am
At 1:19pm yesterday, exactly 1 hour after I deleted the backup files, the data started logging/graphing in dataMine. I made no other changes to the Vera's config, did not reboot, nada.

So there is something going on.

I'll wait until your next version releases and then if it happens again I'll redo the USB setup.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 24, 2013, 12:46:51 pm
Interesting. I'm not completely convinced it's the directory entries for 2 reasons...

However, let's see how things go. I am going to restructure the "database" at some stage to put files for each logged variable into a different directory to avoid these sorts of issues. This will take a bit of thought  - ok, not much thought, but the issue is "seemlessly" moving everyones data around....

The new version will help a bit in that it will keep the number of backup files down - these can really build up, so can be an issue, but if that really is your problem, given you only had something like 125 files in the directory (from memory) the problem will likely reoccur in the near future as DM produces 1 file per week per variable...

Let me know how it goes...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on February 24, 2013, 12:59:18 pm
I'd suggest, at least at first, creating only a few folders to separate out things, like...

\dataMine  (current config lives here)
\dataMine\channels
\dataMine\notifications
\dataMine\history  (what is this for?)
\dataMine\configbak

Title: Re: New dataMine graphing and logging plugin
Post by: Chris on February 24, 2013, 01:17:30 pm
Unfortunately, I only want to do this once as there's potential for hassle when moving all the data around... My current thought is to simply have a separate folder for each channel - there may then be a subfolder under that for history data...

History isn't released yet - it's something I've been playing with for a while, and I think it's kinda working now. It basically generates stats for the channels - things like min/max/total etc over different periods - hourly, daily, weekly... I've not looked at it for a couple of weeks as I've been focusing on other things, but it's something to roll out in the future...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: skruffy on March 05, 2013, 03:28:24 pm
Is there any possible way to track temperature from my thermostat down to a decimal place?  I know Vera works with the temperature as an integer, but is it possible for dataMine to report it down to one decimal place?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 05, 2013, 04:30:02 pm
Is there any possible way to track temperature from my thermostat down to a decimal place?  I know Vera works with the temperature as an integer, but is it possible for dataMine to report it down to one decimal place?
dataMine will only log the data that Vera shows. So, if your thermostat/thermometer logs data with higher resolution, then dataMine will show it. Vera itself doesn't stop this, but Veras z-wave implementation does! So, I could recommend other thermometers (not thermostat) that can monitor temperature, and log it with dataMine, with less than 1 deg resolution...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: skruffy on March 06, 2013, 05:13:45 pm
Any recommendations you have for more accurate temperature readings would be welcome.  Something that works indoors and out would be nice.

Also, is there a way to get different variables to graph with the same scale?  In the attached image, I'm graphing temperature, but the two variables don't line up.  Sometimes it lines up, other times not.  Some of the documentation I've been reading seems out of date so I'm not sure what I'm missing.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 06, 2013, 05:32:41 pm
Any recommendations you have for more accurate temperature readings would be welcome.  Something that works indoors and out would be nice.
Personally I'm using a number of 1-wire sensors and the EDS OW-SERVER. http://www.embeddeddatasystems.com/OW-SERVER-1-Wire-to-Ethernet-Server-Revision-2_p_152.html. There's a vera plugin available, and these report at 0.1degC. The server is about $100, and the temperature sensors are a couple of $?? each. The RFXCom with various temperature sensors are also probably ok, but I can't vouch for them as I've not got this...

Also, is there a way to get different variables to graph with the same scale?  In the attached image, I'm graphing temperature, but the two variables don't line up.  Sometimes it lines up, other times not.  Some of the documentation I've been reading seems out of date so I'm not sure what I'm missing.
If you save the graph, then the latest version allows you to specify which axis each channel is on, so you can be sure everything ends up on the same axis (or not if that's what you wanted). The manual is reasonably up to date - there's a couple of pictures that I haven't updated for a while, but the main bits like this are all up to date...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on March 10, 2013, 12:10:41 pm
Chris,
dataMine too a crap again. I attached the folder. all channels gone!

I noticed 2 things right away...

tons of Notifications.txt on 3/6, then not a single one until early morning 3/10 - and only 1 on 3/10
dataMineConfig.json.backup went from 3k to 0k on 3/6 and never changed
dataMineConfig.json is 156b ... I suspect it created a new one when I logged in today.

I have done absolutely nothing to dataMine except look at the graph once or twice over the past week or so.


I ran  cat /var/log/cmh/LuaUPnP.log | grep DataMine ... see _dataMine_log.txt file for output.


Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 10, 2013, 12:25:04 pm
Hi Aaron,
There's nothing obviously wrong - the log looks fine, but for some reason dataMine seems to have reset the config file. The backups look fine, and your data appears ok, so the best thing to do is to rename the last (most recent) backup file over the dataMineConfig.json.

I don't know why it should have done this - the log hasn't captured the event so there's no way to know. Maybe I should implement an auto recovery to backup in this event rather than reinitialising the configuration...

I wonder if you have a lot of Vera restarts? Something must be different about your setup as I assume most people don't have this problem (he sais, waiting for the deluge of emails to the contrary  ;)). Something that could cause this is if Vera resets and files are open on the USB drive, it could corrupt them. This is especially "likely" with the FAT file system - using a Linux journalled file system should significantly reduce the probability of this. That said, I'm using a FAT formatted USB, and I don't have such issues - my Vera resets about once a week at the moment (and I've not upgraded to the latest Beta on my Vera 3 for this reason!).

Anyway, renaming your backup file should get you going again pretty much where you left off...

Cheers
Chris


Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on March 10, 2013, 01:05:40 pm
I have the Vera reboot at 6am every morning... if I don't I find there are issues.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 10, 2013, 01:14:15 pm
I have the Vera reboot at 6am every morning... if I don't I find there are issues.
Fair enough. I'd be surprised if this is causing dataMine issues, but at the end of the day, it's a numbers game - the more often it reboots the more likely it is that there will be issues with files being left open... I'll look at implementing an "auto resort to backup" function in the next version (not the one that will be released in the next day or so though).

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on March 10, 2013, 01:25:26 pm
sounds good, thx for helping on this one.
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on March 10, 2013, 02:48:58 pm
sounds good, thx for helping on this one.

Aaron, out of curiosity... are you running the "ergy" plugin? This caused my Vera to hang almost every day... now with that uninstalled it's very stable...
Title: Re: New dataMine graphing and logging plugin
Post by: Xanlle on March 15, 2013, 06:17:22 am
The dataMine plugin looks great. I installed it and plugged in a USB key. Specified the SetMountUUID property. The problem I have is that there are no channels. Device 'dataMine graphing and logging' shows -> Channels 0.
Any idea why?

http://192.168.1.100:3480/data_request?id=lr_dmCtrl&control=debug

-1-===========================================================================
-2-===========================================================================
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro)
none on /dev type devfs (rw)
none on /proc type proc (rw)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw)
/dev/mtdblock/4 on /overlay type jffs2 (rw)
mini_fo:/overlay on / type mini_fo (rw)
/dev/loop/0 on /mios type squashfs (ro)
none on /proc/bus/usb type usbfs (rw)
/dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine type ext3 (rw)
/dev/scsi/host0/bus0/target0/lun0/part1 on / type ext3 (rw)
-4-===========================================================================
-5-===========================================================================
SetDataDirectory-/dataMine
SetMountUUID    -6d314264-8a0a-48da-84a9-b6067aa2c6a3
SetMountPoint   -
SetManualMount  -0
-6-===========================================================================
{"Events":{"count":0,"last":0},"Version":"0.973","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 15, 2013, 08:32:19 am
Looking at your debug, there's a lot of issues that aren't clear... There is nothing in the log (this is the first section), there's nothing coming back from the report of USB sticks available (this is in the second block), and there's nothing coming back from the fdisk report (in the 4th section)....

On the positive side, the mount data does show that your USB is mounted - are you doing this manually? I also see the same USB stick mounted in two places - this is definitely bad news and needs to be looked at. If you haven't done anything manually, then I'd restart your Vera to remove this and see what comes up.

Code: [Select]
/dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine type ext3 (rw)
/dev/scsi/host0/bus0/target0/lun0/part1 on / type ext3 (rw)

The fact that there's nothing logged from dataMine itself is a bit strange - clearly it's running or it wouldn't be returning this debug report, but I would expect to see a lot of log entries when dataMine starts. Possibly the logs have been rotated between the time dataMine started and you getting the debug report, so I'd suggest that you click the reload button, then once Luup has restarted (say 1 minute later), click on the debug link and see what is there.

Hopefully this will provide more information that I can comment on.

Regarding your actual question of Channels=0, this is normal if you've not enabled any channels. To enable channels, you need to go to the Configuration tab, select the variables, select Logging to true and hit save....

Let me know how you get on...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Xanlle on March 15, 2013, 10:14:35 am
Did a restart, removed the MountUUID and specified it again on the datamine graphing page. Enabled logging on some variables. Now it's working great.

Thanks for helping.
Title: Re: New dataMine graphing and logging plugin
Post by: darrenp on March 23, 2013, 03:18:13 pm
Hi,

I've been trying to get the plugin working but not having much luck.

I've installed everything as per the wiki page, but when opening the device and clicking on the Graph tab, I just get stuck on "Opening showHistory..." and a constant barber pole. Similar, just with "Opening showConfig..." instead.

The USB disk is definitely mounted ok, and from the log file it looks like it starts ok, just can't find a config file but as this is a first run that doesn't surprise me.

Code: [Select]
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 4352      4352         0 100% /rom
tmpfs                    31240       392     30848   1% /tmp
tmpfs                      512         0       512   0% /dev
/dev/mtdblock7           11264      2400      8864  21% /overlay
overlayfs:/overlay       11264      2400      8864  21% /
/dev/mtdblock8            4352      4352         0 100% /mios
/dev/sdb1              7653196       176   7264252   0% /tmp/log/cmh
/dev/sda1               519604        20    493188   0% /dataMine

Code: [Select]
root@MiOS_XXXXXXXX:/tmp/log/cmh# grep -i mine LuaUPnP.log
09 03/23/13 18:25:37.474 JobHandler_LuaUPnP::Run device 55 dataMine room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xce2cf8 upnp: 0 <0x2b7b8000>
50 03/23/13 18:25:44.549 luup_log:55: dataMine: Initialising Data Mining System __LEAK__ this:139264 start:692224 to 0x1060000 <0x2c0eb680>
50 03/23/13 18:25:44.550 luup_log:55: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x2c0eb680>
06 03/23/13 18:25:44.550 Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xdf3008/NONE duplicate:1 <0x2c0eb680>
06 03/23/13 18:25:44.551 Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2c0eb680>
50 03/23/13 18:25:44.551 luup_log:55: dataMine: Reinitialising configuration structure <0x2c0eb680>
50 03/23/13 18:25:44.552 luup_log:55: dataMine: Startup complete <0x2c0eb680>

Any pointers?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 23, 2013, 03:32:10 pm
I've installed everything as per the wiki page
Hi,
With UI-5, you should just install the plugin through the UI or apps.mios.com - the version on code.mios.com (which I assume is the one you mean) is for UI4 (from memory it wont work on UI5 as the graphing is very different.

If you've not used the version on apps.mios.com, I'd suggest removing the app, and installing from apps.mios.com so that we're starting from a known point.

If you have used this version, then can you click on the debug link in the dataMine app page (under the link to the GUI) and post the result.

Let me know how you go...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: darrenp on March 23, 2013, 06:24:11 pm
Ah, that did it. Thanks!

Too used to the market stuff being the old software. :)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on March 23, 2013, 06:39:19 pm
Perfect - I should really make a note on the code.mios.com site to make this clearer...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: oberjon on April 01, 2013, 04:55:23 pm
I have been trying unsuccessfully for some time to install the dataMine plugin.  I am using a VeraLite with firmware 1.5.622.  After installing, I set the SetMountPoint variable to /dev/sda1.  When I reload Luup, the status area shows dataMine graphing and logging[10] : Mounting dataMine storage device (/dev/sda1).  It is actually mounted, but I am unable to enter the dataMine GUI, and the message remains indefinitely.  If I reload again, I am given the message dataMine graphing and logging[10] : Startup Lua Failed.  If I manually unmount the directory and reload again, it goes back to the original message.

If I clear the SetMountPoint variable and manually unmount, I can enter the dataMine GUI where I am prompted for the USB UUID.  I select the USB drive and reload Luup.  This populates the SetMountUUID variable and mounts the drive, but again hangs with the dataMine graphing and logging[10] : Mounting dataMine storage device (/dev/sda1) message.  At this point, I am again not able to load the dataMine GUI without manually unmounting /dataMine.

I am never able to get past the USB selection dialog in the dataMine GUI.

Any advise on how to remedy this issue?  I've uninstalled, deleted and reloaded everything with no success.

Thanks in advance.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 01, 2013, 05:18:44 pm
Hi,
This is likely to be the same problem as a couple of others have had. There was a bug introduced in the last version - I'll get an update out later in the week, but there is a fix if you want to try it - see here -: http://forum.micasaverde.com/index.php/topic,13517.0.html (http://forum.micasaverde.com/index.php/topic,13517.0.html)

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: oberjon on April 01, 2013, 06:12:38 pm
Chris,

Thanks for the help.  I uploaded the updated L_DataMine1.lua file which took care of the messages I was receiving.  Now when I try to load the dataMine GUI, I get stuck at the screen shown in the attached image.  Any thoughts?

Thanks,
Jon
Title: Re: New dataMine graphing and logging plugin
Post by: haavard on April 09, 2013, 07:40:53 am
Hello,
I have an annoying problem with datamine I hope someone can help me with.
I had some temperature sensors that I logged​​. Then it turned out that I had mixed some of the sensors. Therefore I had to change the name of these in MCV. Now it seams like Datamine has not updated with the new names I have set i MCV. There are now different names in the configuration and the place were you choose channels.

hope someone can point me in the right direction.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 09, 2013, 09:28:29 am
Now it seams like Datamine has not updated with the new names I have set i MCV. There are now different names in the configuration and the place were you choose channels.
I'm not sure that I completely understand what you mean, but dataMine does not update its names when you update a device. This is because there are normally many variables associated with a device, so it can't just change the name or this would upset things where you are logging multiple variables on a device (which is quite normal). If you want to change the display name in dataMine, then simply go to the dataMine configuration, select the variable that you want to change, change "Display Name" and hit save.

I hope this is what you were after?

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 09, 2013, 09:35:27 am
Thanks for the help.  I uploaded the updated L_DataMine1.lua file which took care of the messages I was receiving.  Now when I try to load the dataMine GUI, I get stuck at the screen shown in the attached image.  Any thoughts?
Hi Jon,
Sorry for the slow reply - I've been away from home for work...
I'll get an update out in the next day or so with a fix for this problem - I think that's the easiest way forward...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: haavard on April 11, 2013, 04:09:38 am
Now it seams like Datamine has not updated with the new names I have set i MCV. There are now different names in the configuration and the place were you choose channels.
I'm not sure that I completely understand what you mean, but dataMine does not update its names when you update a device. This is because there are normally many variables associated with a device, so it can't just change the name or this would upset things where you are logging multiple variables on a device (which is quite normal). If you want to change the display name in dataMine, then simply go to the dataMine configuration, select the variable that you want to change, change "Display Name" and hit save.

I hope this is what you were after?

Chris

I am not sure what happened, but I tried to exclude and include the sensor, and that fixed the problem. Thanks anyway! 
Title: Re: New dataMine graphing and logging plugin
Post by: Pshomectrl on April 15, 2013, 10:17:34 am
Hi all
now, also with newest datamineverion...........0.975
Still same error as i had several days before.
dataMine graphing and logging[21] : Startup Lua Failed.
What can i do to have datamine running?
Thanks for help,
Paul
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 15, 2013, 11:54:33 am
Sorry for the hassle - are you able to get the log file?  If you have WinSCP, then first restart Luup to make sure we view the log when there's an error, then in the terminal box in WinSCP type "cat /var/log/cmh/LuaUPnP.log | grep -i DataMine". Hopefully this would tell me where it's failing.

Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Pshomectrl on April 15, 2013, 12:58:37 pm
Hi Chris
I'm not firm with linux comands. Sorry.
But, i saved the Datamine Debug page just before i tried to reload the UI5.
I will attach a "Datamine Debug Page.txt" file. May be it helps?
The entries i can see in the "LuaUPnP.log" are realy newer than the ones from Datamine Debug.
Hope this file also helps you?
Otherwise, please explain how to get the info you need.
Thanks a lot for your work
Paul
Title: Re: New dataMine graphing and logging plugin
Post by: Pshomectrl on April 15, 2013, 01:12:03 pm
Hi Chris.
May be, this is what you asked for:

/var/log/cmh$ cat /var/log/cmh/LuaUPnP.log | grep -i DataMine
02   04/15/13 18:59:51.940   JobHandler_LuaUPnP::FetchMissingFiles need to download D_DataMine1.json <0x2ad2c000>
02   04/15/13 18:59:51.940   JobHandler_LuaUPnP::m_sMissingFile_set D_DataMine1.json <0x2ad2c000>
01   04/15/13 18:59:52.014   Device_LuaUPnP::LoadDeviceDoc can't load /etc/cmh-lu//D_DataMine1.xml <0x2ad2c000>
02   04/15/13 18:59:52.029   JobHandler_LuaUPnP::m_sMissingFile_set /etc/cmh-lu//D_DataMine1.xml <0x2ad2c000>
01   04/15/13 18:59:52.030   JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 21/D_DataMine1.xml so device 21 is offline <0x2ad2c000>
09   04/15/13 18:59:59.483   JobHandler_LuaUPnP::Run device 21  room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x76aba0 upnp: 0 <0x2ad2c000>
03   04/15/13 18:59:59.500   JobHandler_LuaUPnP::Run ready 13 devices installation 35102692 missing /etc/cmh-lu//D_DataMine1.xml version *1.5.408*  allow_evt: 1 allow_ud: 1 upnp 0 netpnp: 1 reprovision: 0 changed: 0 pMem 0x817000 start 7389184 diff: 1093632 <0x2ad2c000>
09   04/15/13 19:00:18.692   JobHandler_LuaUPnP::Run device 21 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x9a0ab0 upnp: 0 <0x2b93d000>
50   04/15/13 19:00:21.472   luup_log:21: dataMine: Initialising dataMine System (0.975) <0x2c271680>
06   04/15/13 19:00:21.473   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetDataDirectory was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.473   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.474   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountPoint was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.475   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetManualMount was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.475   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetTimeoutPeriod was: EMPTY now: 12 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.476   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetHistoryEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.476   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetUseBackup was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.477   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetEventsEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.478   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0x9daf40/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.478   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
50   04/15/13 19:00:21.479   luup_log:21: dataMine: Installing update of dataMine web application <0x2c271680>
50   04/15/13 19:00:25.829   luup_log:21: dataMine: ERROR: Startup state is 'uninitialised'! <0x2c271680>
06   04/15/13 19:00:25.830   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x9daf40/NONE duplicate:1 <0x2c271680>
06   04/15/13 19:00:25.830   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2c271680>
50   04/15/13 19:00:25.830   luup_log:21: dataMine: Reinitialising configuration structure <0x2c271680>
50   04/15/13 19:00:25.831   luup_log:21: dataMine: Startup complete <0x2c271680>
01   04/15/13 19:00:41.074   LuaInterface::CallFunction_Timer-5 function deferredStartup failed [string "-- dataMine Plugin for Vera..."]:725: bad argument #1 to 'sort' (table expected, got nil) <0x2cc71680>
06   04/15/13 19:01:06.105   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was:  now: EC13-1B32 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2e471680>
09   04/15/13 19:01:30.726   JobHandler_LuaUPnP::Run device 21 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xe7a9f0 upnp: 0 <0x2ba0a000>
50   04/15/13 19:01:33.111   luup_log:21: dataMine: Initialising dataMine System (0.975) <0x2c33d680>
06   04/15/13 19:01:33.112   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xeb4ec0/NONE duplicate:1 <0x2c33d680>
06   04/15/13 19:01:33.112   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2c33d680>
50   04/15/13 19:01:33.112   luup_log:21: dataMine: Mounting to UUID 'EC13-1B32' <0x2c33d680>
50   04/15/13 19:01:33.528   luup_log:21: dataMine: UUID = /dev/sda1: LABEL="VERA_PLS" UUID="EC13-1B32" <0x2c33d680>
50   04/15/13 19:01:33.543   luup_log:21: dataMine: Found UUID 'EC13-1B32' at '/dev/sda1' <0x2c33d680>
50   04/15/13 19:01:33.565   luup_log:21: dataMine: Mounted: /dev/sda1 on /dataMine type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) <0x2c33d680>
06   04/15/13 19:01:33.566   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: EMPTY now: /dev/sda1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c33d680>
06   04/15/13 19:01:33.566   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: EMPTY now: vfat #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c33d680>
50   04/15/13 19:01:33.592   luup_log:21: dataMine: ERROR: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x2c33d680>
01   04/15/13 19:01:33.608   LuaInterface::CallFunction_Startup-1 device 21 function initialise failed [string "-- dataMine Plugin for Vera..."]:725: bad argument #1 to 'sort' (table expected, got nil) <0x2c33d680>
01   04/15/13 19:01:33.609   LuImplementation::StartLua running startup code for 21 I_DataMine1.xml failed <0x2c33d680>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 15, 2013, 01:43:38 pm
That's the one - thanks.

Unfortunately I can't test this since I'm not at home. I've attached an updated Lua file. To load this from UI5, go to App, Develop Apps, Luup Files. Select the file, click "Restart Luup after upload" and hit Go.

Hopefully this will resolve the issue, but if not another log dump would be useful...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Pshomectrl on April 15, 2013, 01:56:06 pm
Hi Chris
You are the man............
Thanks, seems this solved my problem.
And.......Chris what ever you do just this time, i hope, it makes fun. Enjoy.
Again, thanks for all your work.
Regards, Paul
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 15, 2013, 05:16:08 pm
Great - thanks for the feedback. I'll commit this to apps...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: ChrisAZ on April 15, 2013, 11:14:20 pm
I started having the same problems this weekend. Tried the patch you posted and it's all working again. Thanks for the support.
Title: Re: New dataMine graphing and logging plugin
Post by: vascodp on April 17, 2013, 06:18:30 pm

Also - is there a way to combine variables into a single graph plot?  Say for example I want to combine several lights in one room into a single plot - is that possible?  Alternatively, is it possible to plot all energy use over time in the "chart" tab (e.g. the total wattage used over time in a single variable to show total energy usage)?
This isn't something that can currently be done. I did think about making the donut graph selectable between categories and rooms, but I haven't done this. I'm happy to think about this once I've finished getting the history stuff integrated reliably (it's not so simple having to account for random Vera/Lua restarts when you're post processing data). The history may support some of what you're after in that it will derive usage stats on an hour/daily basis (and maybe month), and I then hope to support combining these in different ways (ie by category, room...). But this isn't currently planned for the raw data...

Cheers
Chris

@Chris, I'm starting to explore DataMine (great job !) and I'm looking for a way to monitor whole house and segmented (per device / device type, room level...) energy consumption over a period of time (eg: last day / week / month). Are you planning for such features ? Any clue on how to implement them at the meanwhile ?

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on April 18, 2013, 04:46:54 am
Chris,

since 3 or 4 days my vera became unstable and non-resonsive. I expect this to be because of datamine. i have to reload now 2 or 3 times a day else nothing works anymore. It got better since last evening when 0.977 was auto-updated.

Are more people experiencing this? Is this something you can explain?

2nd quesiton. There is now a section:
Disk Space Total (kb)
Disk Space Used (kb)
Disk Space Free (kb)
but it shows no values...

Cloudl it be my 256Mb stick is full? Or should that not stop vera to work. After reset (power physicaly) it works again for a short amount of time (hours).
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 18, 2013, 08:30:44 am
@SanderL,
The disk space etc should show numbers. They are showing on my system, although I can't be 100% sure that they are updating at the moment (they should be!). If they aren't showing, then I wonder if your system is updated to the current Lua version (??).

I would doubt that your memory stick is full, and in any case this shouldn't have the effect your seeing. There's been very little change with dataMine in the past 3 or 4 weeks, so I would be surprised if your problems are related to DM software, but if you can send me the logs then I'm happy to take a look and see if there's anything showing in there...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 18, 2013, 08:39:02 am
@Chris, I'm starting to explore DataMine (great job !) and I'm looking for a way to monitor whole house and segmented (per device / device type, room level...) energy consumption over a period of time (eg: last day / week / month). Are you planning for such features ? Any clue on how to implement them at the meanwhile ?

So, what I'm planning is usage per hour/day/week and month (from memory). This is largely implemented, but I've not spent a lot of time to test it out. When implemented, it will post process your current data, so it will work on historical data.  This will be implemented in the Lua on your Vera. In addition, the idea is for the GUI to then process this into the same energy categories as we've currently got.

That's the plan ;) Most of the data side is done, but I need to test it, and I'm currently away from home for work quite a bit  :( so I don't want to commit to a timescale right now. The other thing I also wanted to do at the same time is to migrate to a better file structure, since this generates a bunch more files....

Unfortunately I don't have any ideas on how to do this now - sorry.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: jouked on April 18, 2013, 08:51:50 am
What you're trying to do looks a lot like rddtool, which has the logic in it to automatically consolidate historical data. This reduces disk usage dramatically while also speeding up analysis work.

Check out http://oss.oetiker.ch/rrdtool/doc/rrdtool.en.html for more information on this.
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 18, 2013, 09:55:19 am
Hi,
Yes, I'm aware of RRD, but this can't really be integrated into dataMine. Also,  while this does ultimately achieve a similar result, there is a difference - with RRD, it only keeps the regular data - so you loose all the changes in between. With dataMine, you get all the real-time data, and also the hourly/daily etc stats, so hopefully the best of both worlds.

This is already coded up, so it just needs to be tested...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on April 19, 2013, 06:23:57 am
@SanderL,
The disk space etc should show numbers. They are showing on my system, although I can't be 100% sure that they are updating at the moment (they should be!). If they aren't showing, then I wonder if your system is updated to the current Lua version (??).

I would doubt that your memory stick is full, and in any case this shouldn't have the effect your seeing. There's been very little change with dataMine in the past 3 or 4 weeks, so I would be surprised if your problems are related to DM software, but if you can send me the logs then I'm happy to take a look and see if there's anything showing in there...

Cheers
Chris

chris,

sent you a PM with logs.

Yes am running latest LUA (622)
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on April 19, 2013, 03:28:29 pm
chris,

sent you a PM with logs.

Yes am running latest LUA (622)
Got it - thanks.

Question - do you have a Vera 2? This may explain why the diskspace doesn't work. I know that the fdisk command is not available on Vera 2, and this is not showing up in your logs (it should be between the -4- and -5-). It may be that the df command (which is used to get the disk space) may also not be available. I'll check this when I get a chance over the weekend.

Otherwise I don't see anything strange in your logs. As I said in my previous post, there's not been a lot of changes to dataMine in the past few weeks, so while I wouldn't rule anything out, I'd be surprised if there was a link here...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on April 19, 2013, 04:10:12 pm
chris,

sent you a PM with logs.

Yes am running latest LUA (622)
Got it - thanks.

Question - do you have a Vera 2? This may explain why the diskspace doesn't work. I know that the fdisk command is not available on Vera 2, and this is not showing up in your logs (it should be between the -4- and -5-). It may be that the df command (which is used to get the disk space) may also not be available. I'll check this when I get a chance over the weekend.

Otherwise I don't see anything strange in your logs. As I said in my previous post, there's not been a lot of changes to dataMine in the past few weeks, so while I wouldn't rule anything out, I'd be surprised if there was a link here...

Cheers
Chris

YES vera2
Title: Re: New dataMine graphing and logging plugin
Post by: vascodp on April 21, 2013, 05:48:13 pm
@Chris, I'm starting to explore DataMine (great job !) and I'm looking for a way to monitor whole house and segmented (per device / device type, room level...) energy consumption over a period of time (eg: last day / week / month). Are you planning for such features ? Any clue on how to implement them at the meanwhile ?

So, what I'm planning is usage per hour/day/week and month (from memory). This is largely implemented, but I've not spent a lot of time to test it out. When implemented, it will post process your current data, so it will work on historical data.  This will be implemented in the Lua on your Vera. In addition, the idea is for the GUI to then process this into the same energy categories as we've currently got.

That's the plan ;) Most of the data side is done, but I need to test it, and I'm currently away from home for work quite a bit  :( so I don't want to commit to a timescale right now. The other thing I also wanted to do at the same time is to migrate to a better file structure, since this generates a bunch more files....

Unfortunately I don't have any ideas on how to do this now - sorry.

Cheers
Chris

Chris, that sounds great ! It would be quite a powerful tool to help reducing energy consumption and to demonstrate the savings. Crossing my fingers  ;)
Title: Re: New dataMine graphing and logging plugin
Post by: oTi@ on April 25, 2013, 07:15:43 am
@Chris,

Only recently got to noodle with your plug-in. (My Dad had a need for a data logger, so I set him up with a Vera 2 and your app. He loves it.)

Excellent stuff!
Title: Re: New dataMine graphing and logging plugin
Post by: testeur765 on May 04, 2013, 06:01:14 pm
Hello,

I'm using you plugin since several weeks and it is very great. Would it be possible to add some features in the energy dashboard ? We can see for now the current energy but would it be possible to see automatically the total energy for 1 day or for 1 month in Kwh (which is in fact just an addition of every stored data) ?

Thanks,

Regards,
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on May 04, 2013, 06:13:44 pm
Hi,
I do plan to add extra functions along this line for a long...

The issue at the moment is retrieving historical data over a "long" period of time. This is especially an issue with energy data which updates very often (maybe every few seconds) - the quantity of data is very large, and this slows Vera down.

Hopefully, over the next few weeks I'll update the database issue, and will then be incorporating more historical functions (most of this is actually written). Then I'll take a look at revising the energy stats.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Mic on May 29, 2013, 01:08:59 pm
Hi chris,
               Thanks for the exellent plugin, maybe it has already ascked, i checked on your documentation and didn't find. I monitor most of my temperature sensor, is there a way to fix the max and min graph, so i will like to define themax and min in the left of the graph.

Thanks
Title: Re: New dataMine graphing and logging plugin
Post by: Mic on May 29, 2013, 01:17:17 pm
Also is there a way to have a direct url to a saved graph. Thank again
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on May 29, 2013, 01:30:24 pm
               Thanks for the exellent plugin, maybe it has already ascked, i checked on your documentation and didn't find. I monitor most of my temperature sensor, is there a way to fix the max and min graph, so i will like to define themax and min in the left of the graph.
No - this is something that's on my list, but it's not currently implemented.


Also is there a way to have a direct url to a saved graph. Thank again
Yes - this should be in the doc. When you save a graph, you can give it a "quickview reference". Then you can call it with the /dm/graph.html?ref=<reference>. There are other options and these should be in the document. This is a new feature, so I welcome feedback...

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: Mic on May 29, 2013, 01:31:32 pm
Thanks will try it.
Title: Re: New dataMine graphing and logging plugin
Post by: Mic on May 29, 2013, 01:43:56 pm
It work, thanks, now i have my graph directly to my sqremote tab.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on June 20, 2013, 11:19:37 pm
Hi Chris,
It has been a while so I thought I'd check in to see if there were any updates coming soon?

I'm hoping for some 'measurement' features... like being able to have the system calculate the time for on/off cycles, etc. Thoughts?
Title: Re: New dataMine graphing and logging plugin
Post by: A.Ardon on July 08, 2013, 01:35:05 pm
I have the Data Mine App running for a few months.

Until last week I only had a smart meter for the KWH.
After they have replaced the gas pipe and gasmeter I also can read the gasmeter now.


Can I monitor and display my usage in datamine for the gas 'part'??
I can select the urn:brutgrink-com:serviceId:DutchSmartMeter, but no data is displayed.

Title: Re: New dataMine graphing and logging plugin
Post by: gerardosamara on July 10, 2013, 05:38:03 am
+1  see post #366/367 

with same error "https://fwd7.mios.com/dm/index.html"  when I click on "datamine graphic page" of the datamine app in MODULE TAB but datamine debug page access is successful.

UPDATE : OK when logging in Veralite using http://cp.mios.com   which points to IP Verlite 192.168.0.xxx
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on July 10, 2013, 11:43:47 am
Datamine can not be accessed from outside your vera network especially through the mios servers via cp.mios.com. The answer follows the posts numbers you posted.

- Garrett

Title: Re: New dataMine graphing and logging plugin
Post by: mal on July 13, 2013, 04:56:16 am
I feel like a voyeur always looking in here and never saying any thing, have had a vera 2 for about 2 years and just purchased the vera 3.
In that time I have muddled my way through getting apps working and some of the weird quirks of some devices EG 4 in one sensor even setup a vpn server at home but this has me stumped
this is what I did
setup 8 g usb stick on the bottom port and mounted it............. ok (I use the top port for logging)
got your app

found 5 of the 6 files in developers and uploaded them ...ok  couldn't  find   j_Datamine1.js

Changed setmount point to /dev/sda1 it is seeing the drive there is nothing in setmountuuid

Created a new device and changed it's setmount

Got winSCP had a bugger of a time with the password until i remembered I used a backup of my vera 2 on my vera 3 so I used the old password from my vera 2

Put the jason library in /usr/lib/lua

data_mine.png is where it should be

directory www/cmh/plugins is not there ??

not really sure what you mean by

Use WinSCP (or another SSH client) to upload the following files to Vera

Directory: /www/cmh/plugins/dataMine

    All the GIF/PNG/JS files (other than the files listed above)

What gif/png/js files  ...scratches his head.....

I can go to the graphing page it does see all the devices but online status is showing errors of course

Probably just me but am stuck in a loop so any help would be great


50   07/13/13 18:48:18.320   luup_log:197: dataMine: Logging error logging variable 4::urn:micasaverde-com:serviceId:EnergyMetering1::Watts <0x2b799680>
50   07/13/13 18:48:18.320   luup_log:197: dataMine: Unable to open file for write /dataMine/database/1/raw/2271.txt <0x2b799680>
50   07/13/13 18:48:18.320   luup_log:197: dataMine: Error: '/dataMine/database/1/raw/2271.txt: No such file or directory' <0x2b799680>
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 15, 2013, 11:12:46 am
@mal,
The instructions you've followed are the UI4 instructions... Anyway, that's probably not the problem - there is a bug in the current version that doesn't create a directory. I hope this is now fixed and a new version has just been loaded to apps so this should be available in the next day or so. It will hopefully fix your problem  - if not, please send the data from the debug page (debug link in the plugin) and I'll take a look.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: mal on July 16, 2013, 04:39:05 am
Hi Chris
Thanks for the reply i read about the instructions the other night luckily i did a backup the day before i installed datamine so i went back to that and re-installed seems to be working.
Some devices are either taking a long time to update or not at all
in the datamine device it show the 8 g drive i use a 2 g for mios logging

this program is awsome well done

Code: [Select]
now: 7693976 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2dfef680>
06 07/16/13 18:14:05.213 Device_Variable::m_szValue_set device: 196 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsed was: 150968 now: 150976 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2dfef680>
06 07/16/13 18:14:05.232 Device_Variable::m_szValue_set device: 196 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskFree was: 7152176 now: 7152168 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2dfef680>
06 07/16/13 18:14:05.233 Device_Variable::m_szValue_set device: 196 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsedPcnt was: 2 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2dfef680>
-2-===========================================================================
/dev/sdb1: LABEL="MiOS" UUID="0ff13b45-8da8-42a2-b40e-f918509e6bf6"
/dev/sda1: LABEL="MiOS" UUID="9638bd90-f7a8-4701-89ba-2dc5c5d8550e"
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sda1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime,errors=continue,barrier=0,data=ordered)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sdb1 on /dataMine type ext4 (rw,relatime,barrier=1,data=ordered)
-4-===========================================================================

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 2011 MB, 2011168768 bytes
62 heads, 62 sectors/track, 1021 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1   *           1         273      524705+ 83 Linux
/dev/sda2             274        1021     1437656  83 Linux

Disk /dev/sdb: 8004 MB, 8004304896 bytes
35 heads, 21 sectors/track, 21269 cylinders
Units = cylinders of 735 * 512 = 376320 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1       21270     7816688   b Win95 FAT32
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -0ff13b45-8da8-42a2-b40e-f918509e6bf6
SetMountPoint   -
SetManualMount  -0
-6-===========================================================================
{"Version":"0.980","dbVersion":2,"Events":{"count":194,"last":1373957630},"guiConfig":{"_dc":"1373948266935","graphOptionShadows":true},"Variables":[{"Ghost":false,"Service":"urn:micasaverde-com:serviceId:EnergyMetering1","LastRec":1373963511,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"EnergyCat":4,"Type":0,"DataType":5,"Logging":1,"Device":4,"FilterMinimum":0,"FilterEnable":0,"DrowsyWarning":0,"Id":1,"DataOffset":0,"Name":"MAIN TV POWER - Watts","LastVal":97,"FirstRec":0,"Variable":"Watts","historyState":0},{"Ghost":false,"Service":"urn:micasaverde-com:serviceId:EnergyMetering1","LastRec":0,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"EnergyCat":2,"Type":0,"DataType":5,"Logging":1,"Device":5,"FilterMinimum":0,"FilterEnable":0,"DrowsyWarning":0,"Id":2,"DataOffset":0,"Name":"bbq ligt - Watts","LastVal":0,"FirstRec":0,"Variable":"Watts","historyState":0},{"Ghost":false,"Service":"urn:micasaverde-
Title: Re: New dataMine graphing and logging plugin
Post by: mal on July 16, 2013, 04:59:23 am
Spoke too soon will get back to you in a few days

is there a way to export the csv files?
Title: Re: New dataMine graphing and logging plugin
Post by: Chris on July 16, 2013, 05:50:46 am
Some devices are either taking a long time to update or not at all
I've just found a bug where the logging doesn't get enabled until dataMine restarts. So, I'd advise restarting dataMine after enabling logging for new variables.

There is no way to export the CSV files from dataMine (at the moment anyway). You can always use WinSCP to grab them if you want though.

Cheers
Chris
Title: Re: New dataMine graphing and logging plugin
Post by: DrZWave on July 31, 2013, 08:25:38 am
Chris,
I rebuilt parts of my zwave network and deleted nearly all of my datamine logging.
Now I'm ready to put them back in but datamine thinks the USB stick is full.

Channels
2

Disk Space Total (kb) 6464
Disk Space Used (kb) 6464
Disk Space Free (kb) 0 100

Here is the debug page - hope that helps...
-1-=========================================================================== 50 07/30/13 20:10:30.569 luup_log:12: dataMine: Next Sun EVENT is 1.37526e+09 <0x1c08> 50 07/31/13 5:39:30.580 luup_log:12: dataMine: Next Sun EVENT is 1.37532e+09 <0x1c08> 50 07/31/13 7:54:35.139 luup_log:12: dataMine: Initialising dataMine System (0.980) <0x402> 06 07/31/13 7:54:35.140 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:54:35.141 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 0 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:0 <0x402> 06 07/31/13 7:54:35.142 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 2 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:54:35.144 luup_log:12: dataMine: Mounting to UUID '2402-C7CE' <0x402> 50 07/31/13 7:54:38.176 luup_log:12: dataMine: Error reading tmpfile during UUID check <0x402> 50 07/31/13 7:54:38.347 luup_log:12: dataMine: No UUID mount '2402-C7CE' was found! <0x402> 50 07/31/13 7:54:38.462 luup_log:12: dataMine: Error reading tmpfile during mountcheck <0x402> 06 07/31/13 7:54:38.463 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:54:38.464 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 50 07/31/13 7:54:39.116 luup_log:12: dataMine: Watching: D[8] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402> 06 07/31/13 7:54:39.118 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 2 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:0 <0x402> 06 07/31/13 7:54:39.119 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 06 07/31/13 7:54:39.259 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 2 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:1 <0x402> 06 07/31/13 7:54:39.280 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 1 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:54:39.283 luup_log:12: dataMine: Startup complete <0x402> 50 07/31/13 7:55:01.019 luup_log:12: dataMine: Initialising dataMine System (0.980) <0x402> 06 07/31/13 7:55:01.020 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:55:01.022 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 0 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:0 <0x402> 06 07/31/13 7:55:01.023 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 2 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:55:01.024 luup_log:12: dataMine: Mounting to UUID '2402-C7CE' <0x402> 50 07/31/13 7:55:03.171 luup_log:12: dataMine: Error reading tmpfile during UUID check <0x402> 50 07/31/13 7:55:03.263 luup_log:12: dataMine: No UUID mount '2402-C7CE' was found! <0x402> 50 07/31/13 7:55:03.471 luup_log:12: dataMine: Error reading tmpfile during mountcheck <0x402> 06 07/31/13 7:55:03.472 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:55:03.472 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 09 07/31/13 7:55:03.641 JobHandler_LuaUPnP::Run device 12 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id parent 0/0x7a63b0 upnp: 0 <0x400> 50 07/31/13 7:55:04.242 luup_log:12: dataMine: Watching: D[8] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402> 06 07/31/13 7:55:04.413 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 2 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:0 <0x402> 06 07/31/13 7:55:04.419 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 06 07/31/13 7:55:04.858 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 2 #hooks: 0 upnp: 0 v:0x8278c8/NONE duplicate:1 <0x402> 06 07/31/13 7:55:04.859 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 1 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:55:04.921 luup_log:12: dataMine: Startup complete <0x402> 09 07/31/13 7:55:30.625 JobHandler_LuaUPnP::Run device 12 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id parent 0/0x7a6388 upnp: 0 <0x400> 50 07/31/13 7:55:33.094 luup_log:12: dataMine: Initialising dataMine System (0.980) <0x402> 06 07/31/13 7:55:33.095 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:55:33.096 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 0 #hooks: 0 upnp: 0 v:0x8278d0/NONE duplicate:0 <0x402> 06 07/31/13 7:55:33.098 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 2 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:55:33.099 luup_log:12: dataMine: Mounting to UUID '2402-C7CE' <0x402> 50 07/31/13 7:55:34.791 luup_log:12: dataMine: Error reading tmpfile during UUID check <0x402> 50 07/31/13 7:55:34.836 luup_log:12: dataMine: No UUID mount '2402-C7CE' was found! <0x402> 50 07/31/13 7:55:34.904 luup_log:12: dataMine: Error reading tmpfile during mountcheck <0x402> 06 07/31/13 7:55:34.905 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 06 07/31/13 7:55:34.906 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: now: #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402> 50 07/31/13 7:55:35.214 luup_log:12: dataMine: Watching: D[8] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402> 06 07/31/13 7:55:35.215 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 2 #hooks: 0 upnp: 0 v:0x8278d0/NONE duplicate:0 <0x402> 06 07/31/13 7:55:35.217 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 06 07/31/13 7:55:35.251 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 2 #hooks: 0 upnp: 0 v:0x8278d0/NONE duplicate:1 <0x402> 06 07/31/13 7:55:35.253 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 1 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402> 50 07/31/13 7:55:35.255 luup_log:12: dataMine: Startup complete <0x402> 50 07/31/13 7:55:50.331 luup_log:12: dataMine: Delete backups complete - deleted 0 of 66 files __LEAK__ this:36864 start:40960 to 0x917000 <0x1c08> 06 07/31/13 7:55:50.381 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskTotal was: 6464 now: 6464 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 __LEAK__ this:4096 start:45056 to 0x918000 <0x1c08> 06 07/31/13 7:55:50.382 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsed was: 6464 now: 6464 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x1c08> 06 07/31/13 7:55:50.383 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskFree was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x1c08> 06 07/31/13 7:55:50.384 Device_Variable::m_szValue_set device: 12 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsedPcnt was: 100 now: 100 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x1c08> -2-=========================================================================== /dev/scsi/host0/bus0/target0/lun0/part1: LABEL="MiOS" UUID="f14f7809-0175-47e6-ad8d-06d121bba91a" /dev/scsi/host1/bus0/target0/lun0/part1: LABEL="MiOS" UUID="b3bdb623-3a04-49d0-95cf-c52f5a2ec45c" -3-=========================================================================== rootfs on / type rootfs (rw) /dev/root on /rom type squashfs (ro) none on /dev type devfs (rw) none on /proc type proc (rw) tmpfs on /tmp type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw) /dev/mtdblock/4 on /overlay type jffs2 (rw) mini_fo:/overlay on / type mini_fo (rw) /dev/scsi/host0/bus0/target0/lun0/part1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime) /dev/loop/0 on /mios type squashfs (ro) none on /proc/bus/usb type usbfs (rw) -4-=========================================================================== -5-=========================================================================== SetDataDirectory-/dataMine/ SetMountUUID -2402-C7CE SetMountPoint - SetManualMount -0 -6-=========================================================================== {"Version":"0.980","dbVersion":2,"Events":{"count":622,"last":1375185005},"LastWrite":1375271735,"Variables":[{"Ghost":true,"Service":"urn:upnp-org:serviceId:SwitchPower1","DrowsyError":0,"LastRec":0,"FilterMaximum":0,"LastHistory":1104516000,"Type":0,"DataType":3,"Logging":1,"Id":3,"FilterMinimum":0,"FilterEnable":0,"DrowsyWarning":0,"Device":7,"DataOffset":0,"Name":"cooper - Status","LastVal":0,"FirstRec":0,"Variable":"Status","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:SwitchPower1","DrowsyError":0,"LastRec":1375185173,"FilterMaximum":0,"LastHistory":1104516000,"Type":1,"DataType":3,"Logging":1,"Id":6,"FilterMinimum":0,"FilterEnable":0,"DrowsyWarning":0,"Device":8,"DataOffset":0,"Name":"Intermatic - Status","LastVal":0,"FirstRec":0,"Variable":"Status","historyState":0}],"Graphs":[],"guiConfig":{"graphOptionAnimation":false,"_dc":"1374624177631","graphOptionMarker":true},"nextId":18} -7-===========================================================================

I'll try reformating the USB stick and see if that cleans it up...
Title: Re: New dataMine graphing and logging plugin
Post by: DrZWave on July 31, 2013, 12:33:41 pm
I uninstalled datamine then reinstalled and now its woprking ok.
Title: Re: New dataMine graphing and logging plugin
Post by: vio on August 05, 2013, 04:56:06 am
Hello,

Is history implemented? I'm looking for hourly/daily history files, but I can see only raw files in the dataMine/database dir.
Regards,
Title: Re: New dataMine graphing and logging plugin
Post by: BlueFlame on August 15, 2013, 07:32:23 am
Hi,

Trying to use your plug-in, but I can't seem to get it to work.

Installed the app.
Plugged a 16GB USB stick in the Vera3 bottom port. (formatted with FAT32 file system in Windows 7)

Now when I start the GUI, it always makes me select the USB stick.

Debug log:
Code: [Select]
06 08/15/13 13:21:40.753 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was: 220F-D2F0 now: 220F-D2F0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ef8a680>
09 08/15/13 13:22:03.538 JobHandler_LuaUPnP::Run device 90 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xc93b78 upnp: 0 <0x2adc3000>
50 08/15/13 13:22:08.123 luup_log:90: dataMine: Initialising dataMine System (0.980) <0x2b6f7680>
06 08/15/13 13:22:08.123 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b6f7680>
06 08/15/13 13:22:08.124 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xd261d8/NONE duplicate:1 <0x2b6f7680>
06 08/15/13 13:22:08.124 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
50 08/15/13 13:22:08.171 luup_log:90: dataMine: Error reading tmpfile during mountcheck <0x2b6f7680>
06 08/15/13 13:22:08.171 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
06 08/15/13 13:22:08.172 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
50 08/15/13 13:22:08.210 luup_log:90: dataMine: Mounting dataMine storage (/dev/sda1) to (/dataMine/) <0x2b6f7680>
50 08/15/13 13:22:08.289 luup_log:90: dataMine: Error reading tmpfile during mountcheck <0x2b6f7680>
06 08/15/13 13:22:08.290 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
06 08/15/13 13:22:08.290 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
50 08/15/13 13:22:08.325 luup_log:90: dataMine: Mount point error: ::/dev/sda1 <0x2b6f7680>
06 08/15/13 13:22:08.326 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b6f7680>
50 08/15/13 13:22:08.356 luup_log:90: dataMine: ERROR: Startup state is 'uninitialised'! <0x2b6f7680>
06 08/15/13 13:22:08.357 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xd261d8/NONE duplicate:1 <0x2b6f7680>
06 08/15/13 13:22:08.357 Device_Variable::m_szValue_set device: 90 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b6f7680>
50 08/15/13 13:22:08.358 luup_log:90: dataMine: Reinitialising configuration structure <0x2b6f7680>
50 08/15/13 13:22:08.358 luup_log:90: dataMine: Startup complete <0x2b6f7680>
-2-===========================================================================
/dev/sda1: LABEL="BFSPUSB7" UUID="220F-D2F0"
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
-4-===========================================================================

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 15.8 GB, 15833497600 bytes
256 heads, 53 sectors/track, 2279 cylinders
Units = cylinders of 13568 * 512 = 6946816 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1   *           1        2280    15462376   c Win95 FAT32 (LBA)
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -220F-D2F0
SetMountPoint   -/dev/sda1
SetManualMount  -0
-6-===========================================================================
{"dbVersion":2,"Events":{"count":0,"last":0},"Version":"0.980","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================

Any help appreciated.
Title: Re: New dataMine graphing and logging plugin
Post by: Disconnect on August 18, 2013, 10:57:11 am
Hi,

Trying to use your plug-in, but I can't seem to get it to work.

Installed the app.
Plugged a 16GB USB stick in the Vera3 bottom port. (formatted with FAT32 file system in Windows 7)

Now when I start the GUI, it always makes me select the USB stick.


I had the same problem and I needed to ssh in and 'mkdir /dataMine'. After going through the setup again it started working, mostly.

Unfortunately "mostly" means "except for showing any data points or graphs".. the variables are all listed in the configuration side, but the graphs side is completely blank.

(Update: I figured it out. You have to turn on logging individually in the config for each sensor. I see it in the docs now, but it could probably be highlighted better..)
Title: Re: New dataMine graphing and logging plugin
Post by: BlueFlame on August 19, 2013, 09:06:44 am
I had the same problem and I needed to ssh in and 'mkdir /dataMine'. After going through the setup again it started working, mostly.
Thanks! I created the dataMine dir on the USB stick. Don't know what I was thinking at that moment...
It works ok now :)
Title: Re: New dataMine graphing and logging plugin
Post by: chris66 on August 29, 2013, 11:59:02 am
Hi Chris, when I select month or year, after downloading data, I have Warning: Data truncated due to timeout. Is there a place to increase timeout ? Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 06, 2013, 10:18:38 am
UPDATED

Chris,
This morning I woke up to a graph that has flat-lined, literally. All has been perfectly fine since I removed the USB logging feature - until today. I have not rebooted the Vera well over a month.

This morning...
Datamine graph did not show any datapoints & the \root\Datamine folder 'looked' empty - no files or folders.
 
After reboot it started logging again and the graph is working, even historical - and the folder had files and folders in it again?

All Vera functions were working fine prior to the reboot... except DM

Debug log attached.

Channels: 30
Disk Space Total (kb): 524640
Disk Space Used (kb): 6672
Disk Space Free (kb): 517968

Title: Re: New dataMine graphing and logging plugin
Post by: HansW on September 15, 2013, 06:27:52 am
Hello,

After upgrading my Vera 3 to the latest .622 firmware, Datamine stopped logging and showing data. All I get is flat lines....

I disabled USB logging on Vera without effect and donot dare change anything else since I am afraid to loos the datamine log data.

Please Help!

Attached is the log
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 15, 2013, 10:55:55 am
Hello,

After upgrading my Vera 3 to the latest .622 firmware, Datamine stopped logging and showing data. All I get is flat lines....

I disabled USB logging on Vera without effect and donot dare change anything else since I am afraid to loos the datamine log data.

Please Help!

Attached is the log

Try removing the existing DM logs. I have had a problem for at least a year now where DM will stop working (graphing and/or logging) if there are too many files in the folder. How many is too many? No clue. It just happens. I remove the files an it starts working again after a reload.
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on September 15, 2013, 11:36:06 am
Aaron thanks but do you have any concrete pointers on how to accomplish that?

I did manage to login the Vera but cannot see any data letalone logs....

root@MiOS_30102864:~# cd /dataMine
root@MiOS_30102864:/dataMine# ls
Notifications [R2250].txt
Notifications [R2251].txt
Notifications [R2252].txt
Notifications [R2253].txt
Notifications [R2254].txt
Notifications [R2255].txt
Notifications [R2280].txt
dataMineConfig.json
dataMineConfig.json.Fri Sep 13 00-25-26 2013.backup
dataMineConfig.json.Fri Sep 13 02-18-46 2013.backup
dataMineConfig.json.Fri Sep 13 03-00-34 2013.backup
dataMineConfig.json.Sat Sep 14 02-22-47 2013.backup
dataMineConfig.json.Sat Sep 14 02-23-31 2013.backup
dataMineConfig.json.Sun Sep 15 02-16-05 2013.backup
dataMineConfig.json.Sun Sep 15 11-49-36 2013.backup
dataMineConfig.json.Sun Sep 15 12-12-12 2013.backup
dataMineConfig.json.Sun Sep 15 13-39-21 2013.backup
database
sunriseSunset.txt
root@MiOS_30102864:/dataMine# cd database
root@MiOS_30102864:/dataMine/database# ls
1
root@MiOS_30102864:/dataMine/database# cd 1
root@MiOS_30102864:/dataMine/database/1# ls
config.json
root@MiOS_30102864:/dataMine/database/1#
Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 15, 2013, 12:27:42 pm
I use SCP to manage the files/folders

The logs are the .txt files - you don't have many. You might want to uninstall - deleted the \dataMine folder - then reinstall.
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on September 15, 2013, 12:52:03 pm
Thanks Aaron.

I just discovered that the USB drive does not seem to mount. So the data I showed is in the unit and not on the USB drive. And for some reason the UUID in the datamine advanced settings tab differs from the two usb drives I have in my system.... One was used for system logging and the other for datamine. I cannot read them on Windows or Mac so I am in the blind. I tried mounting them both but both seem without a database directory......

I do hope I have not lost all logs....
Title: Re: New dataMine graphing and logging plugin
Post by: Brientim on September 15, 2013, 12:59:48 pm
You have a few options. The USB will be formatted in Linux ext2/3 and you can insert them into another unix based system, router and see if they mount and share otherwise you can try a program like the one below.

http://www.pendrivelinux.com/access-ext2-and-ext3-from-windows/
Title: Re: New dataMine graphing and logging plugin
Post by: brettow on September 18, 2013, 06:42:23 pm
Firstly great plugin now i can finally turn off the hungry PC that was doing the same job.  ;D

I seem to have an odd problem though to do with where the data is being stored.

USB is mounted
/dev/sda1: LABEL="DATAMINE" UUID="CEC7-BEE8"

And I can see its registering correctly in dataMine with the size and type etc. (See pic)

However a dataMine folder has been created on the Vera root and everything seems to be logging to there.

Some caveats. Im running through a USB hub and also connected to the hub is a current cost Envi.

I have uninstalled, reinstalled, played around with setting the mount points however still cant get it stable.
Title: Re: New dataMine graphing and logging plugin
Post by: chris66 on September 19, 2013, 06:49:55 am
Hi Chris, when I select month or year, after downloading data, I have Warning: Data truncated due to timeout. Is there a place to increase timeout ? Thanks.

Maybe I was not explicit enough, here the output I get for one year, or even for one month. One  week is ok...
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on September 20, 2013, 01:45:59 am
Have a look here:

http://forum.micasaverde.com/index.php/topic,13981.msg106305.html#msg106305
Title: Re: New dataMine graphing and logging plugin
Post by: chris66 on September 20, 2013, 10:59:04 am
Thanks a-lurker, but I've already tried that with 30 and I have the following error message: Error downloading data from vera response =0
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on September 22, 2013, 07:16:43 am
I'm just getting back into configuring dataMine.  I tried a few years ago on UI4 but wasn't about to get it to work.  I'm now on a Vera 2, UI5, and just this morning I've installed the plugin from the marketplace.

1GB USB formatted FAT32, top USB port. SetMountPoint set to /dev/scsi/host0/bus0/target0/lun0/part1 .  My USB stick is labeled DATAMINE, so my SetMountUUID is DATAMINE.

Upon starting up the graphing page, I get this dialog:

link to screen shot (http://url/dev/scsi/host0/bus0/target0/lun0/part1)

When I click "save," I get a message to restart Luup (at this point, errorStatus in dM is "true"); I do that, then return to the graphing page & see that it says Vera has restarted & I should restart the dM GUI (but that would just put me back into the loop).  I click "OK" and I'm able to see the dM GUI, but there are no channels available:

link to screen shot (http://i.imgur.com/GfFiQYe.png) - Online status is red. dM has errors.

...then when I return to the advanced screen of dM, my SetMountUUID has changed to E2BC-A35C.

EDIT: Vera logging to USB is off.  There are no other USB drives installed in Vera.

EDIT 2: I found via WinSCP there was no "datamine" folder in the root directory.  I added that, rebooted a few times, but errorStatus is still true...  looking for datamineConfig.json.  It's not in the supplied ZIP file.

EDIT 3: I went through & stripped out any dM files from my attempt at getting it working on UI4.  Restarted Luup a few times, then reloaded the plugin through the Marketplace.  errorStatus = false.  Added SetMountPoint and SetMountUUID as above.  errorStatus = true.  Went to dM GUI, selected my USB drive, and then got a new error:

link to screen shot (http://i.imgur.com/CkuBqb9.png)

...couldn't do anything from here except close the window.  Went back to UI5, reloaded, restarted dM GUI.  errorStatus = true, DATAMINE still in SetMountUUID field.  Manually populated that field with E2BC-A35C and restarted.  Returned to dM GUI, reloaded, still in error.

I'm not sure what else to do here.  I have a datamine directory in root, but it's empty.  If I have to populate it, I'm not sure with what.

Title: Re: New dataMine graphing and logging plugin
Post by: Aaron on September 22, 2013, 10:11:37 am
DeltaNu1142
Unless you have a backup of your prior data (dataMine folder & subfolders/files) I suggest you uninstall DM and deleted all folders using WinSCP - then reinstall.

If you DO have a backup and want to keep it, then look at my last post here...
http://forum.micasaverde.com/index.php/topic,16243.msg127910.html#new

Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on September 22, 2013, 10:22:03 am
Unless you have a backup of your prior data (dataMine folder & subfolders/files) I suggest you uninstall DM and deleted all folders using WinSCP - then reinstall.
I already did that:
I went through & stripped out any dM files from my attempt at getting it working on UI4.  Restarted Luup a few times, then reloaded the plugin through the Marketplace.  errorStatus = false.  Added SetMountPoint and SetMountUUID as above.  errorStatus = true.  Went to dM GUI, selected my USB drive, and then got a new error:

link to screen shot (http://i.imgur.com/CkuBqb9.png)

...couldn't do anything from here except close the window.  Went back to UI5, reloaded, restarted dM GUI.  errorStatus = true, DATAMINE still in SetMountUUID field.  Manually populated that field with E2BC-A35C and restarted.  Returned to dM GUI, reloaded, still in error.
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on September 22, 2013, 12:29:28 pm
FYI, attached is a screen shot of all the dM files I could find on Vera post-uninstall via "plugins" page on MiOS.  Now deleted & will try again.

Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on September 22, 2013, 12:54:24 pm
errorStatus = false...  but no channels showing.  I currently have one sensor & one dimmer module in Vera.  This is a fresh install, making no changes to log to the USB drive.

When I start the graphing page, I get the USB dialog.  I have the USB drive installed in the upper slot in Vera, but haven't done any configuration for it on this install.
Title: Re: New dataMine graphing and logging plugin
Post by: n0ir on September 28, 2013, 07:06:35 pm
New user so please be gentle...

I have installed the plug-in from the marketplace and a new device ("dataMine graphing and logging") has appeared.

I have formatted a USB-drive (and named it DATAMINE) and plugged it in the Vera Lite (in a USB-hub together with another USB-drive for logs). I have entered DATAMINE in SetMountUUID under Settings->Advanced.

The installation manual says that I then should run the GUI. When I click "dataMine graphing page" under Settings->dataMine I receive this error:

"Not Found

The requested URL /dm/index.html was not found on this server."

Should I start the GUI from somewhere else? My laptop and my Vera Lite is on the same wifi network.
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on September 29, 2013, 03:25:49 am
When I click "dataMine graphing page" under Settings->dataMine I receive this error:

"Not Found

The requested URL /dm/index.html was not found on this server."

Can you check if the web folder www/dm/ is there?

You can do this by logging on as root e.g Via WinSCP. Looking at mine via a terminal app I can see the following.

root@MiOS_35111111:/www/dm# ls
dataMine.css        extjs               images              js
dataMineWeb.tar.gz  graph.html          index.html

If not, then you may want to try installing it again.
If it is, then it might well be due to how you're accessing it, i.e I don't believe you can access the dm web interface remotely, you have to be on the same LAN for it to work.
Title: Re: New dataMine graphing and logging plugin
Post by: n0ir on September 29, 2013, 04:56:59 am
When I click "dataMine graphing page" under Settings->dataMine I receive this error:

"Not Found

The requested URL /dm/index.html was not found on this server."

Can you check if the web folder www/dm/ is there?

You can do this by logging on as root e.g Via WinSCP. Looking at mine via a terminal app I can see the following.

root@MiOS_35111111:/www/dm# ls
dataMine.css        extjs               images              js
dataMineWeb.tar.gz  graph.html          index.html

If not, then you may want to try installing it again.
If it is, then it might well be due to how you're accessing it, i.e I don't believe you can access the dm web interface remotely, you have to be on the same LAN for it to work.

My fault. I was a retard.

I thought I accessed the Vera locally because the local IP-address was shown in the UI5. However, when I checked my bookmark it was for https://cpui5.mios.com, not the local IP-address.

Datamines GUI is now possible to access and I have learned to use WinSCP, so it was a constructive mistake... ;)
Title: Re: New dataMine graphing and logging plugin
Post by: exxon on October 01, 2013, 04:01:56 pm
Hi,

first of all thanks for awesome stuff :)
I just installed plugin (hence not much data) but already ran into problem:
pretty much every operation such as
1) zoom graph (select portion, or use predefined day/month/year)
2) select different counter and click "update graph"
leads to message "downloading graph data" and no response. have to close and re-open page.
interestingly, _first_ counter that I chose to see always works fine.
Windows8 / IE10 / VeraLite

any suggestions?


thanks!

PS. just found it works fine in opera
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on October 01, 2013, 11:03:36 pm
I would avoid IE. Use of another browser is more beneficial.

- Garrett

Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 13, 2013, 08:32:44 am
Having a problem with first install and getting going. Probably just being stupid...

I have a Vera lite, and 1 usb stick (8GB) connected via a USB hub (powered). From what i have read that might be the problem but i cannot find the solution. I have installed the plugin after i had installed the usb hub and usb stick. So far so good, couple of reloads as usual. When i click on datamine graphing page a new page opens up in my firefox. There i select my usb stick, press save. Reload Vera. The device is now showing o channels. Click on settings. All disk space entries are empty (indicating the usb stick is not there i think). In advanced, i have entered SetMountPoint to /dev/sda1 which is i think right for Vera lite, but from what i understand might have to be different for use with a usb hub ? Error atatus is true. When i go on the graphing page, all variables are there, but online status is red.
Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 13, 2013, 11:20:46 am
This is the log from the debug page:

09   10/13/13 15:31:48.092   JobHandler_LuaUPnP::Run device 322 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xade820 upnp: 0 <0x2b5ac000>
50   10/13/13 15:32:02.761   luup_log:322: dataMine: Initialising dataMine System (0.980) __LEAK__ this:4096 start:2293760 to 0x15ea000 <0x2bcdf680>
06   10/13/13 15:32:02.762   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2bcdf680>
06   10/13/13 15:32:02.764   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xd47060/NONE duplicate:1 <0x2bcdf680>
06   10/13/13 15:32:02.764   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
50   10/13/13 15:32:02.826   luup_log:322: dataMine: Error reading tmpfile during mountcheck <0x2bcdf680>
06   10/13/13 15:32:02.826   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
06   10/13/13 15:32:02.826   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
50   10/13/13 15:32:02.869   luup_log:322: dataMine: Mounting dataMine storage (/dev/sda1) to (/dataMine/) <0x2bcdf680>
50   10/13/13 15:32:02.974   luup_log:322: dataMine: Error reading tmpfile during mountcheck <0x2bcdf680>
06   10/13/13 15:32:02.974   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
06   10/13/13 15:32:02.974   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
50   10/13/13 15:32:03.006   luup_log:322: dataMine: Mount point error: ::/dev/sda1 <0x2bcdf680>
06   10/13/13 15:32:03.007   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2bcdf680>
50   10/13/13 15:32:03.040   luup_log:322: dataMine: ERROR: Startup state is 'uninitialised'! <0x2bcdf680>
06   10/13/13 15:32:03.040   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xd47060/NONE duplicate:1 <0x2bcdf680>
06   10/13/13 15:32:03.041   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bcdf680>
50   10/13/13 15:32:03.041   luup_log:322: dataMine: Reinitialising configuration structure <0x2bcdf680>
50   10/13/13 15:32:03.073   luup_log:322: dataMine: Startup complete <0x2bcdf680>
50   10/13/13 15:32:18.185   luup_log:322: dataMine: Delete backups complete - No files found <0x2eadf680>
50   10/13/13 15:32:18.208   luup_log:322: dataMine: Error reading tmpfile during spacecheck <0x2eadf680>
09   10/13/13 15:33:55.437   JobHandler_LuaUPnP::Run device 322 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xeeb800 upnp: 0 <0x2aac5000>
50   10/13/13 15:34:09.678   luup_log:322: dataMine: Initialising dataMine System (0.980) __LEAK__ this:4096 start:2822144 to 0x1a78000 <0x2b3f9680>
06   10/13/13 15:34:09.679   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b3f9680>
06   10/13/13 15:34:09.681   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x1151880/NONE duplicate:1 <0x2b3f9680>
06   10/13/13 15:34:09.681   Device_Variable::m_szValue_set device: 322 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b3f9680>
50   10/13/13 15:34:09.712   luup_log:322: dataMine: Error reading tmpfile during mountcheck <0x2b3f9680>
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on October 13, 2013, 06:18:43 pm
This may be another case of a missing 'dataMine directory' - as indicated by "Error reading tmpfile during mountcheck"

http://forum.micasaverde.com/index.php/topic,16975.msg131509.html#msg131509
Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on October 14, 2013, 07:13:47 am
I have my logfiles placed on a NAS. It rebooted this morning and now I have 0 channels enabled in DataMine. I don't know if something has gotten corrupted. I have rebooted the Vera unit.

I see there are backup files that DataMine has created. Can I restore my old config and logs somehow?
Title: Re: New dataMine graphing and logging plugin
Post by: brettow on October 16, 2013, 06:48:36 am
Hi All,

I have looked over the thread however cant seem to find if or how its possible to calculate a total accumulated figure per day from the logging.

e.g from a device for power logging in watts is it possible to show total watts used in any given day taken from the logs?

Cheers,
Brett
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 17, 2013, 03:56:48 am
I have my logfiles placed on a NAS. It rebooted this morning and now I have 0 channels enabled in DataMine. I don't know if something has gotten corrupted. I have rebooted the Vera unit.

I see there are backup files that DataMine has created. Can I restore my old config and logs somehow?

Yes, you can. Take a look at
http://forum.micasaverde.com/index.php/topic,15073
Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on October 17, 2013, 08:39:56 am
I have my logfiles placed on a NAS. It rebooted this morning and now I have 0 channels enabled in DataMine. I don't know if something has gotten corrupted. I have rebooted the Vera unit.

I see there are backup files that DataMine has created. Can I restore my old config and logs somehow?

Yes, you can. Take a look at
http://forum.micasaverde.com/index.php/topic,15073
Thanks, Anders. I actually already got it working with the help of Hek (thank you) doing exactly what the post you linked to says. Hopefully everything will be stable now.
Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on October 17, 2013, 02:09:03 pm
Hi All,

I have looked over the thread however cant seem to find if or how its possible to calculate a total accumulated figure per day from the logging.

e.g from a device for power logging in watts is it possible to show total watts used in any given day taken from the logs?

Cheers,
Brett
Agree. I really like dataMine and would like to see more graphing options. Like the ones you mention and also stacked bar and line graphs (like the dashboard but over time). And the possibility to automatically publish graphs/dashboard on a NAS share would be really nice.
Title: Re: New dataMine graphing and logging plugin
Post by: Peter on October 17, 2013, 04:21:45 pm
How can I see my gaz consumption for a day during a year instead of the increasing total of gaz consumption for a day?
See picture?
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 17, 2013, 05:47:11 pm
Hi

i can not get this plugin to work, i tried for many days with an USB stick connect to a VERA3 lite.
the graph page shows correctly but channel count is 0.

As i am not sure if Vera recognises the USB i tried the manual configuration as described in the pdf that comes with the app.
(/tmp/log/cmh/dataMine)

still i do not get any channels.  i have attached the debug info. can anyone help me?

Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 17, 2013, 06:07:03 pm
and herewith a screendump of the advanced tab settings..
sometimes i get a channel count =1 ... ???
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 18, 2013, 01:36:10 am
Hi

i can not get this plugin to work, i tried for many days with an USB stick connect to a VERA3 lite.
the graph page shows correctly but channel count is 0.

As i am not sure if Vera recognises the USB i tried the manual configuration as described in the pdf that comes with the app.
(/tmp/log/cmh/dataMine)

still i do not get any channels.  i have attached the debug info. can anyone help me?

Using /tmp/log/cmh/dataMine is only valid if you use the same USB stick for the Vera logs. If you have a separate USB stick for the dataMine, the default setup should be fine.

If you tell us how your USB stick(s) are used, we should be able to lead you in the right direction  ;)
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 18, 2013, 02:16:27 am
Hi

Maybe i provided some confusing information.

I tried the setup with Vera also using the USB stick for the logs, but i disabled that now.
as far as I know the USB stick is fully available to datamine. i only have one USB stick

but i am not a linux expert, i can not see if the usb is correctly mounted. the graph GUI shows green status and it never asks me for USB setup, so i assume it is ok.

the setdatadirectory = /dataMine
setmountUUID = this is the label of the USB stick
setnountpoint = /dev/sda1

i am not sure if the log file that i attached was helpfull.  i created another log file which i am attaching now.

i hope this information helps?

datamine does not show any channels. it shows a channel count of 1
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 18, 2013, 03:28:33 am
otterlo,
I may be outside my competence here, but:

your screen capture of the advanced tab and the latest text file do not match, as far as I can see.
From the text file, it seems like you use a manual mounting, i.e. the dataMine will not mount the USB stick automatically.
50   10/17/13 23:10:21.449   luup_log:65: dataMine: Manual mounting to (/tmp/log/cmh/dataMine/) <0x2c0db680>
50   10/17/13 23:10:21.497   luup_log:65: dataMine: ERROR: Unable to open config file for read :: /tmp/log/cmh/dataMine/dataMineConfig.json <0x2c0db680>

AFAIK, your config is somewhat confused. You have to either
- let the Vera logs use the USB stick and then you will be able to share the USB between logs and dtaMine (this is how I set up mine).
- let the dataMine use the USB stick alone, but then you cannot use the /tmp/log/cmh directory, as that will be on a file system that disappears at every Vera reboot. In that case, you remove the manual mounting, and set the dataMine directory to /dataMine as instructed in the manual.
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 18, 2013, 06:13:33 am
Hello

many thanks for your efforts to help me.

it is also confusing to me, exactly the same. I have disabled the logs to USB in Vera. So my feeling is that the USB is there to be used by datamine only.
I have not done a manual configuration. i have attached the advanced section of the  plugin and i feel that everything is correct.
what i can not see if the USB is correctly mounted.

i hope that the 2 screenshots will help?
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 18, 2013, 06:55:07 am
From the MountLocation and MountType in the lower of your two screenshots, I think that the USB stick is mounted. The type is ext4, which is a linux filesystem. Most likely the USB stick was formatted ext4 when you tried to use it for the Vera logs.

So, I think your USB is mounted. Then, the next question is whether dataMine actually finds the directory /dataMine.

As previously mentioned, it looks from your text file that dataMine looks for data in the /tmp/logs/cmh, which is incorrect with your USB setup.

What I would reccomend, would be to log into the Vera (ssh or similar) and browse around to see if there is anything under /dataMine.
But, as you state that you are no linux expert, does that imply that you never have used linux, and never logged into your Vera previously? In that case, it might be other approaches that are better.

Are you able to start the graphical user interface (page 5 in the user manual) at all?
If so, do you see anything in the configuration tab?
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 18, 2013, 07:21:38 am
Hello

I formatted the USB stick using my windows pc  on FAT32.
I am not sure if ext 4 is the right entry in this field?

i am not expert on linux but I can connect to Vera using winscp and browse the filestructure.
currently i am not at home and i have to wait until tonight.
i do remember that /datamine is available but i can not judge on the content of that directory.
i will have a look at that.

the GUI works and I can see the configuration tab and see all the devices and parameters. the only things that does not work is that it does not recognise any channel (or variables) so that there is nothing to display in the graph
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 18, 2013, 07:46:57 am
Hello

I formatted the USB stick using my windows pc  on FAT32.
I am not sure if ext 4 is the right entry in this field?

i am not expert on linux but I can connect to Vera using winscp and browse the filestructure.
currently i am not at home and i have to wait until tonight.
i do remember that /datamine is available but i can not judge on the content of that directory.
i will have a look at that.

the GUI works and I can see the configuration tab and see all the devices and parameters. the only things that does not work is that it does not recognise any channel (or variables) so that there is nothing to display in the graph

The USB stick got formatted when you enabled the Vera logging on USB, I presume. But using ext4 should be no problem. The only drawback is that it might be difficult to read the USB stick if you put it into a Windows computer.

Under /dataMine you should see at least one config file with the .json extension, and a directory (I think it's called database). I'm not at home either, so I'm unable to check it out.

What happens if you enable logging (as described on page 16 in the user guide) on one variable that changes rapidly? Are you able to see any graph at all?
The reason for asking: If the dataMine does save all its data outside the USB stick, I presume it woul lose its complete setup at every reboot. But nontheless it should be able to display a graph for data that has changed several times since the reboot.
Title: Re: New dataMine graphing and logging plugin
Post by: Peter on October 18, 2013, 08:01:47 am
How can I see my gaz consumption for a day during a year instead of the increasing total of gaz consumption for a day? Or is this not posible?
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on October 18, 2013, 09:01:26 am
Clearly...  I'm still doing something wrong.  I decided to revisit this after my first UI5 attempt a few weeks ago.  I'm following the installation PDF.  My 8GB USB drive is labelled "DATAMINE" and formatted in FAT32.  I cleared the "MountLocation" field and populated the other fields as shown in Step 1.

Then I reload Luup a few times, start the dM graphing page, and get the screen shown in Step 2.  I select the UUID, go back to MiOS, reload Luup, and get a notification on the dM graphing page that Vera has restarted and I should restart the dM UI.

I close the dM page, go back to Vera, restart, then click on the dM plugin to view Advanced options.  I see the screen shown in Step 3a.  SetMountUUID has been populated, as has mountLocation.  However, check out disk info (Step 3b)...  6464 total? 6464 used?
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on October 18, 2013, 09:07:01 am
I've confirmed through WinSCP that the datamine directory is under root.  Not sure where to go from here.

Online status is good/error status is false.  0 channels showing up on the plugin.  I don't know why the debug page shows up as it does, but I know it takes forever to load.
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on October 18, 2013, 09:25:28 am
Just pulled the USB stick to see what was on it...  see attached.  This was all put on by dM.
Title: Re: New dataMine graphing and logging plugin
Post by: AndersH on October 18, 2013, 10:35:29 am
Just pulled the USB stick to see what was on it...  see attached.  This was all put on by dM.

This is looking much like I would expect on an empty system.
The fact that it shows 0 channels is also normal, provided that you didn't enable logging on any channels yet. I guess that is the problem, that you are unable to access the configuration tab to enable logging?
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on October 18, 2013, 10:52:34 am
...edited...  I think I've got it...

YEP.  My problem was I wasn't drilling down in each device to the points I want to log.  I've now got 6 channels up.  Even though the screens above show my USB UUID, I'm not convinced yet it's logged to USB.  I'll have to pull it out & look later today.

Once I'm convinced this works, I'm likely to move on to logging to Synology NAS.

Thanks!
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 18, 2013, 01:22:05 pm
Hello

I formatted the USB stick using my windows pc  on FAT32.
I am not sure if ext 4 is the right entry in this field?

i am not expert on linux but I can connect to Vera using winscp and browse the filestructure.
currently i am not at home and i have to wait until tonight.
i do remember that /datamine is available but i can not judge on the content of that directory.
i will have a look at that.

the GUI works and I can see the configuration tab and see all the devices and parameters. the only things that does not work is that it does not recognise any channel (or variables) so that there is nothing to display in the graph

The USB stick got formatted when you enabled the Vera logging on USB, I presume. But using ext4 should be no problem. The only drawback is that it might be difficult to read the USB stick if you put it into a Windows computer.

Under /dataMine you should see at least one config file with the .json extension, and a directory (I think it's called database). I'm not at home either, so I'm unable to check it out.

What happens if you enable logging (as described on page 16 in the user guide) on one variable that changes rapidly? Are you able to see any graph at all?
The reason for asking: If the dataMine does save all its data outside the USB stick, I presume it woul lose its complete setup at every reboot. But nontheless it should be able to display a graph for data that has changed several times since the reboot.


Hi

I think it works for me now as well! checked everything but i believe the trick was to enable logging on the variables..... :-[
It was not until you pointed me to this that I started to understand.

thank you very much and i will try to see how everything works!
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 20, 2013, 06:10:53 am
i have another question.

I created a saved graph with reference name "kamertemperatuur", so that I can call this from my favorties in the webexplorer.
I saved a graph to display outside temperatuur with duration for 1 day

when I call the graph on the following way: http://xxx.xxx.x.xx/dm/graph.html?ref=binnentemp, i then see the graph displays nicely but it does not seem to display the recent 2 hours. for example, the time is 12 o clock now and this graph always shows up and to 10 o clock.

Does anybody know why i face this? (hope it is not related to GMT time settings....)
Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 29, 2013, 06:06:24 pm
This may be another case of a missing 'dataMine directory' - as indicated by "Error reading tmpfile during mountcheck"

http://forum.micasaverde.com/index.php/topic,16975.msg131509.html#msg131509

I did get datamine working by creating a datamine directory in root. I am a little worried that datamine is writing data onto Vera rather than the USB. I have this displayed on the device:

Channels
18
Disk Space Total (kb)
518692
Disk Space Used (kb)
32688
Disk Space Free (kb)
459656
7
dataMine graphing page
dataMine debug page

in the advance settings i have

mountLocation  /dev/sda1
mountType  ext3

does this look right ? I am using a 4Gb usb stick
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on October 29, 2013, 07:16:38 pm
Hi

it looks OK. if you see it in the root it may be the mounted datamine directory which is actually on your usb stick.
just to make sure you could post you Datamine log file so that we can take a look to see?
Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 30, 2013, 03:09:19 pm
Here is the log:


50   10/30/13 18:57:31.926   luup_log:327: dataMine: D[177] S[urn:upnp-org:serviceId:WindSensor1] V[GustSpeed] newVal on startup is 7 was 3 __LEAK__ this:65536 start:3473408 to 0x1c4b000 <0x2bc1f680>
50   10/30/13 18:57:31.938   luup_log:327: dataMine: Watching: D[177] S[urn:upnp-org:serviceId:WindSensor1] V[GustSpeed] <0x2bc1f680>
06   10/30/13 18:57:31.938   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 18 #hooks: 0 upnp: 0 v:0x11fac98/NONE duplicate:0 <0x2bc1f680>
06   10/30/13 18:57:31.939   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 18 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2bc1f680>
06   10/30/13 18:57:31.991   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 18 now: 18 #hooks: 0 upnp: 0 v:0x11fac98/NONE duplicate:1 <0x2bc1f680>
06   10/30/13 18:57:31.992   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 18 now: 18 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2bc1f680>
50   10/30/13 18:57:31.992   luup_log:327: dataMine: Startup complete <0x2bc1f680>
50   10/30/13 18:57:46.321   luup_log:327: dataMine: Delete backups complete - deleted 0 of 79 files <0x2ee1f680>
06   10/30/13 18:57:46.345   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskTotal was: 518692 now: 518692 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ee1f680>
06   10/30/13 18:57:46.346   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsed was: 24504 now: 25212 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2ee1f680>
06   10/30/13 18:57:46.346   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskFree was: 467840 now: 467132 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2ee1f680>
06   10/30/13 18:57:46.347   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsedPcnt was: 5 now: 5 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ee1f680>
06   10/30/13 19:02:11.963   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 18 now: 18 #hooks: 0 upnp: 0 v:0x11fac98/NONE duplicate:1 <0x2ee1f680>
06   10/30/13 19:02:11.964   Device_Variable::m_szValue_set device: 327 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 18 now: 18 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ee1f680>
-2-===========================================================================
/dev/sdb1: LABEL="MiOS" UUID="95981ad4-7111-4e4b-992a-95144dfcbd00"
/dev/sda1: LABEL="MiOS" UUID="8f78a627-ef78-4774-8f2a-69d8cab03d92"
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sda1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime,errors=continue,barrier=0,data=ordered)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /dataMine type ext3 (rw,relatime,errors=continue,barrier=0,data=ordered)
-4-===========================================================================

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 8119 MB, 8119648256 bytes
250 heads, 62 sectors/track, 1023 cylinders
Units = cylinders of 15500 * 512 = 7936000 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1   *           1          68      526999+ 83 Linux
/dev/sda2              69        1023     7401250  83 Linux

Disk /dev/sdb: 4026 MB, 4026531840 bytes
31 heads, 30 sectors/track, 8456 cylinders
Units = cylinders of 930 * 512 = 476160 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1        8457     3932032   b Win95 FAT32
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -64F1-A2B1
SetMountPoint   -/dev/sda1
SetManualMount  -0
-6-===========================================================================
{"Version":"0.980","dbVersion":2,"Events":{"count":7586,"last":1383159724},"guiConfig":[],"Variables":[{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383158135,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":1,"Device":256,"FilterMinimum":0,"Id":1,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"MikeHome","FilterEnable":0,"Logging":1,"Variable":"Status8","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383158135,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":0,"Device":256,"FilterMinimum":0,"Id":2,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"MikeAway","FilterEnable":0,"Logging":1,"Variable":"Status7","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383146476,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":1,"Device":256,"FilterMinimum":0,"Id":3,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"AutoHome","FilterEnable":0,"Logging":1,"Variable":"Status6","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383146476,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":0,"Device":256,"FilterMinimum":0,"Id":4,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"AutoAway","FilterEnable":0,"Logging":1,"Variable":"Status5","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383146502,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":0,"Device":256,"FilterMinimum":0,"Id":5,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"JustNear","FilterEnable":0,"Logging":1,"Variable":"Status4","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383072862,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":0,"Device":256,"FilterMinimum":0,"Id":6,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"MikeNear","FilterEnable":0,"Logging":1,"Variable":"Status3","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383146502,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":1,"Device":256,"FilterMinimum":0,"Id":7,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"JustHome","FilterEnable":0,"Logging":1,"Variable":"Status2","historyState":0},{"Ghost":false,"Service":"urn:dcineco-com:serviceId:MSwitch1","LastRec":1383146502,"DrowsyError":0,"LastHistory":1104516000,"FilterMaximum":0,"Units":0,"Type":0,"FirstRec":0,"LastVal":0,"Device":256,"FilterMinimum":0,"Id":8,"DrowsyWarning":0,"DataType":0,"DataOffset":0,"Name":"MikeHome","FilterEnable":0,"Logging":1,"Variable":"Status1","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:TemperatureSetpoint1_Heat","LastRec":1383146476,"DrowsyError":0,"FilterMaximum":0,"DataType":7,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":246,"FilterMinimum":0,"LastVal":20,"LastHistory":1104516000,"Id":9,"DataOffset":0,"Name":"Heating Setpoint","FilterEnable":0,"Logging":1,"Variable":"CurrentSetpoint","historyState":0,"Lookup":{"Setpoint":0}},{"Ghost":false,"Service":"urn:upnp-org:serviceId:TemperatureSensor1","LastRec":1383119195,"DrowsyError":0,"FilterMaximum":0,"DataType":1,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":246,"FilterMinimum":0,"LastVal":20,"LastHistory":1104516000,"Id":10,"DataOffset":0,"Name":"Heating current temp","FilterEnable":0,"Logging":1,"Variable":"CurrentTemperature","historyState":0,"Lookup":{"Current temp":0}},{"Ghost":false,"Service":"urn:micasaverde-com:serviceId:SecuritySensor1","LastRec":1383146202,"DrowsyError":0,"FilterMaximum":0,"DataType":6,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":148,"FilterMinimum":0,"LastVal":0,"LastHistory":1104516000,"Id":11,"DataOffset":0,"Name":"Porch door tripped","FilterEnable":0,"Logging":1,"Variable":"Tripped","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:TemperatureSensor1","LastRec":1383159886,"DrowsyError":0,"FilterMaximum":0,"DataType":1,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":178,"FilterMinimum":0,"LastVal":11.7,"LastHistory":1104516000,"Id":12,"DataOffset":0,"Name":"Outside temp","FilterEnable":0,"Logging":1,"Variable":"CurrentTemperature","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:SwitchPower1","LastRec":1383118689,"DrowsyError":0,"FilterMaximum":0,"DataType":3,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":297,"FilterMinimum":0,"LastVal":0,"LastHistory":1104516000,"Id":13,"DataOffset":0,"Name":"Front door lights","FilterEnable":0,"Logging":1,"Variable":"Status","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:SwitchPower1","LastRec":1383072976,"DrowsyError":0,"FilterMaximum":0,"DataType":3,"Type":0,"DrowsyWarning":0,"FirstRec":0,"Device":296,"FilterMinimum":0,"LastVal":0,"LastHistory":1104516000,"Id":14,"DataOffset":0,"Name":"Porch light","FilterEnable":0,"Logging":1,"Variable":"Status","historyState":0,"Lookup":{"On":0}},{"Ghost":false,"Service":"urn:upnp-org:serviceId:IPhoneLocator1","DrowsyError":0,"LastRec":1383158136,"LastHistory":1104516000,"FilterMaximum":0,"DrowsyWarning":0,"Type":0,"Logging":1,"FilterEnable":0,"Id":15,"FilterMinimum":0,"Device":317,"Units":0,"DataType":0,"DataOffset":0,"Name":"Mike present","LastVal":1,"FirstRec":0,"Variable":"Present","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:IPhoneLocator1","DrowsyError":0,"LastRec":1383158804,"LastHistory":1104516000,"FilterMaximum":0,"DrowsyWarning":0,"Type":0,"Logging":1,"FilterEnable":0,"Id":16,"FilterMinimum":0,"Device":318,"Units":0,"DataType":0,"DataOffset":0,"Name":"Justine present","LastVal":1,"FirstRec":0,"Variable":"Present","historyState":0},{"Ghost":false,"Service":"urn:upnp-org:serviceId:WindSensor1","DrowsyError":0,"LastRec":1383159955,"LastHistory":1104516000,"FilterMaximum":0,"DrowsyWarning":0,"Type":0,"Logging":1,"FilterEnable":0,"Id":17,"FilterMinimum":0,"Device":177,"Units":0,"DataType":0,"DataOffset":0,"Name":"Windspeed","LastVal":4,"FirstRec":0,"Variable":"AvgSpeed","historyState":0,"Lookup":{"Windspeed":0}},{"Ghost":false,"Service":"urn:upnp-org:serviceId:WindSensor1","DrowsyError":0,"LastRec":1383159955,"LastHistory":1104516000,"FilterMaximum":0,"DrowsyWarning":0,"Type":0,"Logging":1,"FilterEnable":0,"Id":18,"FilterMinimum":0,"Device":177,"Units":0,"DataType":0,"DataOffset":0,"Name":"Wind gust speed","LastVal":5,"FirstRec":0,"Variable":"GustSpeed","historyState":0,"Lookup":{"Gust speed":0}}],"Graphs":[],"LastWrite":1383159731,"nextId":19}
-7-===========================================================================
Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 30, 2013, 03:10:36 pm
I could not post the whole log as it exceeds maximun characters, i hope you can see what you need to see

thanks
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on October 30, 2013, 03:34:38 pm
Looks perfectly OK.
Title: Re: New dataMine graphing and logging plugin
Post by: mikee123 on October 30, 2013, 03:44:43 pm
Great. thanks. I have a few strange things happening so needed to rule out memory problems.
Title: Re: New dataMine graphing and logging plugin
Post by: Peter on October 31, 2013, 02:15:35 pm
How do I change the quantity of gas per day in a given year chart instead of total consumption which become increasingly increment

sample

day 1 - 3 cubic meters of gas
day 2 - 5 cubic meters of gas
etc.
Title: Re: New dataMine graphing and logging plugin
Post by: skropko on November 01, 2013, 05:18:25 pm
Need some help...

The plugin has worked very well for the last year or so...  I noticed the other day all 20 of my channels were gone and it just said Channels: 0.  Have been trying to troubleshoot.  Uninstalled plugin, formatted USB stick, reinstalled plugin but no luck.  Here is the log...

-1-===========================================================================
02   11/01/13 17:08:39.195   JobHandler_LuaUPnP::FetchMissingFiles need to download D_DataMine1.json <0x2b272000>
02   11/01/13 17:08:39.195   JobHandler_LuaUPnP::m_sMissingFile_set D_DataMine1.json <0x2b272000>
01   11/01/13 17:08:39.277   Device_LuaUPnP::LoadDeviceDoc can't load /etc/cmh-lu//D_DataMine1.xml <0x2b272000>
02   11/01/13 17:08:39.277   JobHandler_LuaUPnP::m_sMissingFile_set /etc/cmh-lu//D_DataMine1.xml <0x2b272000>
01   11/01/13 17:08:39.277   JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 55/D_DataMine1.xml so device 55 is offline <0x2b272000>
09   11/01/13 17:08:45.275   JobHandler_LuaUPnP::Run device 55  room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xabc328 upnp: 0 <0x2b272000>
03   11/01/13 17:08:45.290   JobHandler_LuaUPnP::Run ready 30 devices installation 35002358 missing /etc/cmh-lu//D_DataMine1.xml version *1.5.622*  allow_evt: 1 allow_ud: 1 upnp 0 netpnp: 1 reprovision: 0 tz:18000 tnow:1383340125 dst: 1 gmt:-14400 tz:EST / EDT changed: 0 pMem 0xd28000 start 10428416 diff: 3366912 <0x2b272000>
09   11/01/13 17:09:00.469   JobHandler_LuaUPnP::Run device 55 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xe0d280 upnp: 0 <0x2b067000>
50   11/01/13 17:09:07.870   luup_log:55: dataMine: Initialising dataMine System (0.980) <0x2b99b680>
06   11/01/13 17:09:07.870   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: EMPTY now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.871   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetDataDirectory was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.872   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.872   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountPoint was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.873   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountOptions was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.873   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetManualMount was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.874   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetTimeoutPeriod was: EMPTY now: 12 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.875   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetHistoryEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.875   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetUseBackup was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.876   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetEventsEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.877   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0xf29150/NONE duplicate:0 <0x2b99b680>
06   11/01/13 17:09:07.878   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b99b680>
50   11/01/13 17:09:07.879   luup_log:55: dataMine: Installing update of dataMine web application <0x2b99b680>
50   11/01/13 17:09:13.170   luup_log:55: dataMine: ERROR: Startup state is 'uninitialised'! <0x2b99b680>
06   11/01/13 17:09:13.170   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xf29150/NONE duplicate:1 <0x2b99b680>
06   11/01/13 17:09:13.170   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b99b680>
50   11/01/13 17:09:13.171   luup_log:55: dataMine: Reinitialising configuration structure <0x2b99b680>
50   11/01/13 17:09:13.172   luup_log:55: dataMine: Startup complete <0x2b99b680>
50   11/01/13 17:09:28.169   luup_log:55: dataMine: Delete backups complete - No files found <0x2cb9b680>
50   11/01/13 17:09:28.187   luup_log:55: dataMine: Error reading tmpfile during spacecheck <0x2cb9b680>
06   11/01/13 17:13:01.799   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was:  now: 0445-251E #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2e3c8680>
09   11/01/13 17:13:18.853   JobHandler_LuaUPnP::Run device 55 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xa487b8 upnp: 0 <0x2b092000>
50   11/01/13 17:13:25.273   luup_log:55: dataMine: Initialising dataMine System (0.980) <0x2b7c5680>
06   11/01/13 17:13:25.274   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b7c5680>
06   11/01/13 17:13:25.274   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xb64520/NONE duplicate:1 <0x2b7c5680>
06   11/01/13 17:13:25.275   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b7c5680>
50   11/01/13 17:13:25.275   luup_log:55: dataMine: Mounting to UUID '0445-251E' <0x2b7c5680>
50   11/01/13 17:13:25.720   luup_log:55: dataMine: UUID = /dev/sda1: LABEL="MIOSDM" UUID="0445-251E" <0x2b7c5680>
50   11/01/13 17:13:25.747   luup_log:55: dataMine: Found UUID '0445-251E' at '/dev/sda1' <0x2b7c5680>
50   11/01/13 17:13:25.775   luup_log:55: dataMine: Error reading tmpfile during mountcheck <0x2b7c5680>
06   11/01/13 17:13:25.776   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b7c5680>
06   11/01/13 17:13:25.777   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b7c5680>
50   11/01/13 17:13:25.795   luup_log:55: dataMine: Mounting dataMine storage (/dev/sda1) to (/dataMine/) <0x2b7c5680>
50   11/01/13 17:13:25.852   luup_log:55: dataMine: Error reading tmpfile during mountcheck <0x2b7c5680>
06   11/01/13 17:13:25.852   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b7c5680>
06   11/01/13 17:13:25.853   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b7c5680>
50   11/01/13 17:13:25.871   luup_log:55: dataMine: Mount point error: ::/dev/sda1 <0x2b7c5680>
06   11/01/13 17:13:25.871   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2b7c5680>
50   11/01/13 17:13:25.883   luup_log:55: dataMine: ERROR: Startup state is 'uninitialised'! <0x2b7c5680>
06   11/01/13 17:13:25.884   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xb64520/NONE duplicate:1 <0x2b7c5680>
06   11/01/13 17:13:25.884   Device_Variable::m_szValue_set device: 55 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b7c5680>
50   11/01/13 17:13:25.885   luup_log:55: dataMine: Reinitialising configuration structure <0x2b7c5680>
50   11/01/13 17:13:25.886   luup_log:55: dataMine: Startup complete <0x2b7c5680>
50   11/01/13 17:13:40.168   luup_log:55: dataMine: Delete backups complete - No files found <0x2cbc5680>
50   11/01/13 17:13:40.186   luup_log:55: dataMine: Error reading tmpfile during spacecheck <0x2cbc5680>
-2-===========================================================================
/dev/sda1: LABEL="MIOSDM" UUID="0445-251E"
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
-4-===========================================================================

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 1998 MB, 1998585344 bytes
255 heads, 63 sectors/track, 242 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1         243     1949696   b Win95 FAT32
Partition 1 has different physical/logical endings:
     phys=(241, 254, 63) logical=(242, 187, 9)
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -0445-251E
SetMountPoint   -
SetManualMount  -0
-6-===========================================================================
{"dbVersion":2,"Events":{"count":0,"last":0},"Version":"0.980","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on November 01, 2013, 05:46:15 pm
This may be another case of a missing 'dataMine directory' - as indicated by "Error reading tmpfile during mountcheck"

http://forum.micasaverde.com/index.php/topic,16975.msg131509.html#msg131509


Use SCP to see if there is a "dataMine" directory under the root directory. If there isn't I would suggest:

There may be another way to do this but that's just my suggestion. I am currently of the belief that all new installs of the current dataMine plugin (0.980) will fail.
Title: Re: New dataMine graphing and logging plugin
Post by: skropko on November 01, 2013, 08:32:52 pm
Thanks... that seems to be related to the problem...

I ended up doing the following....

1. Removed plugin and USB stick.
2. Powered off
3. Inserted USB stick
4. Powered on
5. Manually created dataMine folder in root directory
6. Installed plugin
7. Launched GUI and set UUID

Because I actually backed up the contents of the USB stick before all of the troubleshooting, I copied back the config and 1 year of associated data.  After completing step 7 above, the 20 channels are back!

Either way, it looks like something related to the dataMine directory.  Manually creating it before installing seemed to do the trick on my end.

Thanks for the help.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 01, 2013, 10:56:33 am
Hi all,

Using rfx kit on sdA1 and have a hub plugged into sdB1 with 2x usb sticks in it. one labelled 'datamine' and one labelled 'veralogs'.

I have datamine working fine.

How do I check vera is logging to the other usb stick and not the same as datamine? I have ssh "mount" but I cant see specifics on each usb on the hub?  any thoughts?

-LightsOn
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on December 01, 2013, 10:23:48 pm
If you have the Vera logging to a USB drive and have another USB drive for Datamine, the problems begin.

All works fine UNTIL you do a Vera reboot. Then Vera logging will first re format the Datamine USB, and then discover your logging USB drive and continue logging to the correct USB logging USB drive. But the damage has been done. Datamine USB drive is no longer usable.

This happens at every Vera reboot. I tried changing USB slots, manually setting up the Datamine mount. Nothing helps. In the end I had to stop using Vera logging.
Vera support claims it is a Datamine issue, not Vera logging. I don't believe this. Upon reboot, Vera should know to go directly back to the correct USB drive.
Title: Re: New dataMine graphing and logging plugin
Post by: bubaleta on December 02, 2013, 03:30:05 am
I had the same problem until I installed datamine loging to a NAS.
Everything has been working flawlessly since.
Vera loging to usb and datamine to NAS.
The only precaution is to start NAS before vera.

I have a synology NAS with crashplan installed, so it backs up the data every day.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 02, 2013, 03:51:38 am
Hi,

Thank you for the help.

NAS seems the way forward then, does it have to be naz or could it just be a shared folder on a network on a windows 7 machine?

What settings have you added for NAS logging?

Thank you.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: bubaleta on December 02, 2013, 05:32:26 am
Hello,

You can read about it on this post

http://forum.micasaverde.com/index.php/topic,14910.0.html (http://forum.micasaverde.com/index.php/topic,14910.0.html)

A NAS is basically a network drive, so I don't see why a shared folder from a computer wouldn't work (as long as it is always on).

Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on December 02, 2013, 07:58:11 am
I had the same problem until I installed datamine loging to a NAS.
Everything has been working flawlessly since.
Vera loging to usb and datamine to NAS.
The only precaution is to start NAS before vera.
I have a synology NAS with crashplan installed, so it backs up the data every day.
I'll be working on configuring this next month.  I might ping you for help.  I run a Synology NAS that is on 100% of the time, and I have a feeling logging to USB could really improve performance.  My Vera 2 has been limping along lately.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 02, 2013, 09:38:47 am
Hi @bubaleta
Thank you for this ? I have read the entire thread and I agree this ?should? work but no one has explicitly done what I seek but worth a go.  This is very helpful thank you.  I see Chris has looked at creating a path, user name; password field in the app and when this arrives this would be supper useful to avoid having to do the other bits but for now I too think this is a workable solution.

@DeltaNu1142 ? I will let you know how I get on when I have a go at this.  Success rate seems high so far so hopefully you and I can join a working solution 
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on December 02, 2013, 02:55:34 pm
@DeltaNu1142 ? I will let you know how I get on when I have a go at this.  Success rate seems high so far so hopefully you and I can join a working solution
That would be great!

I will likely not have a chance to try this until a few weeks from now.
Title: Re: New dataMine graphing and logging plugin
Post by: bubaleta on December 03, 2013, 02:27:09 am
I have it running just fine for over a month without any problems...

I wrote a very detailed guide about installing datamine, including the possibility of loving to a sinology NAS.
I will translate it some time soon but for now it is only in spanish, but it has enough images to make it simple to understand.

http://domoticaparatodos.com/book-review/manual-de-instalacion-del-plugin-datamine-en-vera-parte-1/ (http://domoticaparatodos.com/book-review/manual-de-instalacion-del-plugin-datamine-en-vera-parte-1/)




Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on December 06, 2013, 08:29:16 am
All of a sudden I can't reach my saved graphs with the direct html links. Is this a known bug?
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on December 06, 2013, 02:09:00 pm
here everything works OK
Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on December 06, 2013, 02:45:44 pm
Very strange. I can't get night shading to work either. I wonder what has happened? Both this and direct html links have worked before.
Title: Re: New dataMine graphing and logging plugin
Post by: dferrey on December 06, 2013, 02:55:20 pm
Is there a tutorial or something similar where I can get a basic understanding of when I can expect updates from the channels I'm graphing?

I'm starting slowly by adding the current temps from my Trane and Aeon Multisensor.  The trane updates pretty regularly although not on any obvious, to me, schedule.  The Aeon rarely updates.  Where can I look to find out when the updates will happen and if that can be altered?

Thanks.
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on December 06, 2013, 04:58:09 pm
@freddan101

There may be a few reasons: anything happened to your browser? Do you see the same when viewing from another computer? Maybe rebooting vera might help

@dferrey
My experience is that the update of the signal depends on how frequently the sensor sends out an update. There is no regular polling with fixed frequency. Any change will be logged and sometimes the signal does not change for a lomg time.  i could not find a manual for this.
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on December 15, 2013, 10:16:48 am
I love Datamine but I have had problems with the data disappearing multiple times now.

After reinstalling everything workes fine for about a week or a bit more and then I loose everything again.

I have 2 usb drives connected via a hub and it looks like Datamine without cause switches to sda1 to sdb1.

Is this a known bug and what can I do? I need the Hub since I have a RFX433 device as well.

Any help appreciated even if I have to reconfigure all 34 channels again for the xxxth time!

Hans
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on December 15, 2013, 11:52:03 am
If you search a little harder, you'll notice that this is a Vera issue when using multiple usb drives. The mount point can change on Vera reboot and this could be the reason why you loose data or data is not being recorded. What is the second usb drive being used for logging?

- Garrett
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on December 15, 2013, 01:06:03 pm
Thanks Garret,

indeed it is used for logging. In the Datamine manual I read that a second USB is preferred. I have seen a reformatting of the logging USB drive also so I understand that. What can I do to get a properly working system though without having to check its status weekly?

Hans
Title: Re: New dataMine graphing and logging plugin
Post by: bucko on December 15, 2013, 08:09:12 pm
This is a known issue by MCV. I put in a support ticket for this problem a few weeks ago, and "They are still working on it" according to Chris. Until then I have turned off USB logging in Vera,
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 16, 2013, 02:31:16 am
Back a few posts are links to logging to NAS instead of USB. Then USB can be used for Vera logging only. This aparently solves the issues by removing the need for two USB drives. I am still to impliment but initial reading of posts seems positive.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on December 16, 2013, 07:56:09 am
Back a few posts are links to logging to NAS instead of USB. Then USB can be used for Vera logging only. This aparently solves the issues by removing the need for two USB drives. I am still to impliment but initial reading of posts seems positive.
I would love to do this, but I've run into a snag with CIFS for Vera 2. Apparently, the packages work for Vera 3 only. I could not install them.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 16, 2013, 08:34:19 am
Oh, not great news there then. Sory that's all I am aware of as a soloution untill mcv make changes. Good luck.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: RichardTSchaefer on December 16, 2013, 06:43:09 pm
I do not use datamine ... but if I did this is how I work around the problem:

1) Turn off USB logging on Vera ... This is the flaky software that's causing the problem.
2) Mount the USB disk for Data Mine ... that seems to be reliable.
3) Make a cmh directory on the Data Mine USB disk.
4) Remame your existing /tmp/log/cmh to /tmp/log/cmhtemp
5) Make /tmp/log/cmh  a symbolic link to the cmh directory on the data mine USB disk.
6) Move all of your existing log files to cmh disk on the Datamine disk.
    You may have to kill all programs writing to this area for now ... the will restart when you HARD Boot Vera.


Now you have one USB disk for Data Mine and Vera Logs.


Title: Re: New dataMine graphing and logging plugin
Post by: Freddan101 on December 18, 2013, 02:36:36 am
@freddan101

There may be a few reasons: anything happened to your browser? Do you see the same when viewing from another computer? Maybe rebooting vera might help
I've tried different browsers from different machines and rebooted Vera but I just can't load any of my saved graphs. I just get the animated, blue circle.

The DataMine interface works flawlessly otherwise.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on December 31, 2013, 08:46:22 pm
@bubaleta,  thank you for the link to your detailed guide and link to storing to nas.  I am using to store to an always on w7 machine instead of NAS and works great so far, thank you for sharing. I can now use USB for Vera logging only and other USB port for rfx without a need for a USB hub and extra USB pen drive.

Just to add a little extra detail - I too had the usb selecting issues originally so I just uninstalled the datamine app then installed it again adding in the correct parameters under the advanced tab as described in @bubaleta's post / blog. this fixed that issue.

As for the w7 part I simply created a user on my w7 machine and logged into it so it became active.  then under the main user that is always logged in I created a folder called Vera logs and simply shared this with the new user created.  I then used this user in the data mine details as would be done with a NAS.

Sent from my HTC HD2 using Tapatalk 2
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 06:56:21 am
I can't seem to get on first base with this plugin.  Formatted the USB with fat32 default partition size.  Drive works fine in PC but plugin just doesn't seem to pick it up.  SetMountPoint is /dev/sda1 for my VeraLite...
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 07:21:08 am
Hi ageis,

Doulble check mount point. Reload Vera a few times if recently installed plugin, last resort uninstall and reinstall plugin.  On USB drive no specific partition needed just format and lable entire volume.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 07:23:57 am
Hi ageis,

Doulble check mount point. Reload Vera a few times if recently installed plugin, last resort uninstall and reinstall plugin.  On USB drive no specific partition needed just format and lable entire volume.

Sent from my HTC HD2 using Tapatalk 2

I set "Store Logs on USB device" and apparently Vera configured the USB then dataMine recognized the USB.  I guess I will now uncheck that function so logs are stored at MCV with no conflict?  edit: If I uncheck "Store Logs on USB device," it no longer recognizes the USB...
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 07:44:38 am
Yes. Back a few posts is the USB conflict explained, the cifs soloution works well. Uncheck store logs for vera to USB so it works normaly then start a fresh with datamine going to its own USB. Hope this helps.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 08:33:29 am
So now I have it working again with "Store Logs on USB Device" checked.  The SetMountPoint & mountlocation variables are blank however.  Is this possibly the issue?  Do I need to specify these variables before I uncheck "Store Logs on USB Device?"
Title: Re: New dataMine graphing and logging plugin
Post by: C4Vette on January 07, 2014, 08:42:18 am
I also had a lot of problems with keeping the USB drives correct mounted. Although I know how to correctly edit the fstab and use UUID's it still went wrong after a few reboots. My problem was I wanted to use two seperate sticks, one for logging and one for datamine. I did manage to get it working, but after a few reboots the logging grabbed the wrong stick and formatted it.
Now I use just one stick with two partitions. I took a 4Gig and let the logging feature grab the first 500MB and then stuck the stick in a Linux-machine (Windows could not do anything with the unused 3.5 Gig's) and used Parted to create a second partition with FAT
I think (not sure!) that datamine doesn't 'see' the stick if it is not FAT formatted. That's why the manual says to do the formatting on a Windows-PC. The put it back in my VeraLite and datamine saw the second partition and is still using it. The Vera is not often rebooted since then a lot but it is looking good.
@Aegis:
You wrote: "I guess I will now uncheck that function so logs are stored at MCV with no conflict?"
But that's not correct, unless you mean with "MCV" your Vera. If you don't use a stick the logging uses internal memory and there isn't a lot of that available!
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 08:46:52 am
I also had a lot of problems with keeping the USB drives correct mounted. Although I know how to correctly edit the fstab and use UUID's it still went wrong after a few reboots. My problem was I wanted to use two seperate sticks, one for logging and one for datamine. I did manage to get it working, but after a few reboots the logging grabbed the wrong stick and formatted it.
Now I use just one stick with two partitions. I took a 4Gig and let the logging feature grab the first 500MB and then stuck the stick in a Linux-machine (Windows could not do anything with the unused 3.5 Gig's) and used Parted to create a second partition with FAT
I think (not sure!) that datamine doesn't 'see' the stick if it is not FAT formatted. That's why the manual says to do the formatting on a Windows-PC. The put it back in my VeraLite and datamine saw the second partition and is still using it. The Vera is not often rebooted since then a lot but it is looking good.
@Aegis:
You wrote: "I guess I will now uncheck that function so logs are stored at MCV with no conflict?"
But that's not correct, unless you mean with "MCV" your Vera. If you don't use a stick the logging uses internal memory and there isn't a lot of that available!

Thanks for the clarification.  After reading your post, I looked up the Vera Lite internal memory to discover my 4GB USB far exceeds whats in the device.  I still would like to know how I keep it working if I uncheck "Store Logs on USB Device."
Title: Re: New dataMine graphing and logging plugin
Post by: C4Vette on January 07, 2014, 08:50:51 am
So now I have it working again with "Store Logs on USB Device" checked.  The SetMountPoint & mountlocation variables are blank however.  Is this possibly the issue?  Do I need to specify these variables before I uncheck "Store Logs on USB Device?"
Sooner or later, after a reboot, the MiOS logging will steal the partition and format it  :'(
I did fill in "SetMountpoint" with /dev/sda2 (dito for 'mountlocation') but that is because it is the second partition on the same stick as the logging is. And I'm still not sure if it will be OK after more reboots!
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 08:52:51 am
So now I have it working again with "Store Logs on USB Device" checked.  The SetMountPoint & mountlocation variables are blank however.  Is this possibly the issue?  Do I need to specify these variables before I uncheck "Store Logs on USB Device?"
Sooner or later, after a reboot, the MiOS logging will steal the partition and format it  :'(
I did fill in "SetMountpoint" with /dev/sda2 but that is because it is the second partition on the same stick as the logging is. And I'm still not sure if it will be OK after more reboots!

Gotcha...wait and see.  When I filled in those variables, it stopped working.  So I will leave them blank and keep the box checked for a few days at least.   ???
Title: Re: New dataMine graphing and logging plugin
Post by: C4Vette on January 07, 2014, 08:59:15 am
If I do understand correctly you are using just one USB-stick. Right?
And you have 'Store logs on USB device' enabled. Right?
If these two are correct then all your datamine data is going to the Vera and not to the stick.
Check with the command 'mount' (via PuTTY SSH) which directory is mounted to which device.
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 09:30:59 am
If I do understand correctly you are using just one USB-stick. Right?
And you have 'Store logs on USB device' enabled. Right?
If these two are correct then all your datamine data is going to the Vera and not to the stick.
Check with the command 'mount' (via PuTTY SSH) which directory is mounted to which device.

Yes, Vera Lite only has 1 USB slot
Yes 'Store logs on USB device' enabled
Have no idea how to Check with the command 'mount' (via PuTTY SSH) which directory is mounted to which device.
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 07, 2014, 09:44:38 am
You can safely use a usb hub and multiple usb sticks. Works perfectly also after reboot
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 09:53:20 am
You can safely use a usb hub and multiple usb sticks. Works perfectly also after reboot

I have to disagree in line with the many points in this thread. Using a hub has either data mine and or Vera getting confused on reboot, maybe not every time but test show it is unreliable and deletions of data is seen. The cifs route is the way to go.

If however you have managed to sucsessfuly use of a hub, two sticks one for Vera logging and one for datamine, then please do share the settings as this would be a far easier soloution if you do in fact have it running as desired. :-)

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 07, 2014, 09:54:43 am
What do you want me to share?
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 10:21:15 am
Hi sander,

Not entirely sure but if you say you have a USB hub conected into a USB slot and two USB sticks attached respectivly to the hub, one for Vera logging and one for datamine, and this is working long term even with power cycles, then you are, I believe, the only one able to achive so would be great to know what's different with your set up compared to others.

Not trying to be facetious just keen to know why yours is working when others are not and the cifs soloution implimented as a fix.

Are you certain everything is mounting corectly each time after restart? Is data staying put after set periods of time?

Mine worked great until a reboot then needed fixing each time it rebooted, I have a nightly reboot so this was a pain.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 07, 2014, 10:46:02 am
Hi sander,

Not entirely sure but if you say you have a USB hub conected into a USB slot and two USB sticks attached respectivly to the hub, one for Vera logging and one for datamine, and this is working long term even with power cycles, then you are, I believe, the only one able to achive so would be great to know what's different with your set up compared to others.

Not trying to be facetious just keen to know why yours is working when others are not and the cifs soloution implimented as a fix.

Are you certain everything is mounting corectly each time after restart? Is data staying put after set periods of time?

Mine worked great until a reboot then needed fixing each time it rebooted, I have a nightly reboot so this was a pain.

Sent from my HTC HD2 using Tapatalk 2

100% sure. It works for over more than one year ad I reboot often. Previously on my Vera2. I recently migrated to Vera Lite and then I had no issues at all and datmine picked the corret stick directly...
So what do you want to know/see. I'll attach some screenshots.

Update:
I have 3 USB devices:
1 stick for datamine
1 stick for USB logging
1 USB cable for the dutch power meter

previously on vera2 on 2 usb ports and one hub (2 stick in the hub)
now on vera lite all 3 in the hub
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 11:15:28 am
So could it be that the issue is only seen on UI5 Vera 3?  (this is what I use and others I saw that had my problem also) so could there be a difference for Vera Lite compared to Vera 3? I am envious if it is working well for you as I spent a bit of time changing my set up to get it stable and working and although I have achieved it a simple set up with a hub would have been much easier.

So I guess out to the forum:

- do you have a hub + Vera logs on one USB stick and Datamine on another USB stick working long term and in harmony?
         -- If you do are you on a Vera Lite?

Interesting - thank you for sharing :-)
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 07, 2014, 11:17:31 am
So could it be that the issue is only seen on UI5 Vera 3?  (this is what I use and others I saw that had my problem also) so could there be a difference for Vera Lite compared to Vera 3? I am envious if it is working well for you as I spent a bit of time changing my set up to get it stable and working and although I have achieved it a simple set up with a hub would have been much easier.

So I guess out to the forum:

- do you have a hub + Vera logs on one USB stick and Datamine on another USB stick working long term and in harmony?
         -- If you do are you on a Vera Lite?

Interesting - thank you for sharing :-)
No problem, let me know if you need more information abou my setup.
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on January 07, 2014, 02:11:46 pm
It's great to hear the use of two separate USB keys has worked long term for both Vera logging and DataMine data

I had it working on my VeraLite for a month or two, but then for some reason Vera got into a mess and reformatted my DataMine data (months and months worth of data lost) and made that key the focus for USB logging.

It's for that reason - I'm so glad I finally got DataMine writing to an external share on a NAS.
I still use the hub for Current Cost energy and USB logging, but it seems more comfortable with just the one storage mount.
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 07, 2014, 02:19:47 pm
I havent read that writing to a nas is also sticky after reboot. I it is I also want to think about changing that. Is it possible to also "start" datamine directly on the nas without using the vera interface? Thus now you can start via http:// vera/dm. But you can't do that via cp.mios.com. and can then maybe be done directly on the nas... in another thread I asked for a step by step guide to get datamine running on a nas. Is that somewhere available?
Title: Re: New dataMine graphing and logging plugin
Post by: parkerc on January 07, 2014, 02:27:25 pm
SanderL

As I understand it DataMine only runs locally on Vera (when you are on the same network), your Nas is just used for storage of the config file for the captured data.

My external logging is working well and survives any reboots, I've just updated the How To - post I started. Maybe that will help you if you are looking to do it.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 07, 2014, 02:49:20 pm
There is also a gd how to by an anouther user just a handfull of posts back, its in Spanish but google tanslate fixed that ;) was what I used. Have not seen parkerc,s how to but bound to be gd; kudos to you sir.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 07:48:42 pm
This plugin continues to perplex me.  I have now had "Store Logs on USB Device" unchecked all day and it appears that the USB remains unrecognized although formatted in fat32.  However, the channels I monitor (ambient temps of 2 Tstats) show data all day long.  Where is that data being stored?  Also note the yellow triangles with exclamation points on the channels though the upper right corner green circle indicates communication/online status.
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on January 07, 2014, 09:35:56 pm
If the  \dataMine  directory has been created under root by the plugin (noting that clean new installs will fail to do this) then there are two possibilities:

a) the USB stick is not mounted or incorrectly mounted:
Data will be still be stored and will be found under the \dataMine  directory. However it will be stored in Vera's memory.

b) the USB stick is correctly mounted:
Data will be stored on the stick and will also be found under the \dataMine  directory. However, if any data was previously stored in Vera's memory, before the stick was mounted in the dataMine directory, it will still be there but hidden. Mounting the stick hides/overrides any existing files located in Vera's memory in the mount directory.

This can be determined by looking at the memory size available as seen on the "dataMine" tab. A correct installation will, for say a 8GB stick, show:  "Disk Space Total (kb) 7,812,864".

If you using Vera memory then you will see figures that are about one thousand of this, as per image above, that only shows "11,264".

Some people use another computer to put a \dataMine directory on their stick and then try and mount it. This is incorrect but may work.

Users should always look at the dataMine's own error log. If it says there is an error then there probably is.

For those doing a brand new install this works:
http://forum.micasaverde.com/index.php/topic,7750.msg134271.html#msg134271

All of the above becomes more problematic when users try to log to the USB stick as well - for reasons not fully understood?

Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 10:10:38 pm
If the  \dataMine  directory has been created under root by the plugin (noting that clean new installs will fail to do this) then there are two possibilities:  I don't think the folder was ever created by the plugin.  I labeled it when I formatted the USB stick, but it was never recognized, so I checked "Store Logs on USB Device" and  the label was wiped.

a) the USB stick is not mounted or incorrectly mounted:
Data will be still be stored and will be found under the \dataMine  directory. However it will be stored in Vera's memory.  How would you access Vera's memory to verify?

b) the USB stick is correctly mounted:
Data will be stored on the stick and will also be found under the \dataMine  directory. However, if any data was previously stored in Vera's memory, before the stick was mounted in the dataMine directory, it will still be there but hidden. Mounting the stick hides/overrides any existing files located in Vera's memory in the mount directory.  How do you mount the stick?

This can be determined by looking at the memory size available as seen on the "dataMine" tab. A correct installation will, for say a 8GB stick, show:  "Disk Space Total (kb) 7,812,864".

If you using Vera memory then you will see figures that are about one thousand of this, as per image above, that only shows "11,264".

Some people use another computer to put a \dataMine directory on their stick and then try and mount it. This is incorrect but may work.  How do you get the plugin to create the directory?

Users should always look at the dataMine's own error log. If it says there is an error then there probably is.How do you access error log?

For those doing a brand new install this works:
http://forum.micasaverde.com/index.php/topic,7750.msg134271.html#msg134271

All of the above becomes more problematic when users try to log to the USB stick as well - for reasons not fully understood?

Sorry for all the questions, I lack some of the basic foundational principles of this stuff...namely Linux I guess...just downloaded puTTy...babysteps
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on January 07, 2014, 10:42:42 pm
I labeled it when I formatted the USB stick, but it was never recognized, so I checked "Store Logs on USB Device" and  the label was wiped.
It appears that "Store Logs on USB Device" formats the stick - that's a big problem.

How would you access Vera's memory to verify?
A lot of users download SCP and use that to login into Vera and then manipulate the files therein:
http://winscp.net/eng/download.php

How do you mount the stick?
Normally the plugin does this, if it finds the dataMine directory to mount to. Review the mount command in any Linux doco. You can use SCP to do it manually.

How do you get the plugin to create the directory?
New installs don't - that's the current problem. You can use SCP to do it manually.

How do you access error log?
Normally that's available in UI5 on dataMine's device page on the dataMine tab labelled "dataMine debug page". To see all logs you can install the infoViewer plugin:
http://forum.micasaverde.com/index.php/topic,13477.0.html
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 07, 2014, 11:31:20 pm
PuTTy and this...http://forum.micasaverde.com/index.php/topic,13512.0.html

Seems to be working.
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 08, 2014, 06:37:37 am
Maybe not...
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 08, 2014, 09:08:10 am
Thanks... that seems to be related to the problem...

I ended up doing the following....

1. Removed plugin and USB stick.
2. Powered off
3. Inserted USB stick
4. Powered on
5. Manually created dataMine folder in root directory
6. Installed plugin
7. Launched GUI and set UUID

Because I actually backed up the contents of the USB stick before all of the troubleshooting, I copied back the config and 1 year of associated data.  After completing step 7 above, the 20 channels are back!

Either way, it looks like something related to the dataMine directory.  Manually creating it before installing seemed to do the trick on my end.

Thanks for the help.

I will try this. At what point in your process did u copy back the configuration?
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 09, 2014, 08:27:07 am
If anyone can look at this debug file to review the errors, I would greatly appreciate it.  It just seems like the program doesn't find the USB, however, it seems to work for an hour or so after I set it up.  I set the UUID using PuTTy and the blkid command.  I populate the SetMountPoint with /dev/sda1 for the Vera Lite.  I labeled the drive dataMine in Win7.  I tried to label it while plugged into the Vera Lite and it said it was already created.  The only thing I wonder about is when the drive is plugged into the Vera, when you mkdir, how do you ensure it is making the directory on the USB and not on the Vera?

The times and dates on the USB files make no sense.  They haven't even occurred yet.
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on January 09, 2014, 05:10:41 pm
The report looks far improved indicating you have now successfully mounted a 4GB stick.

"how do you ensure it is making the directory on the USB " That why in the instructions I linked to:  it said, create the directory when the stick is unplugged. Likewise you can't create a directory on it, if it's not mounted. To be sure - plug it into another PC and check there is no directory. From the log however it all looks OK.

"The times and dates on the USB files make no sense". What do you mean here - they look OK.

If dataMine continually stops or restarts it may be something you are trying to log causing problems. I would stop logging all and add them back in one by one.
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 10, 2014, 06:53:27 am
The report looks far improved indicating you have now successfully mounted a 4GB stick.

"how do you ensure it is making the directory on the USB " That why in the instructions I linked to:  it said, create the directory when the stick is unplugged. Likewise you can't create a directory on it, if it's not mounted. To be sure - plug it into another PC and check there is no directory. From the log however it all looks OK.

"The times and dates on the USB files make no sense". What do you mean here - they look OK.

If dataMine continually stops or restarts it may be something you are trying to log causing problems. I would stop logging all and add them back in one by one.


Times and dates were actually in the future to when the activity actually occurred.

I'm monitoring a couple Tstats where 4 variables (temp & modestate) haven't changed for days, so maybe that is the issue.  Thank you.
Title: Re: New dataMine graphing and logging plugin
Post by: Aegis on January 12, 2014, 02:56:30 pm
I believe this plugin is in fact working for me now.  My confusion was over the loss of up-to-date data.  What made me take a deeper look was the dataMine Overview pdf file.  I read this file several times, but what I finally realized today was that the images even show the icons for tick, yellow exclamation & red exclamation for several channels.  I was taking all my red exclamations as complete loss of communication, when in fact I believe that if the device (thermostats in my case) do not change state over a period of time, no new information is sent.  So a couple furnaces running non stop during the deep freeze will appear to have "lost communications."  I uninstalled everything today, deleted all dataMine directories then reinstalled.  All 4 channels I set up would not show anything but red exclamation for quite some time.  I then changed the setting on the thermostat and the channel immediately registered fresh data with a tick mark indicator.

I should add a couple notes of things that I came to understand despite my best efforts at comprehension of what is already documented and what people told me:

1) You format the USB stick and lable it dataMine.  You do not have to also create a folder on it called dataMine.
2) If the plugin doesn't create a dataMine folder on Vera, you have to create it.  You can do this with PuTTy or SCP.  It's easier to verify its location in SCP as that is Windows based.  When I created this folder in root with PuTTy it didn't work.  When I created it in /<root> (after clicking the up arrow) in SCP, it worked.  I don't know the difference between the two locations.
Title: Re: New dataMine graphing and logging plugin
Post by: C4Vette on January 14, 2014, 05:33:58 am
I should add a couple notes of things that I came to understand despite my best efforts at comprehension of what is already documented and what people told me:

1) You format the USB stick and lable it dataMine.  You do not have to also create a folder on it called dataMine.
2) If the plugin doesn't create a dataMine folder on Vera, you have to create it.  You can do this with PuTTy or SCP.  It's easier to verify its location in SCP as that is Windows based.  When I created this folder in root with PuTTy it didn't work.  When I created it in /<root> (after clicking the up arrow) in SCP, it worked.  I don't know the difference between the two locations.

note 1: Labeling is optional I discovered.
Note 2: "/" is the top-level of the directory-structure
            "/root" is a subdirectory used as the home-dir for user 'root'.
Title: Re: New dataMine graphing and logging plugin
Post by: bore on January 15, 2014, 11:21:40 am
Hi,
I need a help...
Here we have 2 costs per kWh...
Between 6:00 and 22:00 the cost is 0,09255 EUR and between 22:00 - 6:00 is 0,05803 EUR...
I have an AEON Home Energy Monitor and I have the real consumption in my house in kWh, but I want to know what i need to pay...
Can you help me, please

Title: Re: New dataMine graphing and logging plugin
Post by: RichardTSchaefer on January 15, 2014, 12:27:07 pm
I do not have an AEON home energy monitor.
If there is a variable in the ADVANCED tab of the monitor with cost per KWh then you can do a simple PLEG (or use LUA code if you wish)

PLEG:
DaySchedule:     ON  Sunday-Saturday at 6:00  Off  Interval 16:00:00

Condition:
RATE                 DaySchedule ? 0.99255 : 0.05803

Action:
RATE                 SetVariable   ServiceID, VariableName, DeviceID, Value

You need to use the Advanced Tab of actions  to set the Rate. If they have an action to set the rate ... you should use that.
Otherwise you might also have to force a LUA restart to cause the Monitor to use the new variable value. 
Setting up the action will take a little trial and error to sort out the strategy.  PM me if you decide to proceed here.




Title: Re: New dataMine graphing and logging plugin
Post by: bore on January 15, 2014, 01:07:18 pm
AEON has only kWh and they are from the first day that I insert it...
No variable for the costs...
May be I need to learn PLEG...
10x

Title: Re: New dataMine graphing and logging plugin
Post by: RichardTSchaefer on January 15, 2014, 03:19:46 pm
Actually ... you could have PLEG generate and update a variable that is KWh * Cost ... that would get updated every time the KWh value was updated.
Title: Re: New dataMine graphing and logging plugin
Post by: axbl on January 20, 2014, 06:20:12 am
Hi,
is it possible to integrate the stored graphs directly into the UI5 ?
Title: Re: New dataMine graphing and logging plugin
Post by: sle118 on January 21, 2014, 04:54:45 pm
I have 2 USB sticks on my USB hub, and after I unplugged a USB to serial adapter for a CM11A X10 device and rebooted, my Vera Lite decided to use the 2nd usb stick as the log storage area.

root@MiOS_35016391:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/mtdblock7 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/sda1 on /tmp/log/cmh type ext3 (rw,noatime,nodiratime,errors=continue,barrier=0,data=ordered)
/dev/mtdblock8 on /mios type squashfs (ro,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
/dev/sda1 on /dataMine type ext3 (rw,relatime,errors=continue,barrier=0,data=ordered)

Luckily, the device had already been formatted so I did not loose any of my dataMine data.  The issue, however, is that now the logs as well as the dataMine database are stored in the same USB stick, and I might just loose the data at some point in time when the Vera decides it's time to format it.

I cold swear that /dev/sda1 was the vera log stick and /dev/sdb1 the dataMine stick. Somewhere along the line, though, it seems the OS actually reversed them after a reboot. This happened even if I did not move the sticks in different ports.

Does anyone have an explanation yet?

P.S. am I the only one unable to identify the verification letters to post on this forum in a single attempt??
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 21, 2014, 05:33:11 pm
Hi @sle118,

Go back a few posts to see fix. 2 USB sticks has issues as you found.  Soloution is on using ciffs and nas.

Sent from my HTC HD2 using Tapatalk 2

Title: Re: New dataMine graphing and logging plugin
Post by: sle118 on January 22, 2014, 08:00:26 am
Go back a few posts to see fix. 2 USB sticks has issues as you found.  Soloution is on using ciffs and nas.

Thanks for the quick reply.

I thought I could just edit fstab to mount by UUIDs rather than by device, but after a quick search I found out that this solution would also eventually fail. 

So basically, as I understand it, the issue is that multiple usb sticks on a hub (I have 2 of the same brand/model) are not always detected in the same order by the OS and this means one stick may appear as /dev/sda at one time, and /dev/sdb at a different time after a reboot or after another device was added to/removed from the hub (for example a USB/Serial adapter).

I'll leave a link to a forum topic that discuss this for others having the same issue
http://forum.micasaverde.com/index.php/topic,16452.0.html
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on January 22, 2014, 08:13:14 am
I think its fair to say that's correct.  Some have had things working for some time and some not but all in all its safe to say that it seems unstable on such a solution.  the ciffs / Nas worjk around appears to work well long term.

good luck :-)
Title: Re: New dataMine graphing and logging plugin
Post by: alsilco on January 30, 2014, 06:12:26 pm
Hiya Lights and other gurus!
You seem to be a verrry knowledgeable guys and I was hoping you could shed some light on my Datamine Bug.  Vera support Configured DM and it was working for 12 days, then suddenly stopped 4 days ago.  It was logging perfectly onto a USB Drive in the top slot, even with Vera's developed TemperatureFix app on 3x Aeon 4in1 Multisensor, the 3M CT50 Wifi Thermostat, and Weatherunderground's World Weather App. Vera's Tech's got DM up again, (sensors showing last updated as green), but all graphs are showing as Flat Lines since Jan 27th). Have scoured the forums, and see about USB issues, but I have SCP'ed into the Vera, and do not see any current dates, only the earlier ones. I just got my Vera3, and am surprised I have gotten this far!  Please find attached a screen shot from DM, after it stopped 2 days ago, showing online status as grey, but currently it shows red, and says "Vera is online, however Datamine has errors (error count 146).  When I have uninstalled and reinstalled DM, the errors go away, then start climbing again as the data is collected. Now have the Flash Drive in the lower USB, and still having exactly the same problem. Was logging mostly Temps & Humidity for an engineering project, and I'm now back at square Zero!!   Please Help if you can!! Thanx!!! Al
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on January 31, 2014, 03:42:22 am
Look at "datamine cifs" on this forum
Title: Re: New dataMine graphing and logging plugin
Post by: alsilco on January 31, 2014, 11:56:12 am
Look at "datamine cifs" on this forum

SanderL, Saw the cifs posts, but was hoping to just get back to logging data at present. Here are a few lines from my Datamine log:

50   01/31/14 10:58:46.944   luup_log:63: dataMine: Unable to open file for write /dataMine/database/24/raw/2300.txt <0x2ca41680>
50   01/31/14 10:58:46.944   luup_log:63: dataMine: Error: '/dataMine/database/24/raw/2300.txt: Read-only file system' <0x2ca41680>
06   01/31/14 10:58:46.944   Device_Variable::m_szValue_set device: 59 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ca41680>
50   01/31/14 10:58:46.946   luup_log:63: dataMine: Logging error logging variable 62::urn:micasaverde-com:serviceId:HumiditySensor1::CurrentLevel <0x2ca41680>
50   01/31/14 10:58:46.947   luup_log:63: dataMine: Unable to open file for write /dataMine/database/25/raw/2300.txt <0x2ca41680>
50   01/31/14 10:58:46.963   luup_log:63: dataMine: Error: '/dataMine/database/25/raw/2300.txt: Read-only file system' <0x2ca41680>
06   01/31/14 10:58:46.963   Device_Variable::m_szValue_set device: 62 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2ca41680>
50   01/31/14 11:00:03.442   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt <0x2cc41680>
50   01/31/14 11:01:32.957   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt __LEAK__ this:151552 start:2256896 to 0x1143000 <0x2cc41680>
50   01/31/14 11:01:32.969   luup_log:63: dataMine: Logging error logging variable 54::urn:upnp-org:serviceId:TemperatureSensor1::CurrentTemperature <0x2b641680>
50   01/31/14 11:01:32.969   luup_log:63: dataMine: Unable to open file for write /dataMine/database/10/raw/2300.txt <0x2b641680>
50   01/31/14 11:01:32.970   luup_log:63: dataMine: Error: '/dataMine/database/10/raw/2300.txt: Read-only file system' <0x2b641680>
06   01/31/14 11:01:32.970   Device_Variable::m_szValue_set device: 54 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: true #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2b641680>
50   01/31/14 11:02:50.397   luup_log:63: dataMine: Unable to open file for write /dataMine/Notifications [R2299].txt <0x2cc41680>

I SCP'ed into the Vera3 and tried to change the permissions on the "Notifications [R2299].txt" and the "2300.txt" files, but was denied in both cases.  See attached photos.

Are these files supposed to be read only?
Why is DM trying to write to read only files?
Do you know to fix this ??
Thanks for your help!
Al
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on January 31, 2014, 03:56:22 pm
First - R2229 File permissions.jpg shows the correct permissions for this file and yes dataMine writes to these files. These particular files store the notifications and associated time stamps.

The operating system will sometimes change a device to read only if it encountered an error with it eg bad sectors, file system corruption, power fail during write, stick pulled out of box while in operation, etc.

If you execute, using SCP:

   cat /proc/mounts

you will see a line with dataMine mentioned. It should show rw, not ro.

If the latter, you can try a full power on/off of Vera and recheck. If that fails - suggest you back up all the files on the stick and copy them over to a new one and retry. Alternatively reformat existing stick and copy files back on to that and retry.

Why this might have occurred is unclear and the suggested fix is not guaranteed.
Title: Re: New dataMine graphing and logging plugin
Post by: alsilco on January 31, 2014, 04:26:20 pm
A-Lurker,
The Cat /proc/mounts yields 14 lines, and yes, the last one is Datamine:

/dev/sda1 /dataMine vfat ro,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

I've seen lots of controversy re: corrupted USB drives getting corrupted, so do I:
1. just yank the power from the Vera3, or is there a better way to power down the unit? (Like is there a way to power down from the UI5 GUI?)
2. Is there a right way to unplug the Flash drive? Do I need to somehow unmount the Flash drive? (I do not need to remove Datamine App to remove the Flash drive, do I?)

Thanks so much,
Al
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on January 31, 2014, 05:59:32 pm
Have a look here:

http://forum.micasaverde.com/index.php/topic,12395.msg96211.html#msg96211

and no need to uninstall the plugin. Make sure you backup everything using SCP first!
Title: Re: New dataMine graphing and logging plugin
Post by: mntlvr on February 15, 2014, 10:32:29 am
Chris
Can you give me some information as to how you have this plugin configure itself to a Temperature sensor or a power meter.
When I set the "parent ID" to my honeywell thermostat ,whose  ID is "108" and vera3 version 1..5.622, then save and reload the plugin completely disappears. When I go to "APPS" and left click the datamine app, it show no installed plugin that it created earlier. So perhaps I am setting up the "parentID" incorrectly.


 [/quote author=Chris link=topic=7750.msg49749#msg49749 date=1315676792]
Update 6 July 2012 - UI5
dataMine on UI5 should now have a seemless install. I've also started an overview document that I'll try and keep up to date with information rather than expecting everyone to sift through what is now a long thread...

To install -:

Once you've installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I'm not 100% sure. I've been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I'm told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.

Note that this should not be used with Veras USB logging enabled. If you have USB logging enabled, you should use a separate USB stick - don't be tempted to use the same drive as this can cause problems.

Original Post
I like the Vera UI - it's clean and simple (although I hope UI5 improves on this). As others have found though, reading logs to get sensor history is not useful. I also am not a fan of the model of hosting data on a separate website - I really like to have my data under my control. This is mainly for security and communication tollerance reasons.
 
I've seen other plugins that allow data to be logged, and then either use a script to produce graphs, or use an external server to process and visualise the data. This is quite possibly a more powerful solution, but I wanted the simplicity of a single unit, and the flexibility of on the fly graphing rather than fixed graphs processed daily in a script.
 
So... the dataMine plugin is born.  This logs all value changes in specified variables to log files on Vera, and provides a GUI graphing interface complete with zoom, scroll etc. The system really must be used with an external USB memory stick since it does produce a relatively large amount of data. I've mounted a 4GB flash drive (FAT32) as /dataMine, and all data files are logged here. I estimate that a 4GB stick would support the system for 10 years or more, so memory isn't really an issue.
 
In addition to the "raw" data logging, the system also logs hourly min/max, and daily min/max. This data is used to support different zoom levels so that the large raw files don't need to be heavily processed to see long timeframes. The system does work with large raw data sets, but it starts to slow down (taking maybe 10 seconds or so to return a 1 year history of temperature data). The creation of these files is still work in progress - currently only the hourly log is generated, and then only manually.
 
Installation
As I've made use of a number of javascript libraries (primarily the flotr graphing package) installation isn't quite as straight forward as a standard plugin (sorry). The usual files (XML, JSON, LUA) are loaded in the normal way through the "developpers" button in the control panel and all files specific to dataMine are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. The config file needs to be stored in another directory /dataMine, which is also where all the data files are stored.  The ZIP file also includes a set of files with approximately 1 year history of outside temperature at my house to allow you to play with the system - these can of course be deleted. There are basic installation instructions in the readme.txt file. (I had to trim down the data due to ZIP size limitations on the server!).

Use
The dataMine plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse.
The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren't hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There's also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon).
 
Limitations

This is still very much work in progress (consider this an alpha version that needs further work). Feel free to give it a go and send any feedback and I'll try and fix any problems. It's worth noting that I've only had my Vera a couple of weeks, so I'm very new to it, and the Lua/Luup system, so I've probably screwed things up, or at least poorly written parts of the code, so go easy on me :) (and maybe a "use at own risk" warning is in force!).

---
A wiki, and the latest code, is available at http://code.mios.com/trac/mios_datamine (http://code.mios.com/trac/mios_datamine).
---

Cheers
Chris

---
Attached ZIP file for UI4: 18-Jan-2012
---
[/quote]
Title: Re: New dataMine graphing and logging plugin
Post by: erkme73 on February 22, 2014, 02:36:47 am
Just installed the plugin, set the mountpoint to /dev/sda1.  I was not prompted to use specific USB drive when I ran the graphing page link.

However, right out the gate, the number of channels shows zero.  The graphing page has nothing listed.  It's green (connected) and no errors, but I can't see anything...

I hate being such a noob, but all of the 'fixes' I've read presume the reader has more experience than I do.  Is there something obvious I'm missing?  Running Vera Lite, UI5 1.5.622



Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on February 22, 2014, 02:59:02 am
Looks OK - indicates you have 512 mbyte stick installed. In your 1st image above, you need to click on "Configuration" tab (top left) and select something to graph. It's a tree type structure - just drill down to the sensors of interest on the left side and enable them on the right hand side. Don't forget to save your selections.
Title: Re: New dataMine graphing and logging plugin
Post by: tiwas on March 04, 2014, 05:21:58 pm
Will this work with a passive USB hub or will it need to be powered?

Cheers,

Lars
Title: Re: New dataMine graphing and logging plugin
Post by: Sender on March 05, 2014, 02:11:17 am
Try it. But with me before I went to CIFS it worked
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on April 27, 2014, 04:41:45 pm
Hi all,  I am noticing that every so often my data is lost. I use cifs set up and all works fine. All data colected remains on the cifs but every so often it loses conection to it, however all still works so I can open the datamine page and set items to monitor again so its like it resets but info on the cifs remains it just disconects from it.

Any ideas whats going on?

Sent from my GT-I9300 using Tapatalk

Title: Re: New dataMine graphing and logging plugin
Post by: sabolcik on April 29, 2014, 01:24:30 am
I'm been trying to get the plugin running for the last couple of hours without any success.  Hoping that someone can assist me with getting this resolved as I'm very excited to try out the graphing.  It looks very useful.

I installed a 500MB USB stick both through a USB hub and directly to the VeraLite (note that I am running UI5).  In both cases it shows up via dmesg as being installed.

I tried configuring the setmountpoint variable to /dev/sda1 and also let the graphing startup screen prompt me for the volume name and set it for me (it also chose /dev/sda1 as the mount point).  Restarting vera didn't cause the variable mountlocation and mounttype to get updated, which I think is supposed to happen when a proper install launches.

Checking the debug panel I get the following error message numerous times - luup_log:155: dataMine: Error reading tmpfile during mountcheck <0x2bfd9680>.

At the end of the logfile I get the following information on the disks:

Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/mtdblock0 doesn't contain a valid partition table

Disk /dev/sda: 524 MB, 524288000 bytes
17 heads, 59 sectors/track, 1020 cylinders
Units = cylinders of 1003 * 512 = 513536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1   *           1        1019      511028   b Win95 FAT32
/dev/sda2            1020        1020         501+ 83 Linux
-5-===========================================================================
SetDataDirectory-/dataMine/
SetMountUUID    -F6F4-DEE5
SetMountPoint   -
SetManualMount  -0
-6-===========================================================================
{"dbVersion":2,"Events":{"count":0,"last":0},"Version":"0.980","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================

I'm not sure if the "doesn't contain a valid partition table" message is targeted at my USB drive.

I have attached the complete log file from datamine.

Can anyone help?
Title: Re: New dataMine graphing and logging plugin
Post by: sabolcik on April 29, 2014, 02:01:58 am
Very strange - I swapped out the USB drive for another one that I tried earlier.  This time when I clicked on the graph tab it showed memory was available but much less than I have on the USB drive (a few 100KB instead of 16GB).  I assumed that  meant that  I was using the vera internal memory for logging.  I poked around some more then when I got back to the datamine device it now showed 16GB.  I reinstalled my USB hub and it still worked.  As far as I know I didn't change anything with the configuration but it just started working.

Weird.  Not sure if anyone can learn anything from my heartache, but I'll let you know if this keeps working and is stable.

Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on April 29, 2014, 05:16:13 am
Hi sabolcik,

Yes I found a similar thing historically - there are some posts regarding such issues - or at least similar issues.  the system is quite picky with the type of USB stick and format of it.  I had some that worked all the time - some that worked not at all - and some that worked then broke.

I moved to full cifs and has been a huge improvement other than I issue I am having mentioned above.  all the data still in the cifs - just that data mine no longer connects to it?
Title: Re: New dataMine graphing and logging plugin
Post by: sabolcik on April 29, 2014, 01:11:37 pm
LightsOn:

Thanks for the feedback.  After messing around with some different configurations, it looks like a combination of issues bit me. 

1) When moving the USB stick around the file system on it was corrupted.  Reformatting the USB stick and power cycling the VERA seems to have addressed that issue
2) My USB hub was apparently flaky with the USB drive.  I have been using it with the AD2USB successfully but the USB drive seems unhappy.  I switched out to a new USB hub and now it appears to be working.

Now I just need to figure out how to backup my data to my NAS.  I saw some discussions on that topic in this thread but will defer tackling that until I convince myself that the data logging is solid and I'm using it regularly.

Thanks for your help!
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on May 29, 2014, 05:28:09 am
@Chris,
Awesome plugin. Kudos to you.

Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.

Unfortunately this won't work  :(  MCV use a reverse link to connect to theirs server to avoid NAT issues in firewalls and DNS problems.  Remote access isn't impossible to configure, but it requires setting up a bunch of stuff that's not so simple (Dynamic DNS and NAT in your firewall)...

Chris

Chris

Hi Chris,

I am familiar with NAT and DYNDNS and would like the option to work with Datamine remotely - are there any pointers anywhere as to how to get started or is it just to much of a pain to sort?  if it is possible and there is any documents / instructions form others you know if I would be great full for the pointers if at all possible?

Thank you.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on May 29, 2014, 02:35:03 pm
Chris is no longer on the forum nor is datamine being actively developed. So support will be very limited.

- Garrett

Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on May 29, 2014, 05:21:23 pm
Hi garrettwp

Thank you for the heads up. Thats a shame as like datamine a lot.

Sent from my GT-I9300 using Tapatalk

Title: Re: New dataMine graphing and logging plugin
Post by: akbooer on May 29, 2014, 06:49:42 pm
You might, perhaps, give DataYours a go instead.  What functionality are you after, particularly? Under UI5, DataYours can serve graphs over the MiOS servers (or email them to you.). It can also read dataMine files.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on May 30, 2014, 02:14:59 am
Hi akbooer,

impressive work - thank you  :)

This is still in BETA if i understand things correctly so may hold out for a little while. (bet you i end up having a play before hand though!)

My main desire are to be able to see graphs on data from devices such as temp against heating on/off etc.  i also want to ensure i can store this off vera like via a cifs in datamin and i understand possible but as you have stated in a post still a "Thorny" issue.

more reading i think but looks great stuff - thank you :-)
Title: Re: New dataMine graphing and logging plugin
Post by: akbooer on May 30, 2014, 02:31:51 am
This is still in BETA if i understand things correctly so may hold out for a little while. (bet you i end up having a play before hand though!)
It is still beta, but very stable.  I have been using it in "production" mode for over three months gathering data across three separate machines.

Quote
My main desire are to be able to see graphs on data from devices such as temp against heating on/off etc.  i also want to ensure i can store this off vera like via a cifs in datamin and i understand possible but as you have stated in a post still a "Thorny" issue.
It's only the USB handling which always seemed to cause problems for dataMine users.  I simply remain USB agnostic - all DataYours needs is a (single) directory in which to store database files and it provides absolutely no functionality to mount devices.  Personally, I've been using CIFS to an external NAS for perhaps six months now and it's absolutely perfect.  I did separately provide a little bit of Luup Lua startup code to mount that.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on May 30, 2014, 02:46:44 am
Quote
It's only the USB handling which always seemed to cause problems for dataMine users.  I simply remain USB agnostic - all DataYours needs is a (single) directory in which to store database files and it provides absolutely no functionality to mount devices.  Personally, I've been using CIFS to an external NAS for perhaps six months now and it's absolutely perfect.  I did separately provide a little bit of Luup Lua startup code to mount that.

So I could go back to using a large USB on a hub for both vera logging and also datayours?  i think i like the idea of keeping separate though but good to know.  if separate and i understand you correctly i can point datayours at the same cifs set up as i did in data mine - sounds like you do this also? 
Title: Re: New dataMine graphing and logging plugin
Post by: akbooer on May 30, 2014, 02:52:41 am
So I could go back to using a large USB on a hub for both vera logging and also datayours? 
I do not have the expertise to advise on use of USB storage.

Quote
i think i like the idea of keeping separate though but good to know.  if separate and i understand you correctly i can point datayours at the same cifs set up as i did in data mine - sounds like you do this also?
Yes, that's exactly what I do.  In fact, for a while, I ran both together and then, later, just used dataMine to mount the CIFS and storing the Whisper database in a sub-directory.  In the end, I wrote a separate snippet of code to do the mount and removed dataMine.

DataYours can plot data from dataMine and its own database simultaneously, which was helpful in the transition period.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on May 30, 2014, 03:03:07 am
Quote
Yes, that's exactly what I do.  In fact, for a while, I ran both together and then, later, just used dataMine to mount the CIFS and storing the Whisper database in a sub-directory.  In the end, I wrote a separate snippet of code to do the mount and removed dataMine.

DataYours can plot data from dataMine and its own database simultaneously, which was helpful in the transition period.

Brilliant thank you - I shall look into this.
Title: Re: New dataMine graphing and logging plugin
Post by: tommysharp on June 25, 2014, 01:13:25 am
Can't seem to work out how to search just this topic..... Anyone know the specific mount point setting for the vera lite? I tried the suggestion in the first post but it doesn't seem to work. WHen I load up the GUI it asks me to select the USB stick but none are listed.
Title: Re: New dataMine graphing and logging plugin
Post by: dpackham on July 02, 2014, 12:03:18 am
So one thing i need a bit of clarification on  is...  I have an Android app that ImperaHome that says it only shows graphs with DataMine...  can this also work with DataYours?
Title: Re: New dataMine graphing and logging plugin
Post by: akbooer on July 02, 2014, 03:05:52 am
Not at the moment, but if enough people start asking then probably yes.

DataYours conforms to a well-documented 'industry standard' interface for retrieving data, so it should be straight-forward.
Title: Re: New dataMine graphing and logging plugin
Post by: brettow on July 17, 2014, 01:05:01 am
Not at the moment, but if enough people start asking then probably yes.

DataYours conforms to a well-documented 'industry standard' interface for retrieving data, so it should be straight-forward.

+1 for DataYours
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on July 17, 2014, 02:57:38 am
Also +1 for datayours2 great app.  Remote viewing of graphs via app orbotherwise would be great. :-)
Title: Re: New dataMine graphing and logging plugin
Post by: akbooer on July 17, 2014, 03:44:22 am
You can already, under UI5, access DataYours graphs remotely.
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on July 17, 2014, 04:41:04 am
Arrr yes - just brilliant! :-)

Replace ip and port with https://fwdNUMBER_OF_FORWARD_SERVER.mios.com

datayours2 keeps on giving :-)
Title: Re: New dataMine graphing and logging plugin
Post by: johnes on October 07, 2014, 06:50:12 pm
is the graph data being stored on the mios.com server, or just accessible from the IP?
Title: Re: New dataMine graphing and logging plugin
Post by: LightsOn on October 08, 2014, 08:54:57 am
Just assessable form the IP.  Mine is stored locally. many use NAS. :-)
Title: Re: New dataMine graphing and logging plugin
Post by: tomtcom on November 20, 2014, 02:51:48 pm
Hello, I'm interested in this plugin and using it in Imperihome. I'm now on UI7 and I only have 1 device which is a plug for 1 lamp and it's not run often.

Is the USB stick a must?
Is there an auto delete or max storage option?

Does this still work on UI7?
Title: Re: New dataMine graphing and logging plugin
Post by: hugopinto7 on December 03, 2014, 06:49:54 pm
Hi LightsOn!
Did not understand how you managed to access to datamine remotly. Can you be more explicit?
thanks
Title: Re: New dataMine graphing and logging plugin
Post by: elordude on December 22, 2014, 04:56:38 pm
You need to have your vera port forwarded to the outside world - if you have a ddns service you dont have to memorize your IP address. I recommend locking locking you vera with a strong password in order to do this.  I use microsoft dns service because it is free if you own a windows homeserver license.

Vera address will be
http://xxxx.homeserver.com:ABCD/cmh/

For datamine it will be
http://xxxx.homeserver.com:ABCD/dm/index.html

regards
harry
Title: Re: New dataMine graphing and logging plugin
Post by: chocorem on January 01, 2015, 09:00:07 am
Hello,

I'm just struggling organizing all my data variables... I was wondering if it is possible to add a variable to an existing Graph ?
Title: Re: New dataMine graphing and logging plugin
Post by: Otje on January 01, 2015, 04:55:27 pm
Hi Chocorem

not sure if i understood your question correctly but i assume you have a saved graph existing, and you want to add/del signals to that chart?

that is not possible.
you have to save a new chart.
Title: Re: New dataMine graphing and logging plugin
Post by: DeltaNu1142 on January 02, 2015, 08:31:37 am

Yes, that's exactly what I do.  In fact, for a while, I ran both together and then, later, just used dataMine to mount the CIFS and storing the Whisper database in a sub-directory.  In the end, I wrote a separate snippet of code to do the mount and removed dataMine.
I need to look into this.

I have a Synology NAS and was using DataMine on my Vera 2... but I couldn't save data to the NAS because the cifs package wasn't compatible with Vera 2. Now I'm on a Vera 3, but I haven't used DataMine in quite a while. Maybe I'll give DataYours a whirl this weekend.
Title: Re: New dataMine graphing and logging plugin
Post by: AndreasE on January 13, 2015, 11:31:49 am
Is the "dataMine graphing page" link supposed to be functional via a remote connection? I'm getting a 404 Not Found.
Unfortunately this won't work  :(  MCV use a reverse link to connect to theirs server to avoid NAT issues in firewalls and DNS problems.  Remote access isn't impossible to configure, but it requires setting up a bunch of stuff that's not so simple (Dynamic DNS and NAT in your firewall)...

Is there someone who has managed to configure dataMine for remote management under UI7?
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on January 13, 2015, 12:07:38 pm
Quote

Is there someone who has managed to configure dataMine for remote management under UI7?

Datamine has issues under ui7, I would recommend using datayours2 which works better and is easier to set up as well. I switched to it and am very happy!

Hans
Title: Re: New dataMine graphing and logging plugin
Post by: AndreasE on January 13, 2015, 12:28:45 pm
Datamine has issues under ui7, I would recommend using datayours2 which works better and is easier to set up as well. I switched to it and am very happy!

But do you run it remotely?
Title: Re: New dataMine graphing and logging plugin
Post by: HansW on January 15, 2015, 04:44:06 pm
Quote
But do you run it remotely?

Well you can easily if you configure your router to open the port and donot mind about security too much...

Hans
Title: Re: New dataMine graphing and logging plugin
Post by: jwwilhelm on February 04, 2015, 10:03:41 pm
Has anyone had any luck getting dataMine up and going on a Vera Edge UI7?  After installing it, and navigating to the GUI, I get the prompt to select the USB drive to target; however, the drop down is empty.  I've tried most of the tips contained in this thread that appear to have worked with previous versions, but have not been able to get the plugin to recognize the USB drive.  Any other suggestions would be appreciated.
Title: Re: New dataMine graphing and logging plugin
Post by: garrettwp on February 04, 2015, 10:35:04 pm
Until the last day or so, the Vera edge had a bug which caused the USB port not to function. The latest firmware which was just released should fix that. If you have not upgraded the firmware in the last day, you'll hit this issue.

- Garrett

Title: Re: New dataMine graphing and logging plugin
Post by: dmckenna on February 05, 2015, 07:51:17 am
Firmware fix still 'in test' according to Marc S @ Vera

@ 5th Feb
Title: Re: New dataMine graphing and logging plugin
Post by: jjwalker on March 07, 2015, 02:00:28 pm
I've been using dataMine with USB logging for a couple weeks with no issues.  They are both mounted on separate USB sticks.  The only issue I have is if the VeraLite reboots, it will detect the dataMine USB stick and reformat it for USB logging.  It's connected to a powered hub with three things connected. USB logging stick, dataMine USB stick, and an Arduino gateway.  I've experimented with the reformat issue by unplugging the Vera, then unplugging the dataMine stick.  Once Vera is booted up, and I see zero channels in the dataMine device, I plug the dataMine USB stick back in and hit Reload in UI5.  Once UI5 reloads, dataMine is back up with its' USB stick.   Is it possible to have a Startup Lua where it will prohibit the system from reformatting a USB stick with a certain UUID?  My case BE3F-E373. 

Thanks



Title: Re: New dataMine graphing and logging plugin
Post by: TwannieM on March 10, 2015, 03:18:39 am
This does not seem to run in UI7?
Title: Re: New dataMine graphing and logging plugin
Post by: Bozo on April 22, 2015, 12:29:57 am
I can't excess to the usb stick on the Vera Edge, does anybody knows how to do this ?
I have tryed  "/dev/ttyUSB0/",  "/dev/ttyUSB0"  and  "/dev/sda/" by SetMountPoint  and SetDateDirectory "/dataMine"
I also format the usb stick with Fat 32 and make a directory called dataMine.
Title: Re: New dataMine graphing and logging plugin
Post by: solarshine on April 24, 2015, 11:04:02 am
I think I have DataMine running properly on my Veralite and am trending data from my TED5000. I also added the Radio Thermostat CT30 "TemperatureSensor1" "CurrentTemperature" variable from two thermostats but it reports zero degrees from both.

Title: Re: New dataMine graphing and logging plugin
Post by: solarshine on April 24, 2015, 11:41:17 am
Both temperatures are reporting now! Took some time that's all.

Any ideas how to uncover the humidity variable, not listed in the variables right now?
Title: Re: New dataMine graphing and logging plugin
Post by: Bozo on May 15, 2015, 08:03:46 am
Who knows the configuration of the USB port from the Vera Edge for dataMine?
And how are the settings in the Advance/Variables tab ?
USB port works on the Vera, dataMine don't see it. I tryed many days with differend settings, but it won't work.
Title: Re: New dataMine graphing and logging plugin
Post by: VeraEdger on June 26, 2015, 01:28:05 pm
Who knows the configuration of the USB port from the Vera Edge for dataMine?
And how are the settings in the Advance/Variables tab ?
USB port works on the Vera, dataMine don't see it. I tryed many days with differend settings, but it won't work.

Having the same issue. Did you fix it?
Title: Re: New dataMine graphing and logging plugin
Post by: sebohans on July 22, 2015, 08:08:52 am
Having the same issue.
I wish there was a NOOB guide to mounting your USB succefully .
And knowing what to fill in as LOCAL_DATA_DIR for dataMine or dataYours.
I'm working with a USB hub wit a 16GB usb stick and a rtx433e transceiver.
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on September 05, 2015, 05:12:07 pm
Hi.  I've been using datamine for many months now find it really useful.  One missing feature, however, is the ability to aggregate data over different time periods - e.g. have a graph of my average weekly gas usage over a period of a year without getting lost in the detailed logs.

I had a go at modifying the code and added hourly, daily, weekly and monthly aggregations to the config settings of each variable.  I also added various aggregation functions such as sum, difference, average, min and max.

I've only tested it in UI5 on a Veralite but if you want to have a go using it too, you will need to have datamine installed and working then download the attached file and install the two files as follows.  Be sure to back up the files you are replacing in case it goes wrong.

L_DataMine1.lua -  in Vera go to Apps>Develop Apps>Luup Files>Choose File and select L_DataMine1.lua, check the Restart Luup after upload box and click Go
datamine.min.js - using a tool such as WinSCP replace the existing file in the Vera directory www/dm/js.  You will need your Vera root password to do this.

Once complete, fire up the datamine url then look for the new config settings "Aggregate By (Period)" and "Aggregation Formula" in the config.

Typical usage scenarios:

** Attachment removed - see later post for most recent version of code **
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on September 16, 2015, 12:53:10 pm
Hi

I've also amended the code to make dataMine work over a remote connection.  To do this, install the attached files to your existing DataMine installation (as above):

L_DataMine1.lua and D_DataMine1.json -  in Vera go to Apps>Develop Apps>Luup Files>Choose File and select each file, check the Restart Luup after upload box and click Go
datamine.min.js - using a tool such as WinSCP replace the existing file in the Vera directory www/dm/js.  You will need your Vera root password to do this.

to get access across the public internet, go to cp.mios.com and log in with your existing account details and wait for the usual remote dashboard to come up.  This process will install a needed authentication cookie.  Then find the datamine graphing device and click on the spanner. Then click on the "dataMine graphing page" link which will take you to fwd8.mios.com/port_3480/data_request?id=lr_dmPage where datamine will load.  Alternatively you should be able to access it via fwd8.mios.com/<username>/<password>/<vera device number>/data_request?id=lr_dmPage.  You can still use <local vera ip>/dm for local access.

Whilst I was in the code, I also fixed a couple of bugs that caused it to not work properly in some browsers and some incorrect plotting of data points in certain edge cases.

*** update 2015-10-10 ***
I've fixed a bunch of bugs and improved the performance including fixing some random crashes that were in the original code

*** update 2015-11-01 ***
I've fixed some more bugs including handling of alpha values

*** update 2015-11-02 ***
fixed some broken images shown via remote access
Title: Re: New dataMine graphing and logging plugin
Post by: kennedyj on September 19, 2015, 10:14:14 am
Sorry to bring up the USB mounting issue yet again, but I just cannot get dataMine to mount my USB drive. I have a vera3 and UI5.

I have a 8GB USB drive which I have formatted via Windows in FAT32 and included a top level folder "dataMine" - no joy, will not mount.

I tried formating the USB drive in ext3 with the same result.

In both cases I can SSH into my vera3 and manually mount the USB (/dev/sdb1) onto /mnt no problems

I'm slightly confused by the "SetMountPoint" variable which many people have mentioned. This not the mount point but is the location of the device that is trying to be mounted! Have I got this wrong.

I've been trying to get this working for 2 days but I have now run out of ideas  :(


Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on September 19, 2015, 02:04:59 pm
can you paste the output of http://x.x.x.x:3480/data_request?id=lr_dmCtrl&control=debug somewhere?  where x.x.x.x is the ip address of your vera box.
Title: Re: New dataMine graphing and logging plugin
Post by: a-lurker on September 19, 2015, 05:31:31 pm
@kennedyj 
Quote
and included a top level folder "dataMine"

You do not create the dataMine directory on the stick. You create the directory under root in Vera. The plugin then mounts the stick to Vera at that directory. See:

http://forum.micasaverde.com/index.php/topic,16975.msg131509.html#msg131509

VeraEdge users have bigger problems and may want to look at this post:

http://forum.micasaverde.com/index.php/topic,33555.msg246245.html#msg246245

There is a bug in the plugin, that has never been fixed, that causes the dataMine directory to never be created.

@ConstantSphere  You have being a few doing dataMine code mods. Last time I looked, I thought this failure to create the dataMine directory, was due to this section of code. I never tried changing it - so unproven. But it's in this area somewhere here:

http://forum.micasaverde.com/index.php/topic,16243.msg131478.html#msg131478

The failure to create the directory results in the "...Error reading tmpfile during UUID check..." (likely) in the dataMine log web page. The file can't be read, because it can't be created in the function checkFreeSpace(), because the stick is not mounted.
Title: Re: New dataMine graphing and logging plugin
Post by: kennedyj on September 27, 2015, 03:04:33 pm
Sorry for the delay in responding to you guys - been on a business trip.

Anyway, I manually created a top level directory /dataMine, and lo and behold the USB drive mounted automatically after a reboot  :)

Now I just need to figure out how I collect data from the HEM...

Thanks, JohnK.
Title: Re: New dataMine graphing and logging plugin
Post by: geniepro on October 31, 2015, 09:45:23 pm
The latest version, what is included in dataMine 2015-10-10.zip, doesn't seem to work fine for non Number Values, not sure if this worked before - I just started implementing this.

I'm trying to graph ModeState from the thermostat - the values from vera are Idle, Cooling, Heating, please see below the content of /dataMine/database/1/raw/2391.txt:

Code: [Select]
446299663,Cooling
1446299896,Idle
1446300023,Cooling
1446300069,Idle
1446305776,Cooling
1446306286,Idle
1446321602,Cooling
1446321958,Idle
1446340737,Cooling
1446341036,Idle

When I try save a new Lookup with label 'New Label' and val 'Idle' the below debug messages are returned:

Code: [Select]
50 10/31/15 20:37:34.298 luup_log:15: dataMine: Saving: D[3] S[urn:micasaverde-com:serviceId:HVAC_OperatingState1] V[ModeState] <0x7402a520>
50 10/31/15 20:37:34.299 luup_log:15: dataMine: Graph Type = 1 <0x7402a520>
50 10/31/15 20:37:34.299 luup_log:15: dataMine: Save Lookup=[{"lab":"Heating","val":3},{"lab":"Cooling","val":1},{"lab":"New Label","val":"Idle"}] <0x7402a520>
06 10/31/15 20:37:34.367 Device_Variable::m_szValue_set device: 15 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 10 now: 10 #hooks: 0 upnp: 0 skip: 0 v:0xcc4a90/NONE duplicate:1 <0x7402a520>
06 10/31/15 20:37:34.368 Device_Variable::m_szValue_set device: 15 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 10 now: 10 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x7402a520>
50 10/31/15 20:37:34.370 luup_log:15: dataMine: {"Variables":[{"Ghost":false,"Service":"urn:micasaverde-com:serviceId:HVAC_OperatingState1","AggregateBy":0,"DrowsyError":0,"AggregationType":0,"LastRec":1446341036,"Units":"0","Device":3,"FilterMaximum":0,"Logging":1,"Type":1,"FilterEnable":0,"DataType":0,"Id":1,"FilterMinimum":0,"Alpha":1,"LastHistory":1104516000,"DrowsyWarning":0,"DataOffset":0,"Name":"UThermostat - ModeState","LastVal":"Idle","FirstRec":0,"Variable":"ModeState","historyState":0,"Lookup":{"Heating":3,"Cooling":1}}]} <0x7402a520>

If I interpret correctly the logs the request is received but for some reason is ignored, it responds with the original lookup ""Lookup":{"Heating":3,"Cooling":1}"

Am I doing something wrong ?

Thank you.
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on November 01, 2015, 03:07:16 am
I'm pretty sure it used to work, so I suspect I've broken it. I'll take a look in the next couple of days.
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on November 01, 2015, 06:32:35 pm
Actually, what you need to do is set your label to the recorded data (e.g. Idle) and your value to a unique number on the graph Y-axis (e.g. 0). You cannot store non-numerics in the value field - nor does it make sense to do this. In this example, the recorded state of Idle will show in position 0 on the Y-axis which will be re-labelled as "Idle", you can then set "Cooling" to value 1 and "Heating" to value 2 etc.

Having said that, I also noticed a bug in the way my code handles alpha values and will upload a new version with a fix.
Title: Re: New dataMine graphing and logging plugin
Post by: geniepro on November 01, 2015, 11:12:29 pm
Thanks @ConstantSphere I tried that as well but I can't graph, it will always show only the current state - in a continuous line.
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on November 02, 2015, 04:30:58 am
Hi @geniepro. I've tried your input file with the latest version of the code (that contained a fix for graphing alpha values) and it seems to work for me.  Can you confirm the following:
1. You have the latest version of the code I uploaded the other night (dataMine 2015-11-01.zip a few posts back)
2. Properties for the variable are set as follows: Data Offset 0, Data Multiplier 1, Aggregate By (Period) None (raw data), Aggregation Formula Sample (default).

If it still doesn't work, can you post a link to the output of your debug page at http://<vera IP>/port_3480/data_request?id=lr_dmCtrl&control=debug
Title: Re: New dataMine graphing and logging plugin
Post by: geniepro on November 02, 2015, 08:45:36 am
Thank you @ConstantSphere it works now.

This time I logged in through "/port_3480/data_request?id=lr_dmPage" and I get broken images in the "Configuration" tab. The img tag is:

<img src="images/light-bulb-small-off.png">

for the working images the tag is:

<img src="?id=lr_dmPage&amp;page=images/exclamation-small.png">
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on November 02, 2015, 03:03:38 pm
Well spotted! I've coded a potential fix to the broken images issue (dataMine 2015-11-02.zip a few posts back).
Thanks for all the feedback.
Title: Re: New dataMine graphing and logging plugin
Post by: geniepro on November 09, 2015, 08:40:10 am
Sorry for the late feedback.

The images work perfectly now.

Since my understanding is that the graphs should be working while accessing the URL remotely I tried to access:

Code: [Select]
https://vera-us-oem-relay41.mios.com/port_3480/data_request?id=lr_dmPage
but I get:

Code: [Select]
ERROR:Module not found:port_3480
I access it the same way as when I'm local, by clicking on "dataMine graphing page"
Title: Re: New dataMine graphing and logging plugin
Post by: ConstantSphere on November 09, 2015, 10:41:52 am
The best way to access the site remotely is to log into you mios account here: https://cp.mios.com/login.php and navigate through to the datamine device and click on the "dataMine graphing page" link.  By going in through this route an authentication cookie will be set on your machine