DataMine 2.0

[ConstantSphere]

Hi.

I have been working on some improvements to the excellent DataMine app.  Note that as this plug-in uses the same web addresses and data storage locations as the original version of datamine it is not possible to run them at the same time (at least I wouldn't expect them both to work), so it is necessary to uninstall the original version before installing this one.  It is a derivative work using the same code base and same data and config files so it should just work with the data you already have.  If it doesn't work, just uninstall and go back to the original version.  That said please make a copy of any USB advanced settings in dataMine1 before moving to dataMine2 as these will be lost in the transition.

On top of the original functionality it has
- the ability to aggregate data by none (as before), hour, day, week and month
- aggregation functions of sample (as before), difference, sum, weighted average, minimum, maximum and on-duration
- plot charts as columns (as well as spline, line, area and scatter)
- a new API call to provide aggregated data
- a new API call to provide data in a format suitable for Google Charts
- the ability to access web UI remotely (ALTUI plug-in needed for this to work in UI7)
- ability to save and edit aggregation and chart data with a graph (since 1.910)
- ability to clone a variable and use it multiple times on the same graph (since 1.911)
- ability to customise graphs via themes (coming in 1.912)
- Ability to time shift data for direct comparison (new in 1.913)
- various performance improvements
- improvements to the documentation

I have now published this version 2.0 in the MiOS Marketplace here: http://apps.mios.com/plugin.php?id=8526

The dataMine2 User Guide is now available for download as a pdf here: http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf.  Comments specific to the documentation should be directed to a separate forum here: http://forum.micasaverde.com/index.php/topic,35724.0.html

*update 26 April 2016: updated new features and documentation for version 1.914 (awaiting approval in the app store)*

[ConstantSphere]

Whilst the plugin works on my machine, it seems I made an error when adding it to the marketplace.  A major part of the functionality is in the web front end which consists of over 500 files packed into a tar.gz bundle in the /www/dm directory.  Upon running the app for the first time it unpacks the bundle and deletes the tar.gz.  However, it looks like this file is some how corrupt and therefore cannot be found and unpacked, it's also too large to attach to this post!

As a workaround, most of the files are the same as the original version of dataMine.  If you have previously had that installed then use a tool such as WinSCP to replace the existing datamine.min.js file in the Vera directory www/dm/js and dataMine.css, graph.html, graph2.html and index.html files in www/dm.  The other files should already be correctly in place from the installation. You will need your Vera root password to do this.

* Attachment removed - please download the latest version from the marketplace store which doesn't need this workaround *

[ConstantSphere]

TE-Rex wrote...

I am using datamine Graph to show data from a rain sensor. I am currently using graph type Spline but is not fully satisfied with the graph.

High charts has the graph type "column" which is nice for visualizing rain fall data.
 
This link shows an example of that:
http://www.highcharts.com/demo/column-basic

Any plans to add this graph type in the future? Hoping... 

I did have a look at adding a column chart a while ago to dataMine but ran into trouble when trying to combine it with other chart types.  It might be that I just need to do some more investigation but I'd like to avoid upgrading to a newer version of Highcharts as it would be a big undertaking. 

As an alternative I built a new API for Google Charts and a sample page which does allow displaying charts in columns. It's still a work in progress, so can be a bit tricky to get right.  However, if you follow the above install instructions you will now have a new file  accessible from <vera ip>/dm/graph2.html?ref=<graph name>

This page uses Google Charts to display a graph that you have saved using the Save Graph button on the main chart tab of dataMine.  The <graph name> is what you have entered on the Save Graph dialog under Quickview Reference.  To make a chart appear as a column type, chose Area/Line in datamine as the Graph Type under the Configuration Properties tab.  Area/Line charts will be rendered as column charts on the graph2.html page.

Note that the background colour of this page exactly matches the background colour of the Imperihome mobile app web view.

Also attached is an example of some column charts I created, the first shows the number of hours the heating has been on over the last few days. To create this I used graphed the heating SwitchPower status of my heating controller and set the Aggregate By (Period) to "Day" and the Aggregation Formula to "On Duration (Hours)".  The second shows monthly gas and electricity usage where I graphed the number of KWH from my energy monitor and set the Aggregate By (Period) to "Month" and the Aggregation Formula to "Difference (Max - Min)".

I'll look into something a bit easier to use, if I have time.

[TeTTiweTTi]

Whilst the plugin works on my machine, it seems I made an error when adding it to the marketplace.  A major part of the functionality is in the web front end which consists of over 500 files packed into a tar.gz bundle in the /www/dm directory.  Upon running the app for the first time it unpacks the bundle and deletes the tar.gz.  However, it looks like this file is some how corrupt and therefore cannot be found and unpacked

I installed DataMine 2.0 whilst storing the data on a NAS. It's logging OK for 28h now.
Is this the reason why some Icons aren't displaced on the datamine dashboard? I don't really bother though, since I only use it for setting things up. I view the graphs on the ImperiHome app.

[ConstantSphere]

I guess you got lucky! I'm pleased to hear it is working for you.

[ttopic]

Hi @ConstantSphere, do you have an installation procedure? Because I installed it from Apps --> Install Apps from UI7 but I do not know how to use it. I click on the link "datamine grapphing page", I chosed the UUID USB, and then appears "Please restart Luup (press the reload button in UI5). Wait a moment before then refreshing the GUI." Then I can't do anything. Best Regards,

[kyle.dawson]

I uninstalled the older 2.0 then reinstalled today with new app in App Store.  It seems to install ok, but when I go to the "dataMine graphing page", I just get the loading icon from the app.    Attached screen shot.   I tried to mount my USB on my own, then point to /usb/DataMine.    FYI. /usb is the name of my mount point, and I have directory in there named DataMine.  Any Suggestions on getting this setup?  I am running vera lite, UI5. I have reboot the device a few times as well.

[ConstantSphere]

@kyle.dawson any chance you could send me a copy of your log (http://<vera ip>/cgi-bin/cmh/log.sh?Device=LuaUPnP) or the output of http://<vera ip>/port_3480/data_request?id=lr_dmCtrl&control=debug (I'd be interested to know if this works)?

Do you get the same result from http://<vera ip>/dm and http://<vera ip>/port_3480/data_request?id=lr_dmPage ?

Can you also confirm that you have the latest version (downloaded after the 4th January 2016)? None of the previous versions work!

Thanks

[ConstantSphere]

@ttopic it sounds like you progressed further than @kyle.dawson but could you also send me a copy of your logs so I can see what is going on. As above, can you also confirm that you have the latest version (downloaded after the 4th January 2016)? In the meantime you may wish to download the DataMine overview from here: http://forum.micasaverde.com/index.php/topic,13057.0.html. Thanks.

[kyle.dawson]

@kyle.dawson any chance you could send me a copy of your log (http://<vera ip>/cgi-bin/cmh/log.sh?Device=LuaUPnP) or the output of http://<vera ip>/port_3480/data_request?id=lr_dmCtrl&control=debug (I'd be interested to know if this works)?

Debug link does not work

Do you get the same result from http://<vera ip>/dm and http://<vera ip>/port_3480/data_request?id=lr_dmPage ?

Both just clock with loading image

Can you also confirm that you have the latest version (downloaded after the 4th January 2016)? None of the previous versions work!
Thanks

Yes, I uninstalled and reinstalled from app store just now.




Looking at the logs, I see this:

01/05/16 9:14:40.088    luup_log:172: dataMine: Initialising dataMine System (1.891) <0x2c2e1680>
01/05/16 9:14:40.088    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1merrorStatus^[[0m was: EMPTY now: false #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.089    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetDataDirectory^[[0m was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2
01/05/16 9:14:40.090    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetMountUUID^[[0m was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.091    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetMountPoint^[[0m was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.092    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetMountOptions^[[0m was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.092    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetManualMount^[[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.093    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetTimeoutPeriod^[[0m was: EMPTY now: 12 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.094    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetHistoryEnable^[[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.117    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetUseBackup^[[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.118    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mSetEventsEnable^[[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.120    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mChannelCnt^[[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0x14421b8/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.120    Device_Variable::m_szValue_set device: 172 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ^[[35;1mChannelRec^[[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c2e1680>
01/05/16 9:14:40.123    ^[[31;1mluup_require can't find json-dm^[[0m <0x2c2e1680>
01/05/16 9:14:40.134    luup_log:172: dataMine: ERROR: Startup state is 'uninitialised'! <0x2c2e1680>


Also, looks like the icon does not show up, does that mean anything?

[ttopic]

Hi guys, I solved the problem by allowing the VERA to log data in my USB. Just go to the UI7 Settings --> Logs and then check "Store logs on USB device". At the beginning the USB Status said "Not mounted". So, I plugged my USB (it can be a FAT32, which is the filesystem I used) on port USB1 and turned off and then turned on the VERA. After that, the USB Status said "Enabled" and my Datamine Plugin worked perfectly. I am learning the UI of the plugin. It is working awesome.

[ttopic]

Hi @kyle.dawson you just need to reload the page. I had the same problem many times, I didn't know what to do. Reloading page was my solution.

[ConstantSphere]

@kyle.dawson - the interesting thing to note in the logs for me is the error "luup_require can't find json-dm".  json-dm is the pre-installed JSON library used by UI5 but it cannot be found which is why it's failing to start.  Which UI version are you running and have you ever had DataMine1 working on your box? 

After looking through the forums it seems that UI7 uses a different json decoder - as an experiment, can you try uploading the attached file to your luup files folder under develop apps and restarting your box?  It first tries to load the UI7 decoder "dkjson" and then tries the UI5 decoder "json-dm" if that fails.

Actually - I just noticed you are on UI5 from your signature.  Can you try uninstalling DataMine2 and install DataMine1 and see if that works? If it does then uninstall DataMine1 and reinstall DataMine2 and retest.  I want to see if DataMine1 included a json library in the distribution.  If you are able to tell me the names of any files with "json" in the name in the /usr/lib/lua directory that would also be useful.

Let me know how you get on. Thanks

*attachment removed as it dosn't fix the problem*

[a-lurker]

Using the plugin AltUI (highly recommended) you can do:   Misc-->OS Command-->Find Json.  Example output:

Code: [Select]

/mios/usr/lib/lua/dkjson.lua
/overlay/usr/lib/lua/json.lua
/overlay/usr/lib/lua/json-dm.lua
/overlay/usr/lib/lua/dkjson.lua
/usr/lib/lua/json.lua
/usr/lib/lua/json-dm.lua
/usr/lib/lua/dkjson.lua
UI5 required the file "json-dm.lua" to be manually transferred to Vera. in UI7 the file "dkjson.lua" appears to be automatically included as part of Vera and if I remember correctly, is used regularly by @akbooer's plugins. "json.lua" is required by the OWserver plugin and was also manually transferred to Vera UI5, so you could check for that as well. It would be nice if the log indicated which is being used.

I'm not too sure if they are all interchangeable (hopefully they are for this usage case) but Chris was having some trouble with the various decoders at one point.

[ConstantSphere]

thanks @a-lurker that's really useful. I've done a bit of investigation and discovered that the original DataMine app included the json-dm.lua file as part of the distribution whereas I'd assumed it was already installed on the box (I guess it was -dm for DataMine - obvious really!) I've now added the file to my distribution and submitted for approval.  I've actually called it json-dm2.lua as you are not allowed to reuse file names. It's probably not the cleanest solution but was the one used in the original version  and I'm assuming this approach will work on any version of the UI.

Whilst waiting for approval, there are a couple of work-arounds:
1. install the original DataMine, uninstall it and then install DataMine2
2. put the attached json-dm.lua file into /usr/lib/lua

*it's now approved so just install the latest version*

I also saw that the Data_Mine.png icon was missing from my distribution but I'll come back and fix that later.