We have moved at community.getvera.com

Author Topic: dataMine 2 documentation  (Read 22668 times)

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
dataMine 2 documentation
« on: January 11, 2016, 11:09:16 am »
This is the forum for the dataMine2 User Guide is now available for download as a pdf here: http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf

dataMine 2 is a derivative work that builds on the excellent dataMine system for logging variables. It adds the ability to aggregate data over various time periods, provides access to the system over a remote connection and now has a new data source compatible with Google charts. As this version uses the same web locations as dataMine 1, you will need to uninstall it for version 2 to work - all your data and configuration is shared and will be retained.

On top of the original functionality it has:

  • the ability to aggregate data by none (as before), hour, day, week and month
  • aggregation functions of sample (as before), difference, sum, weighted average, minimum, maximum and on-duration
  • ability to plot data as a column chart
  • a new API call to provide aggregated data
  • a new API call to provide data in a format suitable for Google Charts
  • the ability to access web UI remotely (ALTUI plug-in needed for this to work in UI7)
  • ability to save and edit aggregation and chart data with a graph (since 1.910)
  • ability to create duplicate variables and display multiple times on a single graph (since 1.911)
  • ability to customise a graph though themes (since 1.912)
  • Ability to time shift data for direct comparison (since 1.913)
  • various performance improvements
  • improvements to the documentation

Version 2.0 is now published in the MiOS Marketplace here ​http://apps.mios.com/plugin.php?id=8526

Please only use this forum for comments on the documentation.  Comments on the plug-in itself should be directed to http://forum.micasaverde.com/index.php/topic,35592.0.html

*update 25 April 2016: updated new features and documentation for version 1.914*
« Last Edit: April 28, 2016, 06:21:10 pm by ConstantSphere »
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: dataMine 2 documentation
« Reply #1 on: January 11, 2016, 11:29:50 am »
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7.  I run UI5 so have no idea about this and am not even sure if it works.
2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick.  I've seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven't tried it myself.

Your help would be much appreciated.  Thanks!
« Last Edit: January 11, 2016, 11:32:58 am by ConstantSphere »
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline SchattenMann

  • Sr. Newbie
  • *
  • Posts: 42
  • Karma: +4/-1
Re: dataMine 2 documentation
« Reply #2 on: January 21, 2016, 07:32:55 am »
Help Needed!
2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick.  I've seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven't tried it myself.

I can confirm dataMine works fine with CIFS by installing the required packets:

Code: [Select]
cd /tmp/
opkg update
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-md5_2.6.37.1-1_ramips.ipk
opkg install kmod-crypto-md5_2.6.37.1-1_ramips.ipk
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-hmac_2.6.37.1-1_ramips.ipk
opkg install kmod-crypto-hmac_2.6.37.1-1_ramips.ipk
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-fs-cifs_2.6.37.1-1_ramips.ipk
opkg install kmod-fs-cifs_2.6.37.1-1_ramips.ipk
opkg install cifsmount
rm kmod-crypto-md5_2.6.37.1-1_ramips.ipk kmod-fs-cifs_2.6.37.1-1_ramips.ipk kmod-crypto-hmac_2.6.37.1-1_ramips.ipk

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #3 on: January 22, 2016, 07:34:50 am »
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline SchattenMann

  • Sr. Newbie
  • *
  • Posts: 42
  • Karma: +4/-1
Re: dataMine 2 documentation
« Reply #4 on: January 22, 2016, 07:40:22 am »
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene

I was using the same code for dataYours but you don't really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #5 on: January 23, 2016, 06:02:00 am »
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene

I was using the same code for dataYours but you don't really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own
You can, but after several Vera updates I got tiered of having to perform several of these steps manually over and over. It is all about automation isn't it  ;D.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline bucko

  • Hero Member
  • *****
  • Posts: 659
  • Karma: +27/-9
Re: dataMine 2 documentation
« Reply #6 on: January 23, 2016, 08:44:55 am »
I'm guessing this applies to UI7 only?
Vera 3, 40+ Fibaro relays & dimmers, IPUX cams, Elk M1 Gold, M1XEP, M1SP, XVOR, M1RB, Elk124, Monoprice HDMI matrix & 6 zone amp, Denon AVR, Panasonic A8000, BI DV,On Controls

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #7 on: January 23, 2016, 08:48:49 am »
I'm guessing this applies to UI7 only?
Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline mcalistair

  • Full Member
  • ***
  • Posts: 179
  • Karma: +6/-3
  • "Luctor et Emergo"
Re: dataMine 2 documentation
« Reply #8 on: January 23, 2016, 09:52:05 am »
I'm guessing this applies to UI7 only?
Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene

So to check... on UI5 I guess It wont be a problem as there will be no updates anymore?!
1x Vera3@UI5 = PROD (1x Edge@UI7 = SandBox ), 15x ZWAVE Devices, 8x 'legacy' X10 devices controlled via Visonic PowerMax Alarm Panel Plugin, 5x Philips HUE devices, 1x iTach IP2CC, 1x Netatmo Weather Station, AltUI

Offline Terryleier

  • Sr. Newbie
  • *
  • Posts: 28
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #9 on: January 23, 2016, 04:14:14 pm »
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #10 on: January 23, 2016, 07:34:49 pm »
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #11 on: January 24, 2016, 02:15:35 pm »
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene
Hi,
I tweaked the datamine draw routing in ALTUI so it works remotely box UI7 users. You must go via ALTUI though. I will send this to amg0 to put in the standard ALTUI code.

You have to do this in the J_ALTUI_plugins.js code (don't forget to refresh your browser.

Code: [Select]
function _drawDataMine( device) {
var html ="";
var url = "";
// var ipaddr = MultiBox.getIpAddr(device.altuiid);
// var hostname = (ipaddr=='') ? window.location.hostname : ipaddr;
// var url = window.location.protocol+'//'+hostname+"/dm/index.html";
if (window.location.protocol=='https:') {
var controller = MultiBox.controllerOf(device.altuiid).controller;
var isUI5 = MultiBox.isUI5(controller);
if (isUI5 == false) {
var main_url = window.location.href;
var url_parts = main_url.split("?");
url = url_parts[0]+"?id=lr_dmPage"
} else {
url = 'https://'+hostname+"/port_3480/data_request?id=lr_dmPage";
}
} else {
var ipaddr = MultiBox.getIpAddr(device.altuiid);
var hostname = (ipaddr=='') ? window.location.hostname : ipaddr;
url = 'http://'+hostname+"/dm/index.html";
}
html+= ("<button id='altui-datamine-{0}' type='button' class='pull-right altui-datamine-open btn btn-default btn-sm ' >{1}</button>" .format( device.altuiid,_T("Open") )) ;
html += "<script type='text/javascript'>";
html += " $('button#altui-datamine-{0}.altui-datamine-open').on('click', function() { window.open('{1}','_blank'); } );".format(device.altuiid,url);
html += "</script>";
return html;
};

Have fun.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: dataMine 2 documentation
« Reply #12 on: January 28, 2016, 05:24:34 pm »
@reneboer - very cool  8)
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: dataMine 2 documentation
« Reply #13 on: February 07, 2016, 03:36:23 pm »
I've updated the documentation to cover the new features of the pending 1.910 release, i.e. the ability to save and edit various graph parameters in the Save Graph dialog. I've also added a section to clarify connecting to a remote CIFS store and a paragraph on connecting remotely via UI7 - thanks for all those who contributed. Much appreciated  :)
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline cb4

  • Newbie
  • *
  • Posts: 16
  • Karma: +1/-0
dataMine 2 with CIFS on UI7
« Reply #14 on: February 28, 2016, 10:43:55 am »
I can confirm dataMine works fine with CIFS by installing the required packets:

Currently, opkg commands are failing with this error:
Code: [Select]
wget: server returned error: HTTP/1.1 404 Not Found

The server that opkg uses is down so any opkg command related to updating or installing will fail. For additional info, see this post (once approved by Mods): http://forum.micasaverde.com/index.php/topic,16452.msg272134.html#msg272134

cb4