The Vera Community forums have moved!

Advanced => Plugins & Plugin Development => Programming => dataMine Plugin => Topic started by: ConstantSphere on January 11, 2016, 11:09:16 am

Title: dataMine 2 documentation
Post by: ConstantSphere on January 11, 2016, 11:09:16 am
This is the forum for the dataMine2 User Guide is now available for download as a pdf here: http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf

dataMine 2 is a derivative work that builds on the excellent dataMine system for logging variables. It adds the ability to aggregate data over various time periods, provides access to the system over a remote connection and now has a new data source compatible with Google charts. As this version uses the same web locations as dataMine 1, you will need to uninstall it for version 2 to work - all your data and configuration is shared and will be retained.

On top of the original functionality it has:


Version 2.0 is now published in the MiOS Marketplace here ​http://apps.mios.com/plugin.php?id=8526

Please only use this forum for comments on the documentation.  Comments on the plug-in itself should be directed to http://forum.micasaverde.com/index.php/topic,35592.0.html

*update 25 April 2016: updated new features and documentation for version 1.914*
Title: Re: dataMine 2 documentation
Post by: ConstantSphere on January 11, 2016, 11:29:50 am
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7.  I run UI5 so have no idea about this and am not even sure if it works.
2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick.  I've seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven't tried it myself.

Your help would be much appreciated.  Thanks!
Title: Re: dataMine 2 documentation
Post by: SchattenMann on January 21, 2016, 07:32:55 am
Help Needed!
2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick.  I've seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven't tried it myself.

I can confirm dataMine works fine with CIFS by installing the required packets:

Code: [Select]
cd /tmp/
opkg update
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-md5_2.6.37.1-1_ramips.ipk
opkg install kmod-crypto-md5_2.6.37.1-1_ramips.ipk
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-hmac_2.6.37.1-1_ramips.ipk
opkg install kmod-crypto-hmac_2.6.37.1-1_ramips.ipk
wget  http://download.mios.com/firmware/openwrt/ramips/packages/kmod-fs-cifs_2.6.37.1-1_ramips.ipk
opkg install kmod-fs-cifs_2.6.37.1-1_ramips.ipk
opkg install cifsmount
rm kmod-crypto-md5_2.6.37.1-1_ramips.ipk kmod-fs-cifs_2.6.37.1-1_ramips.ipk kmod-crypto-hmac_2.6.37.1-1_ramips.ipk
Title: Re: dataMine 2 documentation
Post by: reneboer on January 22, 2016, 07:34:50 am
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: SchattenMann on January 22, 2016, 07:40:22 am
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene

I was using the same code for dataYours but you don't really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own
Title: Re: dataMine 2 documentation
Post by: reneboer on January 23, 2016, 06:02:00 am
Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

Code: [Select]
-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status))
    if ok
      then log ("mounted OK: "..tostring(ok))
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount"
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene

I was using the same code for dataYours but you don't really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own
You can, but after several Vera updates I got tiered of having to perform several of these steps manually over and over. It is all about automation isn't it  ;D.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: bucko on January 23, 2016, 08:44:55 am
I'm guessing this applies to UI7 only?
Title: Re: dataMine 2 documentation
Post by: reneboer on January 23, 2016, 08:48:49 am
I'm guessing this applies to UI7 only?
Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: mcalistair on January 23, 2016, 09:52:05 am
I'm guessing this applies to UI7 only?
Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene

So to check... on UI5 I guess It wont be a problem as there will be no updates anymore?!
Title: Re: dataMine 2 documentation
Post by: Terryleier on January 23, 2016, 04:14:14 pm
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html (http://forum.micasaverde.com/index.php/topic,24942.0.html)

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484 (http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484)
Title: Re: dataMine 2 documentation
Post by: reneboer on January 23, 2016, 07:34:49 pm
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html (http://forum.micasaverde.com/index.php/topic,24942.0.html)

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484 (http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484)
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: reneboer on January 24, 2016, 02:15:35 pm
Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:
1. Remote access of dataMine2 under UI7...
...

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: http://forum.micasaverde.com/index.php/topic,24942.0.html (http://forum.micasaverde.com/index.php/topic,24942.0.html)

and user wadeb kindly posted a PHP implementation on GitHub described here: http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484 (http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484)
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene
Hi,
I tweaked the datamine draw routing in ALTUI so it works remotely box UI7 users. You must go via ALTUI though. I will send this to amg0 to put in the standard ALTUI code.

You have to do this in the J_ALTUI_plugins.js code (don't forget to refresh your browser.

Code: [Select]
function _drawDataMine( device) {
var html ="";
var url = "";
// var ipaddr = MultiBox.getIpAddr(device.altuiid);
// var hostname = (ipaddr=='') ? window.location.hostname : ipaddr;
// var url = window.location.protocol+'//'+hostname+"/dm/index.html";
if (window.location.protocol=='https:') {
var controller = MultiBox.controllerOf(device.altuiid).controller;
var isUI5 = MultiBox.isUI5(controller);
if (isUI5 == false) {
var main_url = window.location.href;
var url_parts = main_url.split("?");
url = url_parts[0]+"?id=lr_dmPage"
} else {
url = 'https://'+hostname+"/port_3480/data_request?id=lr_dmPage";
}
} else {
var ipaddr = MultiBox.getIpAddr(device.altuiid);
var hostname = (ipaddr=='') ? window.location.hostname : ipaddr;
url = 'http://'+hostname+"/dm/index.html";
}
html+= ("<button id='altui-datamine-{0}' type='button' class='pull-right altui-datamine-open btn btn-default btn-sm ' >{1}</button>" .format( device.altuiid,_T("Open") )) ;
html += "<script type='text/javascript'>";
html += " $('button#altui-datamine-{0}.altui-datamine-open').on('click', function() { window.open('{1}','_blank'); } );".format(device.altuiid,url);
html += "</script>";
return html;
};

Have fun.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: ConstantSphere on January 28, 2016, 05:24:34 pm
@reneboer - very cool  8)
Title: Re: dataMine 2 documentation
Post by: ConstantSphere on February 07, 2016, 03:36:23 pm
I've updated the documentation to cover the new features of the pending 1.910 release, i.e. the ability to save and edit various graph parameters in the Save Graph dialog. I've also added a section to clarify connecting to a remote CIFS store and a paragraph on connecting remotely via UI7 - thanks for all those who contributed. Much appreciated  :)
Title: dataMine 2 with CIFS on UI7
Post by: cb4 on February 28, 2016, 10:43:55 am
I can confirm dataMine works fine with CIFS by installing the required packets:

Currently, opkg commands are failing with this error:
Code: [Select]
wget: server returned error: HTTP/1.1 404 Not Found

The server that opkg uses is down so any opkg command related to updating or installing will fail. For additional info, see this post (once approved by Mods): http://forum.micasaverde.com/index.php/topic,16452.msg272134.html#msg272134

cb4
Title: Re: dataMine 2 documentation
Post by: RWoerz on September 23, 2016, 09:08:49 pm
dataMine 2 and Linux for dummies like me. 

Not being a Linux expert I've struggled for a while trying to get dataMine 2 working on my Very Edge UI7. Starting but not limited to the fact that when you plug the first USB drive into a Vera no matter what size the drive is it creates one 500MB partition which isn't easy to undo.

I've even spent time with Vera support asking if the 2nd USB drive had mounted and according to them everything was working as expected, whatever that means. I'm sure had I known Linux this would have been easy but there is so much info out there it's hard to know what's current and what's not.

One big issue for me is the way many people who understand Linux post things. One post showed the output of what was said to be a "file" command. The output was interesting but they didn't show the command itself so it was of little help.   

If the folks at Vera are reading this it would be very helpful to have a tab that gives some basic systems info. Like what drives are mounted, how full are they, IP address, when a sensor was last seen in a human readable form not Epoch time. Epoch time, that's Linux speak for the number of seconds since midnight Jan 1st 1970, I think that's right.

Some posts say you can have Vera logging turned on some say no. Anyway I'm still not sure exactly what was needed but I'll give you what I did and hopefully someone can tell us what wasn't.

One note to start, I'll show all Linux and Vera commands in quotes i.e. "lsusb" so you don't type in the quotes just what's between them. Also if you don't know this Linux is case sensitive so be aware type everything in as shown.

The whole DataMine thing may just work for you through the Vera interface but that wasn't what happened with me.

First thing I would plug in an old 1GB USB drive then turn logging on in the Vera "Settings, Logs" tab. Vera will grab hold of it and hopefully not think the next drive you plug in belongs to it.

Second you'll need to download a program called PuTTY www.putty.org/, it allows you to "terminal" into the Vera box and run Linux commands in a DOS like window. Unix/Linux was around long before DOS and is a bit of a steep learning curve but to get what you need for dataMine you'll only need a few commands.
Be forewarned you can KILL your system with the wrong command here so be very careful what you type in.   

When you run PuTTY you'll type in the IP address of you Vera box, you'll find the IP at the top of your browser page when you're logged into your Vera. Next make sure the port number is 22 and SSH is selected, then press the Open button.

If all is working a DOS like window will open and after several seconds a username prompt will appear. Type in "root" and press return. Next a password prompt will appear, my password was printed on a label on the bottom of my Vera, yours may vary. Type in your password and hit return. If all is well you've just logged into Linux on your Vera machine. I don't think it drops you into the root directory so to make sure your are type in "cd /", (cd space forward slash), that should put you at the root directory of your machine.

One problem I found with the version Linux installed on the Vera is that it doesn't support many of the traditional tools supported on many Linux system. So if you Google for a command to show the UUID of a USB drive most of the commands you'll find don't work here. All of the commands I show here worked on my Vera.

First you'll need to get the UUID of your USB drive. Before you plug in your new dataMine logging USB drive type in the following command "fdisk -l" (lower case L). At the end of the output you'll see something like this which is my 2GB Vera logging drive with its 500MB partition.

 Disk /dev/sdb: 2003 MB, 2003828736 bytes
62 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1   *           1         273      524705+ 83 Linux
/dev/sdb2             274        1018     1431890  83 Linux
root@MiOS_45011987:~#

Now plug in your dataMine drive and run the same command "fdisk-l". You should see the newly mounted drive. Mine is a 16GB USB drive and it looks like this.

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1        1947    15632384   c Win95 FAT32 (LBA)

The important thing to note here is the /dev/sda1 which you'll type into Vera's dataMine SetMountPoint. Yours may be different so write it down!

Next type in "lsusb" this will give you your drives UUID and should look something like this. Mine is the Cruzer and its UUID is 0781-5530, write yours down!

root@MiOS_45011987:~# lsusb
Bus 001 Device 002: ID 050d:0237 Belkin Components F5U237 USB 2.0 7-Port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 005: ID 8644:800a
root@MiOS_45011987:~#



Next install dataMine 2 on your Vera but make sure to leave the PuTTY window open, go to Vera's "Apps, Install apps" tab then search for dataMine. There will be two pick dataMine 2. After the install is complete go to Vera's Devices tab and find the dataMine 2 graphing device and open it. Then open the Advanced tab. Near the top of the page you'll find 4 tabs, Parms, Variables, New service and Commands. Select the Variables tab. Under the Variables tab add the following information. In SetDataDirectory type in "/dataMine/", after you type that in you have to right click in any of the white space on the page to update the information, strange but it's the way you have to do it. In SetMountUUID you'll have to type in your USB drives UUID. Then last but not least in SetMountPoint type in the mount point for your system each time right clicking in the white area in the web page. It will flash on the screen that the data has been updated.

The SetDataDirectory crates a dataMine folder in the root directory. The SetMountUUID and SetMountPoint attaches your USB drive to the dataMine folder.  Now go back to the PuTTY window and type in "df -h". It should show your drive attached to /dataMine directory and the used and available space.
   
root@MiOS_45011987:/dataMine# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    9.4M      5.3M      4.1M  56% /
/dev/root                 9.5M      9.5M         0 100% /rom
tmpfs                    61.9M      3.4M     58.5M   5% /tmp
/dev/mtdblock6            9.4M      5.3M      4.1M  56% /overlay
overlayfs:/overlay        9.4M      5.3M      4.1M  56% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /storage
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-firmware
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-backup
/dev/mtdblock9            7.8M      7.8M         0 100% /mios
/dev/sda1                14.9G    184.0K     14.9G   0% /dataMine

At this point you should be able in Vera to go to the Devices, dataMine 2 graphing, dataMine tab and select the "dataMine graphing page (local access)" and bring up a new tab in your browser called dataMine - graphing. If it asked select the USB Drive with the correct UUID. Under the Configuration tab add the sensors you want to graph.  If you want to see where the data files for the sensors get built type in the PuTTY window "cd /dataMine" then "du -h". That's all I got hope it helped. Please if I messed something up PLEASE let me know.

root@MiOS_45011987:~# cd /dataMine
root@MiOS_45011987:/dataMine# du -h
16.0K   ./database/1/raw
32.0K   ./database/1
16.0K   ./database/2/raw
32.0K   ./database/2
8.0K    ./database/3/raw
24.0K   ./database/3
96.0K   ./database
184.0K  .
Title: Re: dataMine 2 documentation
Post by: smartloft on October 12, 2016, 02:10:16 am
hi guys,

i have an issue with Datamine 2 resetting its configuration a couple of days ago

Is there any way we can access or download the data from the USB drive?

2 weeks worth of data were logged into the USB drive and we do not knw if there is a way to retrieve this data

help!
Title: Re: dataMine 2 documentation
Post by: loumarjr on October 16, 2016, 05:52:44 pm
Could someone explain the easiest way to access dataMime 2.0 remotely from the VeraEdge on UI7 Im not sure where to place the "?id=lr_dmPage".  Thank you!!
Title: Re: dataMine 2 documentation
Post by: reneboer on October 17, 2016, 04:07:52 am
Could someone explain the easiest way to access dataMime 2.0 remotely from the VeraEdge on UI7 Im not sure where to place the "?id=lr_dmPage".  Thank you!!
Hi,
You cannot via the Vera portal. You can only do that via ALTUI portal. Details of that you can find in the ALTUI forum articles.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: loumarjr on October 17, 2016, 10:52:39 am
Thanks Rene,

I have ALTUI but i cant figure out how to remotely accesss datamine on my Vera Edge.  Would you happen to have a line to the article or post?  I have been searching for ever.  Thank you!!

Lou
Title: Re: dataMine 2 documentation
Post by: reneboer on October 17, 2016, 12:22:07 pm
Hi,

It is fairly simple. Log-on via the ALTUI portal https://vera-ui.strongcubedfitness.com/Veralogin.php (https://vera-ui.strongcubedfitness.com/Veralogin.php), select your Vera and wait for the page to load.

Then in the URL look for the ?id=lr_ALTUI..... and remove everything after the =lr_ (ALTUI...) and add dmPage. Hit enter.

Too funny, I am just trying the Open button you see on the DM2.0 plugin on ALTUI and that launched DataMine just fine. I now I played with this in the past so I cannot recall it is just my version of if it was build in by amg0. Anyhow, if that doesn't work, the above will.

Cheers Rene
Title: Re: dataMine 2 documentation
Post by: smartloft on October 17, 2016, 08:50:38 pm
Hi RWoerz,

thanks for the in-depth explanation

however, how does one extract the data file from /dataMine directory?

is it viewable under windows?







a question, how does one copy
dataMine 2 and Linux for dummies like me. 

Not being a Linux expert I've struggled for a while trying to get dataMine 2 working on my Very Edge UI7. Starting but not limited to the fact that when you plug the first USB drive into a Vera no matter what size the drive is it creates one 500MB partition which isn't easy to undo.

I've even spent time with Vera support asking if the 2nd USB drive had mounted and according to them everything was working as expected, whatever that means. I'm sure had I known Linux this would have been easy but there is so much info out there it's hard to know what's current and what's not.

One big issue for me is the way many people who understand Linux post things. One post showed the output of what was said to be a "file" command. The output was interesting but they didn't show the command itself so it was of little help.   

If the folks at Vera are reading this it would be very helpful to have a tab that gives some basic systems info. Like what drives are mounted, how full are they, IP address, when a sensor was last seen in a human readable form not Epoch time. Epoch time, that's Linux speak for the number of seconds since midnight Jan 1st 1970, I think that's right.

Some posts say you can have Vera logging turned on some say no. Anyway I'm still not sure exactly what was needed but I'll give you what I did and hopefully someone can tell us what wasn't.

One note to start, I'll show all Linux and Vera commands in quotes i.e. "lsusb" so you don't type in the quotes just what's between them. Also if you don't know this Linux is case sensitive so be aware type everything in as shown.

The whole DataMine thing may just work for you through the Vera interface but that wasn't what happened with me.

First thing I would plug in an old 1GB USB drive then turn logging on in the Vera "Settings, Logs" tab. Vera will grab hold of it and hopefully not think the next drive you plug in belongs to it.

Second you'll need to download a program called PuTTY www.putty.org/, it allows you to "terminal" into the Vera box and run Linux commands in a DOS like window. Unix/Linux was around long before DOS and is a bit of a steep learning curve but to get what you need for dataMine you'll only need a few commands.
Be forewarned you can KILL your system with the wrong command here so be very careful what you type in.   

When you run PuTTY you'll type in the IP address of you Vera box, you'll find the IP at the top of your browser page when you're logged into your Vera. Next make sure the port number is 22 and SSH is selected, then press the Open button.

If all is working a DOS like window will open and after several seconds a username prompt will appear. Type in "root" and press return. Next a password prompt will appear, my password was printed on a label on the bottom of my Vera, yours may vary. Type in your password and hit return. If all is well you've just logged into Linux on your Vera machine. I don't think it drops you into the root directory so to make sure your are type in "cd /", (cd space forward slash), that should put you at the root directory of your machine.

One problem I found with the version Linux installed on the Vera is that it doesn't support many of the traditional tools supported on many Linux system. So if you Google for a command to show the UUID of a USB drive most of the commands you'll find don't work here. All of the commands I show here worked on my Vera.

First you'll need to get the UUID of your USB drive. Before you plug in your new dataMine logging USB drive type in the following command "fdisk -l" (lower case L). At the end of the output you'll see something like this which is my 2GB Vera logging drive with its 500MB partition.

 Disk /dev/sdb: 2003 MB, 2003828736 bytes
62 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1   *           1         273      524705+ 83 Linux
/dev/sdb2             274        1018     1431890  83 Linux
root@MiOS_45011987:~#

Now plug in your dataMine drive and run the same command "fdisk-l". You should see the newly mounted drive. Mine is a 16GB USB drive and it looks like this.

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1        1947    15632384   c Win95 FAT32 (LBA)

The important thing to note here is the /dev/sda1 which you'll type into Vera's dataMine SetMountPoint. Yours may be different so write it down!

Next type in "lsusb" this will give you your drives UUID and should look something like this. Mine is the Cruzer and its UUID is 0781-5530, write yours down!

root@MiOS_45011987:~# lsusb
Bus 001 Device 002: ID 050d:0237 Belkin Components F5U237 USB 2.0 7-Port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 005: ID 8644:800a
root@MiOS_45011987:~#



Next install dataMine 2 on your Vera but make sure to leave the PuTTY window open, go to Vera's "Apps, Install apps" tab then search for dataMine. There will be two pick dataMine 2. After the install is complete go to Vera's Devices tab and find the dataMine 2 graphing device and open it. Then open the Advanced tab. Near the top of the page you'll find 4 tabs, Parms, Variables, New service and Commands. Select the Variables tab. Under the Variables tab add the following information. In SetDataDirectory type in "/dataMine/", after you type that in you have to right click in any of the white space on the page to update the information, strange but it's the way you have to do it. In SetMountUUID you'll have to type in your USB drives UUID. Then last but not least in SetMountPoint type in the mount point for your system each time right clicking in the white area in the web page. It will flash on the screen that the data has been updated.

The SetDataDirectory crates a dataMine folder in the root directory. The SetMountUUID and SetMountPoint attaches your USB drive to the dataMine folder.  Now go back to the PuTTY window and type in "df -h". It should show your drive attached to /dataMine directory and the used and available space.
   
root@MiOS_45011987:/dataMine# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    9.4M      5.3M      4.1M  56% /
/dev/root                 9.5M      9.5M         0 100% /rom
tmpfs                    61.9M      3.4M     58.5M   5% /tmp
/dev/mtdblock6            9.4M      5.3M      4.1M  56% /overlay
overlayfs:/overlay        9.4M      5.3M      4.1M  56% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /storage
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-firmware
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-backup
/dev/mtdblock9            7.8M      7.8M         0 100% /mios
/dev/sda1                14.9G    184.0K     14.9G   0% /dataMine

At this point you should be able in Vera to go to the Devices, dataMine 2 graphing, dataMine tab and select the "dataMine graphing page (local access)" and bring up a new tab in your browser called dataMine - graphing. If it asked select the USB Drive with the correct UUID. Under the Configuration tab add the sensors you want to graph.  If you want to see where the data files for the sensors get built type in the PuTTY window "cd /dataMine" then "du -h". That's all I got hope it helped. Please if I messed something up PLEASE let me know.

root@MiOS_45011987:~# cd /dataMine
root@MiOS_45011987:/dataMine# du -h
16.0K   ./database/1/raw
32.0K   ./database/1
16.0K   ./database/2/raw
32.0K   ./database/2
8.0K    ./database/3/raw
24.0K   ./database/3
96.0K   ./database
184.0K  .
Title: Re: dataMine 2 documentation
Post by: RWoerz on November 08, 2016, 03:14:34 pm
New question. OK call me blind but in the DataMine2's PDF on page 16 it says: "If update is specified, the graph will automatically be updated periodically. The update time is in minutes."

I don't seem to be able to find this variable in Vera or the WEB page. The "Display real-time graph" is grayed out". 
Title: Re: dataMine 2 documentation
Post by: korttoma on December 15, 2016, 01:30:13 pm
Did the 7.0.19 firmware brake the cifs on the VeraPlus or is it just me?

50   12/15/16 22:01:03.706   luup_log:205: dataMine: Error reading tmpfile during mountcheck <0x77101520>
06   12/15/16 22:01:03.706   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
06   12/15/16 22:01:03.707   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
50   12/15/16 22:01:03.728   luup_log:205: dataMine: Mounting dataMine storage (//192.168.1.2/home/datamine) to (/dataMine/) <0x77101520>
50   12/15/16 22:01:03.770   luup_log:205: dataMine: mount command output:'mount: mounting //192.168.1.2/home/datamine on /dataMine/ failed: No such device
50   12/15/16 22:01:03.795   luup_log:205: dataMine: Error reading tmpfile during mountcheck <0x77101520>
06   12/15/16 22:01:03.796   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
06   12/15/16 22:01:03.796   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
50   12/15/16 22:01:03.817   luup_log:205: dataMine: Mount point error: :://192.168.1.2/home/datamine <0x77101520>
Title: Re: dataMine 2 documentation
Post by: korttoma on December 16, 2016, 02:14:34 am
I tried reverting to the previous VeraPlus  firmware 7.0.17 (1.7.2138) but I stil cannot mount cifs.
Just checked the files on my NAS and the last time the raw data file was updated was around the time I upgraded my VeraPlus to 7.0.19 (1.7.2414).
Not sure how to debug the situation. Anyone available to assist?
Title: Re: dataMine 2 documentation
Post by: korttoma on December 16, 2016, 09:02:55 am
managed to get it back up and running by issuing the command presented by @mfp in this post -> http://forum.micasaverde.com/index.php/topic,38219.0.html

Title: Re: dataMine 2 documentation
Post by: mfp on December 16, 2016, 01:04:35 pm
Had the same after updating, they were missing. Probably the kernel modules from firmware 7.0.19 have been updated, then a reinstall is needed and normal.
Title: Re: dataMine 2 documentation
Post by: Peren on January 13, 2017, 09:06:34 am
How can you change the Graph theme under graph options?

This is not a drop down. is there any documentation over configurable options?
Title: Re: dataMine 2 documentation
Post by: ConstantSphere on February 12, 2017, 04:16:10 pm
@Peren - you can type the name of the theme as free text in the Graph Options. Available options are on page 17 of the documentation available from http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf

for convenience it says...

Graph Theme ? sets the theme of the graphing area, controlling colours, grid lines, borders, spacing and the like. There are now 4 new themes: classic (default), gray, grid-light and sandsignika. Further themes can be downloaded from https://github.com/highcharts/highcharts/tree/master/js/themes and saved in the /www/dm/themes directory. You will need to reload the GUI (usually press F5) after changing the value to see the new theme.
Title: Re: dataMine 2 documentation
Post by: ConstantSphere on February 12, 2017, 04:20:52 pm
@RWoerz

the manual is referring to the page http://vera-ip/dm/graph.html?ref=<quickref>&period=<days>&update=<mins> where the data can be updated in real time and not the main dataMine gui.  The auto update feature on the main dataMine gui is currently broken and as you note the button is greyed out and disabled.
Title: Re: dataMine 2 documentation
Post by: doinitsideways on July 13, 2017, 05:32:49 pm
Hi,

Just my 2c

I found i had to select the tick box under the 3 variables that we put the UUID data into to get it to work... But beside that, thanks heaps @RWoerz

Cheers,
Josh
Title: Re: dataMine 2 documentation
Post by: tbully on July 14, 2017, 06:41:51 pm
Hello -

Hoping someone can give me a help on a new DM install on a VeraSecure.

I keep getting the "select USB" window during initialization.


lsusb Output
Code: [Select]
root@MiOS_55001685:/# lsusb
Bus 001 Device 011: ID 05dc:a817 Lexar Media, Inc.
Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0e8d:7662 MediaTek Inc.
Bus 001 Device 005: ID 04e2:1420 Exar Corp.
Bus 001 Device 006: ID 04e2:1420 Exar Corp.
Bus 001 Device 010: ID 1bc7:0026 Telit


FDISK Output
Code: [Select]
Disk /dev/sda: 15 GiB, 16039018496 bytes, 31326208 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xba0d28b4

Device    Boot Start       End   Blocks  Id System
/dev/sda1       2048  31324159 15661056   c W95 FAT32 (LBA)

I don't see anything mounted:
Code: [Select]
root@MiOS_55001685:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock6 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/mtdblock10 on /storage type jffs2 (rw,relatime)
/dev/mtdblock10 on /etc/cmh-firmware type jffs2 (rw,relatime)
/dev/mtdblock10 on /etc/cmh-backup type jffs2 (rw,relatime)
/dev/mtdblock9 on /mios type squashfs (ro,relatime)
root@MiOS_55001685:/#

But even if I mount manually to /datamine/ (directory manually created by me/root), it still doesn't seem to help.

Pointers?


-------

EDIT:
I saw I had a capital M in /datamine/.   I made that fix but still not mounting:

Code: [Select]
50 07/14/17 18:47:09.735 luup_log:337: dataMine: Initialising dataMine System (2.000) <0x76a31520>
06 07/14/17 18:47:09.736 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: true now: false #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:0 <0x76a31520>
06 07/14/17 18:47:09.736 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 skip: 0 v:0x1504b40/NONE duplicate:1 <0x76a31520>
06 07/14/17 18:47:09.737 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
50 07/14/17 18:47:09.766 luup_log:337: dataMine: Error reading tmpfile during mountcheck <0x76a31520>
06 07/14/17 18:47:09.767 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
06 07/14/17 18:47:09.767 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
50 07/14/17 18:47:09.794 luup_log:337: dataMine: Mounting dataMine storage (<span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 12.48px; background-color: rgb(240, 244, 247);">/dev/sda1</span>) to (/datamine/) <0x76a31520>
50 07/14/17 18:47:09.841 luup_log:337: dataMine: mount command output:'' <0x76a31520>
50 07/14/17 18:47:09.869 luup_log:337: dataMine: Error reading tmpfile during mountcheck <0x76a31520>
06 07/14/17 18:47:09.870 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
06 07/14/17 18:47:09.870 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
50 07/14/17 18:47:09.897 luup_log:337: dataMine: Mount point error: ::<span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 12.48px; background-color: rgb(240, 244, 247);">/dev/sda1</span> <0x76a31520>
06 07/14/17 18:47:09.898 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: errorStatus was: false now: true #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:0 <0x76a31520>
50 07/14/17 18:47:09.904 luup_log:337: dataMine: ERROR: Startup state is 'uninitialised'! <0x76a31520>
06 07/14/17 18:47:09.905 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 skip: 0 v:0x1504b40/NONE duplicate:1 <0x76a31520>
06 07/14/17 18:47:09.905 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x76a31520>
50 07/14/17 18:47:09.905 luup_log:337: dataMine: Reinitialising configuration structure <0x76a31520>
50 07/14/17 18:47:09.906 luup_log:337: dataMine: Startup complete <0x76a31520>
50 07/14/17 18:47:24.213 luup_log:337: dataMine: Delete backups complete - No files found <0x6f631520>
06 07/14/17 18:47:24.242 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskTotal was: 8192 now: 8192 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x6f631520>
06 07/14/17 18:47:24.243 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsed was: 7284 now: 7288 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:0 <0x6f631520>
06 07/14/17 18:47:24.243 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskFree was: 908 now: 904 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:0 <0x6f631520>
06 07/14/17 18:47:24.244 Device_Variable::m_szValue_set device: 337 service: urn:cd-jackson-com:serviceId:DataMine1 variable: diskUsedPcnt was: 89 now: 89 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x6f631520>
-2-===========================================================================
-3-===========================================================================
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock6 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/mtdblock10 on /storage type jffs2 (rw,relatime)
/dev/mtdblock10 on /etc/cmh-firmware type jffs2 (rw,relatime)
/dev/mtdblock10 on /etc/cmh-backup type jffs2 (rw,relatime)
/dev/mtdblock9 on /mios type squashfs (ro,relatime)
-4-===========================================================================
-5-===========================================================================
SetDataDirectory-/datamine/
SetMountUUID    -05dc-a817
SetMountPoint   -<span style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 12.48px; background-color: rgb(240, 244, 247);">/dev/sda1</span>
SetManualMount  -0
-6-===========================================================================
{"dbVersion":2,"Events":{"count":0,"last":0},"Version":"2.000","Variables":[],"Graphs":[],"guiConfig":[],"nextId":1}
-7-===========================================================================

-------
FINAL EDIT.  Delete this noise if you want!  (sorry)
I reread the message above my post and also checked the three check boxes.  All is well now.

Thanks! 
Title: Re: dataMine 2 documentation
Post by: thomasx on January 01, 2018, 08:24:14 pm
Hi

How can the scale range on the Dashboard dial be changed. I need to increase it quite a bit for my application?

Thanks, Thomasx
Title: Re: dataMine 2 documentation
Post by: tomtcom on March 24, 2018, 12:25:06 pm
Can anyone expand on the details around Online Status on page 5 of the documentation? I have the grey radio button.

?Online Status? indicator ? this is green if the web interface is communicating with Vera, and grey if
Vera is not communicating with the browser. It displays red if communication is OK, however there
are errors within dataMine (eg. Initialisation error, or error writing data to the USB).

Does it mean there is a problem or values might not be updating when grey? I can sit here and watch it but like most people we probably always want to see green. :)
Title: Re: dataMine 2 documentation
Post by: tomtcom on March 24, 2018, 12:26:48 pm
Hi

How can the scale range on the Dashboard dial be changed. I need to increase it quite a bit for my application?

Thanks, Thomasx

Check page 8 of the documentation, that might help you out.
Title: Re: dataMine 2 documentation
Post by: subrat35 on July 10, 2018, 05:37:14 am
Hello,
I am rather a new user of Vera.
I would like to log the data coming from Fibaro wall plugs (that work through Z-wave) connected to my Vera plus controller.
I got a USB successfully mounted on the USB drive and made it to log the data. Attached is a screenshot:

Now, while trying to install the DataMine plugin for Vera, the MiOS Marketplace "doesn't find any unit to connect with." Am I missing any steps?
My computer and the controller are on the same network and I can access the controller via putty and can also see the web interface of the controller.
Please suggest how to install the plugin and proceed with the data handling?