We have moved at community.getvera.com

Author Topic: dataMine 2 documentation  (Read 22782 times)

Offline RWoerz

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #15 on: September 23, 2016, 09:08:49 pm »
dataMine 2 and Linux for dummies like me. 

Not being a Linux expert I've struggled for a while trying to get dataMine 2 working on my Very Edge UI7. Starting but not limited to the fact that when you plug the first USB drive into a Vera no matter what size the drive is it creates one 500MB partition which isn't easy to undo.

I've even spent time with Vera support asking if the 2nd USB drive had mounted and according to them everything was working as expected, whatever that means. I'm sure had I known Linux this would have been easy but there is so much info out there it's hard to know what's current and what's not.

One big issue for me is the way many people who understand Linux post things. One post showed the output of what was said to be a "file" command. The output was interesting but they didn't show the command itself so it was of little help.   

If the folks at Vera are reading this it would be very helpful to have a tab that gives some basic systems info. Like what drives are mounted, how full are they, IP address, when a sensor was last seen in a human readable form not Epoch time. Epoch time, that's Linux speak for the number of seconds since midnight Jan 1st 1970, I think that's right.

Some posts say you can have Vera logging turned on some say no. Anyway I'm still not sure exactly what was needed but I'll give you what I did and hopefully someone can tell us what wasn't.

One note to start, I'll show all Linux and Vera commands in quotes i.e. "lsusb" so you don't type in the quotes just what's between them. Also if you don't know this Linux is case sensitive so be aware type everything in as shown.

The whole DataMine thing may just work for you through the Vera interface but that wasn't what happened with me.

First thing I would plug in an old 1GB USB drive then turn logging on in the Vera "Settings, Logs" tab. Vera will grab hold of it and hopefully not think the next drive you plug in belongs to it.

Second you'll need to download a program called PuTTY www.putty.org/, it allows you to "terminal" into the Vera box and run Linux commands in a DOS like window. Unix/Linux was around long before DOS and is a bit of a steep learning curve but to get what you need for dataMine you'll only need a few commands.
Be forewarned you can KILL your system with the wrong command here so be very careful what you type in.   

When you run PuTTY you'll type in the IP address of you Vera box, you'll find the IP at the top of your browser page when you're logged into your Vera. Next make sure the port number is 22 and SSH is selected, then press the Open button.

If all is working a DOS like window will open and after several seconds a username prompt will appear. Type in "root" and press return. Next a password prompt will appear, my password was printed on a label on the bottom of my Vera, yours may vary. Type in your password and hit return. If all is well you've just logged into Linux on your Vera machine. I don't think it drops you into the root directory so to make sure your are type in "cd /", (cd space forward slash), that should put you at the root directory of your machine.

One problem I found with the version Linux installed on the Vera is that it doesn't support many of the traditional tools supported on many Linux system. So if you Google for a command to show the UUID of a USB drive most of the commands you'll find don't work here. All of the commands I show here worked on my Vera.

First you'll need to get the UUID of your USB drive. Before you plug in your new dataMine logging USB drive type in the following command "fdisk -l" (lower case L). At the end of the output you'll see something like this which is my 2GB Vera logging drive with its 500MB partition.

 Disk /dev/sdb: 2003 MB, 2003828736 bytes
62 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1   *           1         273      524705+ 83 Linux
/dev/sdb2             274        1018     1431890  83 Linux
root@MiOS_45011987:~#

Now plug in your dataMine drive and run the same command "fdisk-l". You should see the newly mounted drive. Mine is a 16GB USB drive and it looks like this.

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1        1947    15632384   c Win95 FAT32 (LBA)

The important thing to note here is the /dev/sda1 which you'll type into Vera's dataMine SetMountPoint. Yours may be different so write it down!

Next type in "lsusb" this will give you your drives UUID and should look something like this. Mine is the Cruzer and its UUID is 0781-5530, write yours down!

root@MiOS_45011987:~# lsusb
Bus 001 Device 002: ID 050d:0237 Belkin Components F5U237 USB 2.0 7-Port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 005: ID 8644:800a
root@MiOS_45011987:~#



Next install dataMine 2 on your Vera but make sure to leave the PuTTY window open, go to Vera's "Apps, Install apps" tab then search for dataMine. There will be two pick dataMine 2. After the install is complete go to Vera's Devices tab and find the dataMine 2 graphing device and open it. Then open the Advanced tab. Near the top of the page you'll find 4 tabs, Parms, Variables, New service and Commands. Select the Variables tab. Under the Variables tab add the following information. In SetDataDirectory type in "/dataMine/", after you type that in you have to right click in any of the white space on the page to update the information, strange but it's the way you have to do it. In SetMountUUID you'll have to type in your USB drives UUID. Then last but not least in SetMountPoint type in the mount point for your system each time right clicking in the white area in the web page. It will flash on the screen that the data has been updated.

The SetDataDirectory crates a dataMine folder in the root directory. The SetMountUUID and SetMountPoint attaches your USB drive to the dataMine folder.  Now go back to the PuTTY window and type in "df -h". It should show your drive attached to /dataMine directory and the used and available space.
   
root@MiOS_45011987:/dataMine# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    9.4M      5.3M      4.1M  56% /
/dev/root                 9.5M      9.5M         0 100% /rom
tmpfs                    61.9M      3.4M     58.5M   5% /tmp
/dev/mtdblock6            9.4M      5.3M      4.1M  56% /overlay
overlayfs:/overlay        9.4M      5.3M      4.1M  56% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /storage
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-firmware
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-backup
/dev/mtdblock9            7.8M      7.8M         0 100% /mios
/dev/sda1                14.9G    184.0K     14.9G   0% /dataMine

At this point you should be able in Vera to go to the Devices, dataMine 2 graphing, dataMine tab and select the "dataMine graphing page (local access)" and bring up a new tab in your browser called dataMine - graphing. If it asked select the USB Drive with the correct UUID. Under the Configuration tab add the sensors you want to graph.  If you want to see where the data files for the sensors get built type in the PuTTY window "cd /dataMine" then "du -h". That's all I got hope it helped. Please if I messed something up PLEASE let me know.

root@MiOS_45011987:~# cd /dataMine
root@MiOS_45011987:/dataMine# du -h
16.0K   ./database/1/raw
32.0K   ./database/1
16.0K   ./database/2/raw
32.0K   ./database/2
8.0K    ./database/3/raw
24.0K   ./database/3
96.0K   ./database
184.0K  .
« Last Edit: September 23, 2016, 09:33:21 pm by RWoerz »

Offline smartloft

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #16 on: October 12, 2016, 02:10:16 am »
hi guys,

i have an issue with Datamine 2 resetting its configuration a couple of days ago

Is there any way we can access or download the data from the USB drive?

2 weeks worth of data were logged into the USB drive and we do not knw if there is a way to retrieve this data

help!

Offline loumarjr

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #17 on: October 16, 2016, 05:52:44 pm »
Could someone explain the easiest way to access dataMime 2.0 remotely from the VeraEdge on UI7 Im not sure where to place the "?id=lr_dmPage".  Thank you!!

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #18 on: October 17, 2016, 04:07:52 am »
Could someone explain the easiest way to access dataMime 2.0 remotely from the VeraEdge on UI7 Im not sure where to place the "?id=lr_dmPage".  Thank you!!
Hi,
You cannot via the Vera portal. You can only do that via ALTUI portal. Details of that you can find in the ALTUI forum articles.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline loumarjr

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #19 on: October 17, 2016, 10:52:39 am »
Thanks Rene,

I have ALTUI but i cant figure out how to remotely accesss datamine on my Vera Edge.  Would you happen to have a line to the article or post?  I have been searching for ever.  Thank you!!

Lou

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: dataMine 2 documentation
« Reply #20 on: October 17, 2016, 12:22:07 pm »
Hi,

It is fairly simple. Log-on via the ALTUI portal https://vera-ui.strongcubedfitness.com/Veralogin.php, select your Vera and wait for the page to load.

Then in the URL look for the ?id=lr_ALTUI..... and remove everything after the =lr_ (ALTUI...) and add dmPage. Hit enter.

Too funny, I am just trying the Open button you see on the DM2.0 plugin on ALTUI and that launched DataMine just fine. I now I played with this in the past so I cannot recall it is just my version of if it was build in by amg0. Anyhow, if that doesn't work, the above will.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline smartloft

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #21 on: October 17, 2016, 08:50:38 pm »
Hi RWoerz,

thanks for the in-depth explanation

however, how does one extract the data file from /dataMine directory?

is it viewable under windows?







a question, how does one copy
dataMine 2 and Linux for dummies like me. 

Not being a Linux expert I've struggled for a while trying to get dataMine 2 working on my Very Edge UI7. Starting but not limited to the fact that when you plug the first USB drive into a Vera no matter what size the drive is it creates one 500MB partition which isn't easy to undo.

I've even spent time with Vera support asking if the 2nd USB drive had mounted and according to them everything was working as expected, whatever that means. I'm sure had I known Linux this would have been easy but there is so much info out there it's hard to know what's current and what's not.

One big issue for me is the way many people who understand Linux post things. One post showed the output of what was said to be a "file" command. The output was interesting but they didn't show the command itself so it was of little help.   

If the folks at Vera are reading this it would be very helpful to have a tab that gives some basic systems info. Like what drives are mounted, how full are they, IP address, when a sensor was last seen in a human readable form not Epoch time. Epoch time, that's Linux speak for the number of seconds since midnight Jan 1st 1970, I think that's right.

Some posts say you can have Vera logging turned on some say no. Anyway I'm still not sure exactly what was needed but I'll give you what I did and hopefully someone can tell us what wasn't.

One note to start, I'll show all Linux and Vera commands in quotes i.e. "lsusb" so you don't type in the quotes just what's between them. Also if you don't know this Linux is case sensitive so be aware type everything in as shown.

The whole DataMine thing may just work for you through the Vera interface but that wasn't what happened with me.

First thing I would plug in an old 1GB USB drive then turn logging on in the Vera "Settings, Logs" tab. Vera will grab hold of it and hopefully not think the next drive you plug in belongs to it.

Second you'll need to download a program called PuTTY www.putty.org/, it allows you to "terminal" into the Vera box and run Linux commands in a DOS like window. Unix/Linux was around long before DOS and is a bit of a steep learning curve but to get what you need for dataMine you'll only need a few commands.
Be forewarned you can KILL your system with the wrong command here so be very careful what you type in.   

When you run PuTTY you'll type in the IP address of you Vera box, you'll find the IP at the top of your browser page when you're logged into your Vera. Next make sure the port number is 22 and SSH is selected, then press the Open button.

If all is working a DOS like window will open and after several seconds a username prompt will appear. Type in "root" and press return. Next a password prompt will appear, my password was printed on a label on the bottom of my Vera, yours may vary. Type in your password and hit return. If all is well you've just logged into Linux on your Vera machine. I don't think it drops you into the root directory so to make sure your are type in "cd /", (cd space forward slash), that should put you at the root directory of your machine.

One problem I found with the version Linux installed on the Vera is that it doesn't support many of the traditional tools supported on many Linux system. So if you Google for a command to show the UUID of a USB drive most of the commands you'll find don't work here. All of the commands I show here worked on my Vera.

First you'll need to get the UUID of your USB drive. Before you plug in your new dataMine logging USB drive type in the following command "fdisk -l" (lower case L). At the end of the output you'll see something like this which is my 2GB Vera logging drive with its 500MB partition.

 Disk /dev/sdb: 2003 MB, 2003828736 bytes
62 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1   *           1         273      524705+ 83 Linux
/dev/sdb2             274        1018     1431890  83 Linux
root@MiOS_45011987:~#

Now plug in your dataMine drive and run the same command "fdisk-l". You should see the newly mounted drive. Mine is a 16GB USB drive and it looks like this.

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1        1947    15632384   c Win95 FAT32 (LBA)

The important thing to note here is the /dev/sda1 which you'll type into Vera's dataMine SetMountPoint. Yours may be different so write it down!

Next type in "lsusb" this will give you your drives UUID and should look something like this. Mine is the Cruzer and its UUID is 0781-5530, write yours down!

root@MiOS_45011987:~# lsusb
Bus 001 Device 002: ID 050d:0237 Belkin Components F5U237 USB 2.0 7-Port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 005: ID 8644:800a
root@MiOS_45011987:~#



Next install dataMine 2 on your Vera but make sure to leave the PuTTY window open, go to Vera's "Apps, Install apps" tab then search for dataMine. There will be two pick dataMine 2. After the install is complete go to Vera's Devices tab and find the dataMine 2 graphing device and open it. Then open the Advanced tab. Near the top of the page you'll find 4 tabs, Parms, Variables, New service and Commands. Select the Variables tab. Under the Variables tab add the following information. In SetDataDirectory type in "/dataMine/", after you type that in you have to right click in any of the white space on the page to update the information, strange but it's the way you have to do it. In SetMountUUID you'll have to type in your USB drives UUID. Then last but not least in SetMountPoint type in the mount point for your system each time right clicking in the white area in the web page. It will flash on the screen that the data has been updated.

The SetDataDirectory crates a dataMine folder in the root directory. The SetMountUUID and SetMountPoint attaches your USB drive to the dataMine folder.  Now go back to the PuTTY window and type in "df -h". It should show your drive attached to /dataMine directory and the used and available space.
   
root@MiOS_45011987:/dataMine# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    9.4M      5.3M      4.1M  56% /
/dev/root                 9.5M      9.5M         0 100% /rom
tmpfs                    61.9M      3.4M     58.5M   5% /tmp
/dev/mtdblock6            9.4M      5.3M      4.1M  56% /overlay
overlayfs:/overlay        9.4M      5.3M      4.1M  56% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /storage
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-firmware
/dev/mtdblock10          64.8M      9.9M     54.8M  15% /etc/cmh-backup
/dev/mtdblock9            7.8M      7.8M         0 100% /mios
/dev/sda1                14.9G    184.0K     14.9G   0% /dataMine

At this point you should be able in Vera to go to the Devices, dataMine 2 graphing, dataMine tab and select the "dataMine graphing page (local access)" and bring up a new tab in your browser called dataMine - graphing. If it asked select the USB Drive with the correct UUID. Under the Configuration tab add the sensors you want to graph.  If you want to see where the data files for the sensors get built type in the PuTTY window "cd /dataMine" then "du -h". That's all I got hope it helped. Please if I messed something up PLEASE let me know.

root@MiOS_45011987:~# cd /dataMine
root@MiOS_45011987:/dataMine# du -h
16.0K   ./database/1/raw
32.0K   ./database/1
16.0K   ./database/2/raw
32.0K   ./database/2
8.0K    ./database/3/raw
24.0K   ./database/3
96.0K   ./database
184.0K  .

Offline RWoerz

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #22 on: November 08, 2016, 03:14:34 pm »
New question. OK call me blind but in the DataMine2's PDF on page 16 it says: "If update is specified, the graph will automatically be updated periodically. The update time is in minutes."

I don't seem to be able to find this variable in Vera or the WEB page. The "Display real-time graph" is grayed out". 

Offline korttoma

  • Hero Member
  • *****
  • Posts: 729
  • Karma: +26/-5
Re: dataMine 2 documentation
« Reply #23 on: December 15, 2016, 01:30:13 pm »
Did the 7.0.19 firmware brake the cifs on the VeraPlus or is it just me?

50   12/15/16 22:01:03.706   luup_log:205: dataMine: Error reading tmpfile during mountcheck <0x77101520>
06   12/15/16 22:01:03.706   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
06   12/15/16 22:01:03.707   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
50   12/15/16 22:01:03.728   luup_log:205: dataMine: Mounting dataMine storage (//192.168.1.2/home/datamine) to (/dataMine/) <0x77101520>
50   12/15/16 22:01:03.770   luup_log:205: dataMine: mount command output:'mount: mounting //192.168.1.2/home/datamine on /dataMine/ failed: No such device
50   12/15/16 22:01:03.795   luup_log:205: dataMine: Error reading tmpfile during mountcheck <0x77101520>
06   12/15/16 22:01:03.796   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
06   12/15/16 22:01:03.796   Device_Variable::m_szValue_set device: 205 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was:  now:  #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x77101520>
50   12/15/16 22:01:03.817   luup_log:205: dataMine: Mount point error: :://192.168.1.2/home/datamine <0x77101520>
« Last Edit: December 15, 2016, 03:08:04 pm by korttoma »
- Tomas

Offline korttoma

  • Hero Member
  • *****
  • Posts: 729
  • Karma: +26/-5
Re: dataMine 2 documentation
« Reply #24 on: December 16, 2016, 02:14:34 am »
I tried reverting to the previous VeraPlus  firmware 7.0.17 (1.7.2138) but I stil cannot mount cifs.
Just checked the files on my NAS and the last time the raw data file was updated was around the time I upgraded my VeraPlus to 7.0.19 (1.7.2414).
Not sure how to debug the situation. Anyone available to assist?
« Last Edit: December 16, 2016, 02:37:14 am by korttoma »
- Tomas

Offline korttoma

  • Hero Member
  • *****
  • Posts: 729
  • Karma: +26/-5
Re: dataMine 2 documentation
« Reply #25 on: December 16, 2016, 09:02:55 am »
managed to get it back up and running by issuing the command presented by @mfp in this post -> http://forum.micasaverde.com/index.php/topic,38219.0.html

« Last Edit: December 16, 2016, 01:42:16 pm by korttoma »
- Tomas

Offline mfp

  • Full Member
  • ***
  • Posts: 103
  • Karma: +3/-2
Re: dataMine 2 documentation
« Reply #26 on: December 16, 2016, 01:04:35 pm »
Had the same after updating, they were missing. Probably the kernel modules from firmware 7.0.19 have been updated, then a reinstall is needed and normal.

Offline Peren

  • Sr. Newbie
  • *
  • Posts: 31
  • Karma: +0/-0
Re: dataMine 2 documentation
« Reply #27 on: January 13, 2017, 09:06:34 am »
How can you change the Graph theme under graph options?

This is not a drop down. is there any documentation over configurable options?

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: dataMine 2 documentation
« Reply #28 on: February 12, 2017, 04:16:10 pm »
@Peren - you can type the name of the theme as free text in the Graph Options. Available options are on page 17 of the documentation available from http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf

for convenience it says...

Graph Theme ? sets the theme of the graphing area, controlling colours, grid lines, borders, spacing and the like. There are now 4 new themes: classic (default), gray, grid-light and sandsignika. Further themes can be downloaded from https://github.com/highcharts/highcharts/tree/master/js/themes and saved in the /www/dm/themes directory. You will need to reload the GUI (usually press F5) after changing the value to see the new theme.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: dataMine 2 documentation
« Reply #29 on: February 12, 2017, 04:20:52 pm »
@RWoerz

the manual is referring to the page http://vera-ip/dm/graph.html?ref=<quickref>&period=<days>&update=<mins> where the data can be updated in real time and not the main dataMine gui.  The auto update feature on the main dataMine gui is currently broken and as you note the button is greyed out and disabled.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender