We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 266376 times)

Offline garrettwp

  • Master Member
  • *******
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: New dataMine graphing and logging plugin
« Reply #825 on: February 04, 2015, 10:35:04 pm »
Until the last day or so, the Vera edge had a bug which caused the USB port not to function. The latest firmware which was just released should fix that. If you have not upgraded the firmware in the last day, you'll hit this issue.

- Garrett


Offline dmckenna

  • Full Member
  • ***
  • Posts: 196
  • Karma: +8/-2
Re: New dataMine graphing and logging plugin
« Reply #826 on: February 05, 2015, 07:51:17 am »
Firmware fix still 'in test' according to Marc S @ Vera

@ 5th Feb

Offline jjwalker

  • Full Member
  • ***
  • Posts: 100
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #827 on: March 07, 2015, 02:00:28 pm »
I've been using dataMine with USB logging for a couple weeks with no issues.  They are both mounted on separate USB sticks.  The only issue I have is if the VeraLite reboots, it will detect the dataMine USB stick and reformat it for USB logging.  It's connected to a powered hub with three things connected. USB logging stick, dataMine USB stick, and an Arduino gateway.  I've experimented with the reformat issue by unplugging the Vera, then unplugging the dataMine stick.  Once Vera is booted up, and I see zero channels in the dataMine device, I plug the dataMine USB stick back in and hit Reload in UI5.  Once UI5 reloads, dataMine is back up with its' USB stick.   Is it possible to have a Startup Lua where it will prohibit the system from reformatting a USB stick with a certain UUID?  My case BE3F-E373. 

Thanks




Offline TwannieM

  • Hero Member
  • *****
  • Posts: 522
  • Karma: +3/-2
Re: New dataMine graphing and logging plugin
« Reply #828 on: March 10, 2015, 03:18:39 am »
This does not seem to run in UI7?

Offline Bozo

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #829 on: April 22, 2015, 12:29:57 am »
I can't excess to the usb stick on the Vera Edge, does anybody knows how to do this ?
I have tryed  "/dev/ttyUSB0/",  "/dev/ttyUSB0"  and  "/dev/sda/" by SetMountPoint  and SetDateDirectory "/dataMine"
I also format the usb stick with Fat 32 and make a directory called dataMine.

Offline solarshine

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #830 on: April 24, 2015, 11:04:02 am »
I think I have DataMine running properly on my Veralite and am trending data from my TED5000. I also added the Radio Thermostat CT30 "TemperatureSensor1" "CurrentTemperature" variable from two thermostats but it reports zero degrees from both.


Offline solarshine

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #831 on: April 24, 2015, 11:41:17 am »
Both temperatures are reporting now! Took some time that's all.

Any ideas how to uncover the humidity variable, not listed in the variables right now?

Offline Bozo

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #832 on: May 15, 2015, 08:03:46 am »
Who knows the configuration of the USB port from the Vera Edge for dataMine?
And how are the settings in the Advance/Variables tab ?
USB port works on the Vera, dataMine don't see it. I tryed many days with differend settings, but it won't work.

Offline VeraEdger

  • Full Member
  • ***
  • Posts: 113
  • Karma: +1/-0
Re: New dataMine graphing and logging plugin
« Reply #833 on: June 26, 2015, 01:28:05 pm »
Who knows the configuration of the USB port from the Vera Edge for dataMine?
And how are the settings in the Advance/Variables tab ?
USB port works on the Vera, dataMine don't see it. I tryed many days with differend settings, but it won't work.

Having the same issue. Did you fix it?

Offline sebohans

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #834 on: July 22, 2015, 08:08:52 am »
Having the same issue.
I wish there was a NOOB guide to mounting your USB succefully .
And knowing what to fill in as LOCAL_DATA_DIR for dataMine or dataYours.
I'm working with a USB hub wit a 16GB usb stick and a rtx433e transceiver.

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #835 on: September 05, 2015, 05:12:07 pm »
Hi.  I've been using datamine for many months now find it really useful.  One missing feature, however, is the ability to aggregate data over different time periods - e.g. have a graph of my average weekly gas usage over a period of a year without getting lost in the detailed logs.

I had a go at modifying the code and added hourly, daily, weekly and monthly aggregations to the config settings of each variable.  I also added various aggregation functions such as sum, difference, average, min and max.

I've only tested it in UI5 on a Veralite but if you want to have a go using it too, you will need to have datamine installed and working then download the attached file and install the two files as follows.  Be sure to back up the files you are replacing in case it goes wrong.

L_DataMine1.lua -  in Vera go to Apps>Develop Apps>Luup Files>Choose File and select L_DataMine1.lua, check the Restart Luup after upload box and click Go
datamine.min.js - using a tool such as WinSCP replace the existing file in the Vera directory www/dm/js.  You will need your Vera root password to do this.

Once complete, fire up the datamine url then look for the new config settings "Aggregate By (Period)" and "Aggregation Formula" in the config.

Typical usage scenarios:
  • graph how many hours a day your heating has been on for having logged the on off times
  • graph how many units of gas you use a month having logged the meter reading
  • graph the maximum daily temperature

** Attachment removed - see later post for most recent version of code **
« Last Edit: October 09, 2015, 09:23:44 am by ConstantSphere »
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #836 on: September 16, 2015, 12:53:10 pm »
Hi

I've also amended the code to make dataMine work over a remote connection.  To do this, install the attached files to your existing DataMine installation (as above):

L_DataMine1.lua and D_DataMine1.json -  in Vera go to Apps>Develop Apps>Luup Files>Choose File and select each file, check the Restart Luup after upload box and click Go
datamine.min.js - using a tool such as WinSCP replace the existing file in the Vera directory www/dm/js.  You will need your Vera root password to do this.

to get access across the public internet, go to cp.mios.com and log in with your existing account details and wait for the usual remote dashboard to come up.  This process will install a needed authentication cookie.  Then find the datamine graphing device and click on the spanner. Then click on the "dataMine graphing page" link which will take you to fwd8.mios.com/port_3480/data_request?id=lr_dmPage where datamine will load.  Alternatively you should be able to access it via fwd8.mios.com/<username>/<password>/<vera device number>/data_request?id=lr_dmPage.  You can still use <local vera ip>/dm for local access.

Whilst I was in the code, I also fixed a couple of bugs that caused it to not work properly in some browsers and some incorrect plotting of data points in certain edge cases.

*** update 2015-10-10 ***
I've fixed a bunch of bugs and improved the performance including fixing some random crashes that were in the original code

*** update 2015-11-01 ***
I've fixed some more bugs including handling of alpha values

*** update 2015-11-02 ***
fixed some broken images shown via remote access
« Last Edit: November 02, 2015, 03:00:25 pm by ConstantSphere »
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline kennedyj

  • Jr. Member
  • **
  • Posts: 50
  • Karma: +2/-0
Re: New dataMine graphing and logging plugin
« Reply #837 on: September 19, 2015, 10:14:14 am »
Sorry to bring up the USB mounting issue yet again, but I just cannot get dataMine to mount my USB drive. I have a vera3 and UI5.

I have a 8GB USB drive which I have formatted via Windows in FAT32 and included a top level folder "dataMine" - no joy, will not mount.

I tried formating the USB drive in ext3 with the same result.

In both cases I can SSH into my vera3 and manually mount the USB (/dev/sdb1) onto /mnt no problems

I'm slightly confused by the "SetMountPoint" variable which many people have mentioned. This not the mount point but is the location of the device that is trying to be mounted! Have I got this wrong.

I've been trying to get this working for 2 days but I have now run out of ideas  :(



Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #838 on: September 19, 2015, 02:04:59 pm »
can you paste the output of http://x.x.x.x:3480/data_request?id=lr_dmCtrl&control=debug somewhere?  where x.x.x.x is the ip address of your vera box.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline a-lurker

  • Hero Member
  • *****
  • Posts: 872
  • Karma: +66/-8
Re: New dataMine graphing and logging plugin
« Reply #839 on: September 19, 2015, 05:31:31 pm »
@kennedyj 
Quote
and included a top level folder "dataMine"

You do not create the dataMine directory on the stick. You create the directory under root in Vera. The plugin then mounts the stick to Vera at that directory. See:

http://forum.micasaverde.com/index.php/topic,16975.msg131509.html#msg131509

VeraEdge users have bigger problems and may want to look at this post:

http://forum.micasaverde.com/index.php/topic,33555.msg246245.html#msg246245

There is a bug in the plugin, that has never been fixed, that causes the dataMine directory to never be created.

@ConstantSphere  You have being a few doing dataMine code mods. Last time I looked, I thought this failure to create the dataMine directory, was due to this section of code. I never tried changing it - so unproven. But it's in this area somewhere here:

http://forum.micasaverde.com/index.php/topic,16243.msg131478.html#msg131478

The failure to create the directory results in the "...Error reading tmpfile during UUID check..." (likely) in the dataMine log web page. The file can't be read, because it can't be created in the function checkFreeSpace(), because the stick is not mounted.