We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 266382 times)

Offline kennedyj

  • Jr. Member
  • **
  • Posts: 50
  • Karma: +2/-0
Re: New dataMine graphing and logging plugin
« Reply #840 on: September 27, 2015, 03:04:33 pm »
Sorry for the delay in responding to you guys - been on a business trip.

Anyway, I manually created a top level directory /dataMine, and lo and behold the USB drive mounted automatically after a reboot  :)

Now I just need to figure out how I collect data from the HEM...

Thanks, JohnK.

Offline geniepro

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #841 on: October 31, 2015, 09:45:23 pm »
The latest version, what is included in dataMine 2015-10-10.zip, doesn't seem to work fine for non Number Values, not sure if this worked before - I just started implementing this.

I'm trying to graph ModeState from the thermostat - the values from vera are Idle, Cooling, Heating, please see below the content of /dataMine/database/1/raw/2391.txt:

Code: [Select]
446299663,Cooling
1446299896,Idle
1446300023,Cooling
1446300069,Idle
1446305776,Cooling
1446306286,Idle
1446321602,Cooling
1446321958,Idle
1446340737,Cooling
1446341036,Idle

When I try save a new Lookup with label 'New Label' and val 'Idle' the below debug messages are returned:

Code: [Select]
50 10/31/15 20:37:34.298 luup_log:15: dataMine: Saving: D[3] S[urn:micasaverde-com:serviceId:HVAC_OperatingState1] V[ModeState] <0x7402a520>
50 10/31/15 20:37:34.299 luup_log:15: dataMine: Graph Type = 1 <0x7402a520>
50 10/31/15 20:37:34.299 luup_log:15: dataMine: Save Lookup=[{"lab":"Heating","val":3},{"lab":"Cooling","val":1},{"lab":"New Label","val":"Idle"}] <0x7402a520>
06 10/31/15 20:37:34.367 Device_Variable::m_szValue_set device: 15 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 10 now: 10 #hooks: 0 upnp: 0 skip: 0 v:0xcc4a90/NONE duplicate:1 <0x7402a520>
06 10/31/15 20:37:34.368 Device_Variable::m_szValue_set device: 15 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 10 now: 10 #hooks: 0 upnp: 0 skip: 0 v:(nil)/NONE duplicate:1 <0x7402a520>
50 10/31/15 20:37:34.370 luup_log:15: dataMine: {"Variables":[{"Ghost":false,"Service":"urn:micasaverde-com:serviceId:HVAC_OperatingState1","AggregateBy":0,"DrowsyError":0,"AggregationType":0,"LastRec":1446341036,"Units":"0","Device":3,"FilterMaximum":0,"Logging":1,"Type":1,"FilterEnable":0,"DataType":0,"Id":1,"FilterMinimum":0,"Alpha":1,"LastHistory":1104516000,"DrowsyWarning":0,"DataOffset":0,"Name":"UThermostat - ModeState","LastVal":"Idle","FirstRec":0,"Variable":"ModeState","historyState":0,"Lookup":{"Heating":3,"Cooling":1}}]} <0x7402a520>

If I interpret correctly the logs the request is received but for some reason is ignored, it responds with the original lookup ""Lookup":{"Heating":3,"Cooling":1}"

Am I doing something wrong ?

Thank you.

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #842 on: November 01, 2015, 03:07:16 am »
I'm pretty sure it used to work, so I suspect I've broken it. I'll take a look in the next couple of days.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #843 on: November 01, 2015, 06:32:35 pm »
Actually, what you need to do is set your label to the recorded data (e.g. Idle) and your value to a unique number on the graph Y-axis (e.g. 0). You cannot store non-numerics in the value field - nor does it make sense to do this. In this example, the recorded state of Idle will show in position 0 on the Y-axis which will be re-labelled as "Idle", you can then set "Cooling" to value 1 and "Heating" to value 2 etc.

Having said that, I also noticed a bug in the way my code handles alpha values and will upload a new version with a fix.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline geniepro

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #844 on: November 01, 2015, 11:12:29 pm »
Thanks @ConstantSphere I tried that as well but I can't graph, it will always show only the current state - in a continuous line.

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #845 on: November 02, 2015, 04:30:58 am »
Hi @geniepro. I've tried your input file with the latest version of the code (that contained a fix for graphing alpha values) and it seems to work for me.  Can you confirm the following:
1. You have the latest version of the code I uploaded the other night (dataMine 2015-11-01.zip a few posts back)
2. Properties for the variable are set as follows: Data Offset 0, Data Multiplier 1, Aggregate By (Period) None (raw data), Aggregation Formula Sample (default).

If it still doesn't work, can you post a link to the output of your debug page at http://<vera IP>/port_3480/data_request?id=lr_dmCtrl&control=debug
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline geniepro

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #846 on: November 02, 2015, 08:45:36 am »
Thank you @ConstantSphere it works now.

This time I logged in through "/port_3480/data_request?id=lr_dmPage" and I get broken images in the "Configuration" tab. The img tag is:

<img src="images/light-bulb-small-off.png">

for the working images the tag is:

<img src="?id=lr_dmPage&amp;page=images/exclamation-small.png">

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #847 on: November 02, 2015, 03:03:38 pm »
Well spotted! I've coded a potential fix to the broken images issue (dataMine 2015-11-02.zip a few posts back).
Thanks for all the feedback.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline geniepro

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #848 on: November 09, 2015, 08:40:10 am »
Sorry for the late feedback.

The images work perfectly now.

Since my understanding is that the graphs should be working while accessing the URL remotely I tried to access:

Code: [Select]
https://vera-us-oem-relay41.mios.com/port_3480/data_request?id=lr_dmPage
but I get:

Code: [Select]
ERROR:Module not found:port_3480
I access it the same way as when I'm local, by clicking on "dataMine graphing page"

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #849 on: November 09, 2015, 10:41:52 am »
The best way to access the site remotely is to log into you mios account here: https://cp.mios.com/login.php and navigate through to the datamine device and click on the "dataMine graphing page" link.  By going in through this route an authentication cookie will be set on your machine which will allow access to the site.  In my case this takes me to https://fwd8.mios.com/port_3480/data_request?id=lr_dmPage, as the forwarding server is dependant on your location. Strictly speaking the port_3480 is not needed in the URL for remote access as remote access will only forward on this port anyway - it's there to make the local access work.

Alternatively you can connect directly using the following URL:
https://<forwarding server>/<user name>/<password>/<device id>/data_request?id=lr_dmPage which avoids the need to have the authentication cookie.  This is the username and password you use to log into mios with and the device id that your normally see at the top of your vera screen. As it is via a secure connection your username and password will be transmitted encrypted. E.g. this might look like: https://vera-us-oem-relay41.mios.com/username/password/12345678/data_request?id=lr_dmPage

Let me know if it works...
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline geniepro

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #850 on: November 30, 2015, 10:14:42 am »
I tried your recommendation but I get "ERROR:Module not found:USERNAME", note that I'm using UI7. That's fine. I don't really need remote access, just wanted to mention it.

Offline ConstantSphere

  • Beta Testers
  • Sr. Member
  • *****
  • Posts: 272
  • Karma: +16/-0
Re: New dataMine graphing and logging plugin
« Reply #851 on: November 30, 2015, 11:18:01 am »
I've only tested it on UI5 and I know that UI7 uses a completely different access system to UI5 so I'm not surprised it doesn't work.  I'll take another look at the code when I feel brave enough to upgrade to UI7.
Vera Lite (UI5), 3xFibaro Universal Dimmers, Fibaro Roller Shutter, 2xFibaro Relay, 2xNorthQ Data Loggers, Secure Wall Thermostat, Secure Boiler Receiver, Remotec AC IR extender

Offline 5546doug

  • Newbie
  • *
  • Posts: 19
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #852 on: November 30, 2015, 06:41:01 pm »
when I access datamine from my laptop in home lan/wan network, the  vera dashboard all is well with the plugin I can get to the graph

 page and do all that I wish.

My question is when remote to the home and trying to do the same thing all is right with the exception of when on data page and hit get grafe, I get page" Not Found"     The requested URL /dm/index.html was not found on this server"

Why is this and how can I fix this so I can view the d/m graphs remotely?

Thanks for some help in advance Doug

Offline edraket

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #853 on: December 08, 2015, 02:12:16 pm »
Does somebody use the embedded graphs option from within Imperihome? Using the reference to the specific graph, I don't get the graph.

Also, it doesn't work from my desktop browser in a LAN/WLAN? I had it working in earlier MIOS version (5 and 6). Now being on 7, could that be a reason?

I'm using this as the reference:
100.100.100.10/dm/graph.html?ref=sensorgraph where 100.100.100.10 is the IP of my vera light. Entering this only shows the waiting spinner of datamine.

Thanks, Ed

Offline TE-Rex

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #854 on: December 29, 2015, 05:37:43 pm »
I am using datamine Graph to show data from a rain sensor. I am currently using graph type Spline but is not fully satisfied with the graph.

High charts has the graph type "column" which is nice for visualizing rain fall data.
 
This link shows an example of that:
http://www.highcharts.com/demo/column-basic

Any plans to add this graph type in the future? Hoping...  :)