We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 245163 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #60 on: November 12, 2011, 08:44:03 am »
Quote
OK - this might be progress - there is lzo appended on the file in both locations - as is the case with the other files in each directory. Are you saying it should be without the lzo?
On my system, there is no lzo on the LUA files - it is appended to the other files though (i.e. the XML/json files). I don't know the mechanism that the files end up with LZO - it's a compression system I think, so I guess somewhere in the Vera code, it does this, but I don't know why LUA files don't have this. We'd need MCV to comment on this (I've asked them to take a look at this thread).
You could try renaming the file to remove the LZO, but I think this won't work. Alternatively (possibly better) is to upload a copy of the file into the /etc/cmh-ludl directory using WinSCP and see what happens.

I assume you're still getting the error with the continuous "Error during LUA startup" message?

Quote
I can open using the spanner, and have four tabs - datamine, graph, config and advanced
This at least means it's loading the other files ok. This stuff is defined in the JSON file.

Quote
Then I can click back into graph and it shows all the menu items, but clicking "select channels" brings up the yellow box has no data in it.
The yellow box etc is generated by the javascript, so again, this file is being read. The population of the list requires the LUA code to be running.

Quote
I also ran a search for the file and it only sits in .etc/cmh-ludl/ and .overlay/etc/cmh-ludl/ with content appearing to be exactly the same in both cases.
Just a comment on this - I think that there's only 1 file, but Linux has the ability to link/map directorys into multiple places so that they appear twice (or more). I haven't tried to confirm this, but I think MCV has a single directory (possibly the overlay diectory) where they store all the Vera specific stuff, and they then map it across to the appropriate place on startup. (that's just an educated guess - not confirmed). I think when you upload a file to one place, you'll find it in both...




Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #61 on: November 12, 2011, 09:06:57 am »
Quote
On my system, there is no lzo on the LUA files - it is appended to the other files though (i.e. the XML/json files). I don't know the mechanism that the files end up with LZO - it's a compression system I think, so I guess somewhere in the Vera code, it does this, but I don't know why LUA files don't have this. We'd need MCV to comment on this (I've asked them to take a look at this thread).
You could try renaming the file to remove the LZO, but I think this won't work. Alternatively (possibly better) is to upload a copy of the file into the /etc/cmh-ludl directory using WinSCP and see what happens.
................................
I assume you're still getting the error with the continuous "Error during LUA startup" message?

I've loaded a copy of the file into each directory, and refreshed, which has got rid of the Error during LUA startup message - which seems to have worked.  :D

Quote
The yellow box etc is generated by the javascript, so again, this file is being read. The population of the list requires the LUA code to be running.

So assuming we're up and running, does data show straight away, or is it after a period of time?

Quote
Just a comment on this - I think that there's only 1 file, but Linux has the ability to link/map directorys into multiple places so that they appear twice (or more). I haven't tried to confirm this, but I think MCV has a single directory (possibly the overlay diectory) where they store all the Vera specific stuff, and they then map it across to the appropriate place on startup. (that's just an educated guess - not confirmed). I think when you upload a file to one place, you'll find it in both...

OK - I just refreshed both directories in WinSCP and files are still there, same size as when I pasted them in.

Thanks, Chris, I appreciate you guiding me thru this. It's great to have the forum, and help within it - I had a houseful of x10 gear a few years ago and there was nothing like this. Thanks again  :)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #62 on: November 12, 2011, 09:52:09 am »
Quote
I've loaded a copy of the file into each directory, and refreshed, which has got rid of the Error during LUA startup message - which seems to have worked.  :D
That's great news - I'm now interested to find out why your system seems to have added the lzo to the files where mine didn't! I'd have to assume that the other user who had the same error was also due to this. Hopefully MCV can comment.

Quote
So assuming we're up and running, does data show straight away, or is it after a period of time?
You should now be able to go into the config tab, and select some variables to monitor. When you click on them, they should go green which means that the LUA file is really running. Also, when you do this you should get the dataMineConfig.json file written into the /dataMine directory.
With that done, you will be recording data every time the variable changes, and you should get data files written into the /dataMine directory, and you should be able to graph them out. If you record something like a lightswitch status or power level, then you can turn it on and off a few times, and you should be able to confirm everything is working.

Quote
Thanks, Chris, I appreciate you guiding me thru this
No problem - it looks like we're nearly there, and it's always interesting to get to the bottom of a problem.

Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #63 on: November 12, 2011, 06:22:00 pm »
Quote
That's great news - I'm now interested to find out why your system seems to have added the lzo to the files where mine didn't! I'd have to assume that the other user who had the same error was also due to this. Hopefully MCV can comment.

Yes, seems strange - I know I (eventually) followed the directions to the tee - I'm the one in the family who reads instructions on all things!. The lzo issue may well be the same for the lost member, too.

Quote
You should now be able to go into the config tab, and select some variables to monitor. When you click on them, they should go green which means that the LUA file is really running. Also, when you do this you should get the dataMineConfig.json file written into the /dataMine directory.
With that done, you will be recording data every time the variable changes, and you should get data files written into the /dataMine directory, and you should be able to graph them out. If you record something like a lightswitch status or power level, then you can turn it on and off a few times, and you should be able to confirm everything is working.

Ahaa - that, too is working - variables selected, all good. It didn't seem to kick in straight away last night but all good this morning.

Quote
No problem - it looks like we're nearly there, and it's always interesting to get to the bottom of a problem.

True - as I said earlier, big learning curve for me - and from what I've read, you're only new-ish to the system/code too - great work, Chris - this is a really useful group of functions! Hopefully I can reduce our power bill to a more reasonable level.

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #64 on: November 13, 2011, 08:50:26 am »
Hi Chris. Thanks for the hard work with your dataMine code. I installed today - clear instructions and all v. straightforward, thanks for that...

For starters I'm monitoring a Horstmann HRT4-ZW and ASR-ZW thermostat and boiler actuator.

I have logging enabled for 3 channels - thermostat setpoint and current temperature, and boiler actuator operating mode.

Having changed setpoints / increased and decreased temp. I can see the log files being created and have a noddy graph :) :)

Q. However one problem I have is that the boiler actuation mode is logged textually -

Code: [Select]
1321190664,HeatOn
1321190884,Off
1321191080,HeatOn
1321191097,Off
1321191129,HeatOn
1321191528,Off

This seems to be playing havoc with the axis limits when I try to graph it.

I'd also like to be able to override the x,y axis limits as currently it seems to be auto-generating them between the minimum and maximum logged values?

Any thoughts on how to deal with this?

Thanks!

Alex
« Last Edit: November 13, 2011, 08:52:00 am by ajlennon »

Offline mcvflorin

  • Administrator
  • Hero Member
  • *****
  • Posts: 1755
  • Karma: +11/-3
Re: New dataMine graphing and logging plugin
« Reply #65 on: November 14, 2011, 05:36:28 am »
@willstar

What firmware version do you have?

Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #66 on: November 14, 2011, 07:47:39 am »
@willstar

What firmware version do you have?

It tells me it's the latest version, 1.1.1350 .

Offline mcvflorin

  • Administrator
  • Hero Member
  • *****
  • Posts: 1755
  • Karma: +11/-3
Re: New dataMine graphing and logging plugin
« Reply #67 on: November 14, 2011, 08:07:32 am »
There were some changes in our internal repository, and somehow some builds used the cgi-bin files (like the one that uploads the files on Vera) from the UI5 repository. 1350 is one of the builds affected. The only difference between these files is that the UI4 file doesn't compress the Lua files, but the UI5 file does.

To fix this, SSH into Vera and delete /www/cgi-bin/cmh/upload_upnp_file.sh. Then, using a program like WinSCP, put the attached file in its place.

Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #68 on: November 14, 2011, 08:23:41 am »
There were some changes in our internal repository, and somehow some builds used the cgi-bin files (like the one that uploads the files on Vera) from the UI5 repository. 1350 is one of the builds affected. The only difference between these files is that the UI4 file doesn't compress the Lua files, but the UI5 file does.

To fix this, SSH into Vera and delete /www/cgi-bin/cmh/upload_upnp_file.sh. Then, using a program like WinSCP, put the attached file in its place.

OK, done - thanks for your help on this

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #69 on: November 16, 2011, 04:49:25 am »
Quote
Q. However one problem I have is that the boiler actuation mode is logged textually

For anybody who runs into this issue, I was using HVAC_UserOperatingMode1 with ModeStatus. If instead I use SwitchPower1 with Status I can retrieve a 0,1 boolean value which gives me the boiler heating status...

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #70 on: November 16, 2011, 08:06:49 am »
You could transform a non-numerical UPnP variable to a numerical one by luup.variable_watching the non-numerical UPnP variable and creating/setting a user defined numerical variable - but I'm not sure whether the dataMine plugin would list user defined variables that are not part of an UPnP device specification.

The GDL/GDP plugin does support non-numerical variables.

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #71 on: November 16, 2011, 08:12:58 am »

Thanks Ap15e. That's helpful to know.

Offline utz

  • Sr. Member
  • ****
  • Posts: 274
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #72 on: November 16, 2011, 08:22:51 am »
but I'm not sure whether the dataMine plugin would list user defined variables that are not part of an UPnP device specification.

it does (if I understand correctly). I monitor variables that I put in as a hack which are not properly part of the specification; but they are listed and it works.

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #73 on: November 16, 2011, 03:40:36 pm »
Code: [Select]
local MY_NUMERICAL_VARIABLES_SID = 'urn:my-numerical-variables'
local HVACO_SID                  = 'urn:upnp-org:serviceId:HVAC_UserOperatingMode1'
local HVACO_DEVICE_ID            = 11

luup.variable_watch( 'alphanumerical2numerical', HVACO_SID, nil, HVACO_DEVICE_ID )

function alphanumerical2numerical( lul_device, lul_service, lul_variable, lul_value_old, lul_value_new )

 if ( lul_service == HVACO_SID ) and ( lul_variable == 'ModeStatus' )
  then

   if lul_value_new == 'HeatOn'
    then
     luup.variable_set( MY_NUMERICAL_VARIABLES_SID, tostring( lul_variable ) .. 'Numerical', '1', lul_device )
   elseif lul_value_new == 'Off'
    then
     luup.variable_set( MY_NUMERICAL_VARIABLES_SID, tostring( lul_variable ) .. 'Numerical', '0', lul_device )
   end

  end

end
« Last Edit: November 16, 2011, 03:42:24 pm by Ap15e »

Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #74 on: November 17, 2011, 08:50:00 am »
Oh cr@p - dataMine Startup LUA failed message is up again. It did seem to coincide with my installing another half a dozen appliance switches this evening and then trying to add them in the confiig tab. Will have to go thru the previous messages again to rectify - although did just swap out the L_DataMine1.lua file again to see if it sorted the message out - but no go.