We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287704 times)

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #270 on: June 09, 2012, 01:58:47 am »
Update...

After many clicks, some of the items are responding and highlighting, then providing a drop-down... but only the two I show in my screenshot... all others I tried do nothing.

Also...

When I open the graph web page I get this error...
"Error downloading startup data (dataMine channel configuration)."


Offline bfromdruten

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #271 on: June 09, 2012, 02:58:30 am »
Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

I just emptied the dataMine folder, the plugin will give 0 channels (data & recording), but the configuration tab still shows colours a certain variables. Don't know how to change that.

Using internet explorer 9

Thanks

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #272 on: June 09, 2012, 04:33:15 am »
What size USB stick do you recommend?

In reality, it's not a lot of data - with about 6 months data logged, with 38 channels, I've got about 20MB of data.

I bought a really small USB stick (size wize) that only poked out the back of the Vera by a few mm. I think it was 4GB or 8GB.  Looking at my log files, if you're logging light switches, then weekly file size is minimal (<1k), and if you're logging analogue data that changes regularly (I have some humidity sensors that log every 0.1% change!) then I have files that range between 40 kb and 80 kb per week.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #273 on: June 09, 2012, 04:41:28 am »
Under the Config, all the items are showing properly and they open when clicked... but how do you 'select' which ones you want to use? I don't see a checkbox, or highlight, etc?
I see you've progressed from this, but I'll answer this anyway... This is the same as UI4 - you click on the devices which open the services, click again to get the variables. If you click the variable name, then the variable should highlight in green (this seems to take about a second on my Vera2, so don't click too quickly). If you click again, it will highlight in yellow (multiple clicks will toggle between green and yellow).

Green means the variable is being logged. Yellow means it's been logged in the past. I don't completely remove a variable that you no longer want to log just in case you want to keep the data history.

I see in your attachment that the variables are highlighted in yellow - this means they aren't actually being logged.

In the dataMine device in the UI, you should have two numbers being displayed - total channels, and channels being logged. Total channels is the total number highlighted in yellow or green, and channels being logged are the ones in green.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #274 on: June 09, 2012, 04:49:53 am »
When I open the graph web page I get this error...
"Error downloading startup data (dataMine channel configuration)."

When the UI starts, it downloads a bunch of data from the Vera - overall Vera configuration, and dataMine configuration (which is the one that is failing). It might be that the plugin isn't running when you start the UI (eg if Lua is restarting, then the UI won't be able to download the data).

If the plugin isn't running for some reason, that may also explain why the config system isn't working properly, since within UI5, the config box communicates with the plugin in a similar way to the dataMine GUI does.

Things I'd look at for starters -:
  • Make sure Lua is running (the "unit busy" statement isn't up in the notification area of UI5)
  • Check the log files to see if there's anything in there relating to dataMine.
  • Have a look in the /dataMine directory (on the USB stick?) to see if the dataMineConfig.json file is being saved.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #275 on: June 09, 2012, 04:56:08 am »
Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

The changing of the background colour may take a second or so since it needs to communicate back to the dataMine plugin, save the data, and send an "Ok" to the UI before the colour changes (maybe this is a bad way to do it and I should have instant feedback, but that's how it currently works). This may (or not!) explain what you're seeing?

One of the next things on my ToDo list is to move the config system into the dataMine UI as I ultimately want to add extra configuration for charting purposes. When I move it over, it will be done in a different way so bear with me for now (so long as you can actually get it to work of course).

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #276 on: June 09, 2012, 11:42:45 am »
Quote from: Chris
    Make sure Lua is running (the "unit busy" statement isn't up in the notification area of UI5)
    Check the log files to see if there's anything in there relating to dataMine.
    Have a look in the /dataMine directory (on the USB stick?) to see if the dataMineConfig.json file is being saved.

It must be running... the /dataMine folder has tons of .backup files and the .json file in it.

Plugin looks great, but I can confirm Aaron his findings. There are problems with configuration list and colours. With an empty list (and empty dataMine folder), clicking doesn't always change colour. If you would get a variable to change it's background colour, it isn't related with number of recorded channels. I did manage to record a variable and show it in the chart, but don't know how many times I had to click.

The changing of the background colour may take a second or so since it needs to communicate back to the dataMine plugin, save the data, and send an "Ok" to the UI before the colour changes (maybe this is a bad way to do it and I should have instant feedback, but that's how it currently works). This may (or not!) explain what you're seeing?

This is definitely not working properly. I tried for 10 minutes to choose data point items... and only those 2 shown in my screenshot every would highlight/color.  The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.


I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #277 on: June 09, 2012, 12:04:23 pm »
It must be running... the /dataMine folder has tons of .backup files and the .json file in it.
Ok - every time you select a variable, dataMine saves a backup, so if you've done a lot of clicking, there will be a ton of files  :). Is there anything else in that directory?

The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.
Strange. Is there anything in the log file from dataMine? Until this error goes away, the graph won't work as the error means it can't get the configuration.

I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.
Don't worry - it will be very different. I'm planning on a properties box type system so that I can have the ability to edit a number of different properties easily. I started doing this with a grid, but that was a bit messy and not scalable, so I'm changing this at the moment.

Anyway, back to your problem. Please take a look at the /tmp/log/cmh/LuaUPnP.log file and do a search for dataMine to see if there's anything there. You can use the command "grep dataMine /tmp/log/cmh/LuaUPnP.log" to get the lines with dataMine.
The other question is are you running Firefox, and if so, do you have Firebug? I'd be interested to see what transactions show up in the console with dataMine starts.

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #278 on: June 09, 2012, 12:26:10 pm »
It must be running... the /dataMine folder has tons of .backup files and the .json file in it.
Ok - every time you select a variable, dataMine saves a backup, so if you've done a lot of clicking, there will be a ton of files  :). Is there anything else in that directory?

The dataMine panel shows 20 total channels with 11 being logged.... but I still get the error on the graph page and nothing shows other than the page template itself. I changed the two items that are highlighting, to green. I'll let you know if the graph starts to work.
Strange. Is there anything in the log file from dataMine? Until this error goes away, the graph won't work as the error means it can't get the configuration.

I definitely suggest moving the Config into the dataMine web page... and please make it more intuitive than color highlights. standard checkboxes work easily and there is no confusion to if it is on/off.
Don't worry - it will be very different. I'm planning on a properties box type system so that I can have the ability to edit a number of different properties easily. I started doing this with a grid, but that was a bit messy and not scalable, so I'm changing this at the moment.

Anyway, back to your problem. Please take a look at the /tmp/log/cmh/LuaUPnP.log file and do a search for dataMine to see if there's anything there. You can use the command "grep dataMine /tmp/log/cmh/LuaUPnP.log" to get the lines with dataMine.
The other question is are you running Firefox, and if so, do you have Firebug? I'd be interested to see what transactions show up in the console with dataMine starts.

Chris

Yeah, tons of backup files... which I manually deleted but that makes a good case for having a cleanup function inside your code ;)

I use FF 14 beta but no Firebug installed... I can if we need to but lets see what you find in the log first.

I have attached the log, it is very big so I removed all lines not pertaining to dataMine...

Quote
/dataMine$ grep dataMine /tmp/log/cmh/LuaUPnP.log
01   06/09/12 8:13:23.156   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:36:48.442   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:36:54.868   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:38:35.238   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:09.268   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:40:10.098   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:40:12.038   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:13.618   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:47.441   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:57.158   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:40:58.508   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
50   06/09/12 8:41:21.260   luup_log:50: dataMine: Watching: D[3] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Cool] V[CurrentSetpoint] <0x2e839680>
01   06/09/12 8:41:21.298   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:26.568   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:27.255   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:27.843   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
50   06/09/12 8:41:30.509   luup_log:50: dataMine: Watching: D[3] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Cool] V[SetpointTarget] <0x2ee39680>
01   06/09/12 8:41:30.548   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:41:32.781   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 8:41:33.807   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>
01   06/09/12 8:41:45.428   LuaInterface::CallFunction_Request function incomingCtrl name dmCtrl failed [string "-- dataMine Plugin for Vera..."]:325: attempt to concatenate field 'LastRec' (a nil value) <0x2e839680>
01   06/09/12 9:03:23.040   LuaInterface::CallFunction_Request function incomingList name dmList failed [string "-- dataMine Plugin for Vera..."]:209: attempt to concatenate field 'LastRec' (a nil value) <0x2ee39680>




Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #279 on: June 09, 2012, 01:05:45 pm »
Yeah, tons of backup files... which I manually deleted but that makes a good case for having a cleanup function inside your code ;)
Yep - I agree... Another one to add to the list  ;)

The log is useful - I think I know what the problem is, although it is something that I've got code in to fix, so it raises more questions. I've added a field to an internal config structure, and I do have code in to initialise it when dataMine starts, so I'm not sure why this isn't running. Have you used the UI4 version previously?

Can you attach your dataMineConfig.json file. The other thing that might be useful is the part of the log where dataMine starts up. It logs some stuff about starting and finishing config, and it would be interesting to see if there are any errors in there when it starts.

Actually, have you restarted dataMine (ie restarted Lua)? This might explain it - I initialise this variable when dataMine is started, but not when logging... Hmmm - let me have a look at that as it might be the issue...

Thanks
Chris

Offline jojoflyhigh

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +0/-1
Re: New dataMine graphing and logging plugin
« Reply #280 on: June 10, 2012, 04:10:41 pm »
Hi Chris,
It seems that you need json.lua file in usr/lib/lua directory like for Google Calendar Switch application.
But your file and GCal file are not using the same file even if it's the same name.
Can you do something for that ?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #281 on: June 10, 2012, 04:45:21 pm »
Hi Chris,
It seems that you need json.lua file in usr/lib/lua directory like for Google Calendar Switch application.
But your file and GCal file are not using the same file even if it's the same name.
Can you do something for that ?

Hmmm - I didn't know there was more than 1 json.lua file.  I'll take a look at what the calendar app uses and see if I can harmonise them. I know MCV are looking at incorporating a C JSON library, but it won't be in the next release (unfortunately).

Anyway, I'll take a look and see if I can find a solution. One option is to copy the lib into the dataMine Lua file - not very attractive, but it would make life easier for everyone!

I'm planning an update (probably tomorrow) to fix the problem Aaron found...

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #282 on: June 10, 2012, 06:28:16 pm »
It seems the GCal plugin is using a newer (but maybe not better?) version of the JSON library. Unfortunately, it's not compatible with decoding the files from dataMine which came from the previous version of the JSON library. I have run the dataMine json files through jsonlint (an online JSON format checker), and it shows they are ok, as are the ones produced by the new library.

I've worked out a work-around that will make it compatible with either format - the old version seems more "compatible" in that it can handle both formats, where the new version wants something done in a specific way. I'll try and put in a fix to convert the dataMine file to the format expected by the new library, and it shouldn't then matter which one you have installed.

It will probably be tomorrow before I get this done though.

Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #283 on: June 11, 2012, 12:38:39 pm »
An updated version is now in the marketplace (V0.92). This updates the Lua plugin to fix two issues that have been found -:
  • A variable was not initialised. This caused problems when selecting channels to log.
  • A fix has been added to provide compatibility with 0.9.20 or 0.9.50 of json.lua (0.9.50 is used by the GCal plugin).

Chris

Offline Aaron

  • Hero Member
  • *****
  • Posts: 1985
  • Karma: +61/-193
Re: New dataMine graphing and logging plugin
« Reply #284 on: June 11, 2012, 02:46:31 pm »
An updated version is now in the marketplace (V0.92). This updates the Lua plugin to fix two issues that have been found -:
  • A variable was not initialised. This caused problems when selecting channels to log.
  • A fix has been added to provide compatibility with 0.9.20 or 0.9.50 of json.lua (0.9.50 is used by the GCal plugin).

Chris

How do we update?  I don't see a way, or notification, in the VERA UI5 to do it?