We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287698 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #105 on: November 27, 2011, 01:12:07 pm »
Quote
Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen
To me, this sounds like the JS files aren't being loaded. The fact that you get the "wait" indicator, indicates to me that the UI is requesting the data from Vera, and the data is probably being returned, but not graphed.

Do you have FireBug installed? This is a FireFox addon that allows you to see the data flying around on the network between the browser and the Vera. With this, you'd be able to see if Vera/dataMine was returning graph data or not - we'd be able to narrow it down a bit more.

Quote
still got the "Running Lua Startup" message though
I'm really not sure what that's about. I'll take a look at the code and try and add some more debug/log data into the software.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #106 on: November 27, 2011, 02:25:42 pm »
Nick,
Please load this version of L_DataMine1.lua. It adds some more logging into the startup. The problem I've got is that from the current logging, it looks like everything except the last log message is happening. The last command before this log is to register a handler that handles the registering of variables (i.e. when you click on a variable and it goes green).

I also need to get more log data. I need all the dataMine log entries. You can do this by typing the following command into the terminal window in WinSCP
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
So, load this version, restart Lua, then try to graph something, run the above command and send me the data...

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #107 on: November 27, 2011, 02:35:17 pm »
Nick,
Please load this version of L_DataMine1.lua. It adds some more logging into the startup. The problem I've got is that from the current logging, it looks like everything except the last log message is happening. The last command before this log is to register a handler that handles the registering of variables (i.e. when you click on a variable and it goes green).

I also need to get more log data. I need all the dataMine log entries. You can do this by typing the following command into the terminal window in WinSCP
Code: [Select]
grep dataMine /tmp/log/cmh/LuaUPnP.log
So, load this version, restart Lua, then try to graph something, run the above command and send me the data...

Thanks for your persistence with this

I uploaded the new file using Winscp, pressed the refresh on the dashboard and still get the starting message

Tried to draw a few graphs and pressed a few of the options on the graph page and grepped the log as requested, log follows


/dataMine$ grep dataMine /tmp/log/cmh/LuaUPnP.log
50   11/27/11 17:57:59.596   luup_log:50: dataMine: ID: 2 Start: 1322330279 Stop: 1322416679 Sample: 123 <0x8d808>
09   11/27/11 19:06:29.416   JobHandler_LuaUPnP::Run device 50 dataMine room 2 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x7e3828 upnp: 0 <0x400>
50   11/27/11 19:06:31.187   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/27/11 19:06:31.918   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:06:31.919   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:06:31.920   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:06:31.921   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:06:31.922   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:06:31.923   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:06:31.924   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
50   11/27/11 19:06:31.926   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[KWH] <0x402>
09   11/27/11 19:29:02.121   JobHandler_LuaUPnP::Run device 50 dataMine room 2 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x7e3618 upnp: 0 <0x400>
50   11/27/11 19:29:03.587   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/27/11 19:29:04.198   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:29:04.200   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:29:04.201   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/27/11 19:29:04.202   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:29:04.204   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/27/11 19:29:04.205   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/27/11 19:29:04.206   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
50   11/27/11 19:29:04.207   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[KWH] <0x402>
50   11/27/11 19:29:04.210   luup_log:50: dataMine: Registering handlers 1 <0x402>
50   11/27/11 19:29:04.211   luup_log:50: dataMine: Registering handlers 2 <0x402>
50   11/27/11 19:29:04.211   luup_log:50: dataMine: Registering handlers 3 <0x402>
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
50   11/27/11 19:29:41.187   luup_log:50: dataMine: ID: 3 Start: 1322335781 Stop: 1322422181 Sample: 123 <0x400d>
50   11/27/11 19:29:57.863   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422197 Sample: 84 <0x3c10>
50   11/27/11 19:29:59.738   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422199 Sample: 84 <0x3c10>
50   11/27/11 19:30:01.338   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422201 Sample: 84 <0x400d>
50   11/27/11 19:30:02.467   luup_log:50: dataMine: ID: 3 Start: 1322363013 Stop: 1322422202 Sample: 84 <0x400d>
50   11/27/11 19:30:03.658   luup_log:50: dataMine: ID: 3 Start: 1322335803 Stop: 1322422203 Sample: 123 <0x3c10>
50   11/27/11 19:30:04.722   luup_log:50: dataMine: ID: 3 Start: 1322335804 Stop: 1322422204 Sample: 123 <0x400d>
50   11/27/11 19:30:05.528   luup_log:50: dataMine: ID: 3 Start: 1322335805 Stop: 1322422205 Sample: 123 <0x400d>
50   11/27/11 19:30:06.313   luup_log:50: dataMine: ID: 3 Start: 1322335806 Stop: 1322422206 Sample: 123 <0x400d>
50   11/27/11 19:30:12.016   luup_log:50: dataMine: ID: 3 Start: 1322335812 Stop: 1322422212 Sample: 123 <0x3c10>
50   11/27/11 19:30:12.022   luup_log:50: dataMine: ID: 4 Start: 1322363013 Stop: 1322422212 Sample: 84 <0x3c10>

Nick

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #108 on: November 27, 2011, 02:40:39 pm »
Quote
Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen
To me, this sounds like the JS files aren't being loaded. The fact that you get the "wait" indicator, indicates to me that the UI is requesting the data from Vera, and the data is probably being returned, but not graphed.

Could be

Do you have FireBug installed? This is a FireFox addon that allows you to see the data flying around on the network between the browser and the Vera. With this, you'd be able to see if Vera/dataMine was returning graph data or not - we'd be able to narrow it down a bit more.

No, I've got wireshark on one of the machines though so I can packet sniff if that helps?

Quote
still got the "Running Lua Startup" message though
I'm really not sure what that's about. I'll take a look at the code and try and add some more debug/log data into the software.

Your help is much appreciated

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #109 on: November 27, 2011, 03:18:07 pm »
Quote
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
Ok - it gets stranger....
The first line shows that the startup is completing correctly, so I don't know why the "Startup error" message is still there.
The second line shows that the Lua file is receiving requests for graph data - and presumably it's sending it back.

I also added something to display an error if the graph library isn't there, and as that's not displayed, I can rule that out.

Maybe you can use Wireshark to confirm that Vera is returning graph data - it's all text, so it'll be easy to find if you're familiar with Wireshark filters. Maybe there's something funny in the response (he says clutching at straws).

Sorry - I'm not being especially helpful  :(

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #110 on: November 27, 2011, 04:22:59 pm »
Quote
50   11/27/11 19:29:04.212   luup_log:50: dataMine: Startup complete <0x402>
50   11/27/11 19:29:35.179   luup_log:50: dataMine: ID: 3 Start: 1322335775 Stop: 1322422175 Sample: 123 <0x400d>
Ok - it gets stranger....
The first line shows that the startup is completing correctly, so I don't know why the "Startup error" message is still there.

The exact message is

dataMine[50]:
Running Lua Startup

It's informative rather than an error??


The second line shows that the Lua file is receiving requests for graph data - and presumably it's sending it back.

Sounds good

I also added something to display an error if the graph library isn't there, and as that's not displayed, I can rule that out.

Sounds good too

Maybe you can use Wireshark to confirm that Vera is returning graph data - it's all text, so it'll be easy to find if you're familiar with Wireshark filters. Maybe there's something funny in the response (he says clutching at straws).

Not very familiar, used it once or twice but not a regular tool

I've packet sniffed a "graph creation attempt" and can see it ask for the data

243   13.608733   192.168.1.104   192.168.1.107   HTTP   639   GET /port_3480/data_request?id=lr_dmData&channel1=2&channel2=5&channel3=0&channel4=0&start=0&stop=0&format=json HTTP/1.1

It then has lots of data flowing down from 107 which is Vera, it seems to fall into groups of 3 with one of these first

244   13.610461   192.168.1.107   192.168.1.104   TCP   60   http > 57781 [ACK] Seq=1 Ack=586 Win=6435 Len=0

followed by 2 of these which I assume are more data associated with the first packet that wouldn't fit into single packets?

300   16.610676   192.168.1.107   192.168.1.104   TCP   1514   [TCP segment of a reassembled PDU]

As I don't know what it looks like when it's working properly this isn't showing me much apart from lots of data flowing when asking for a graph!

Will the firefox extension help more??

Nick



Sorry - I'm not being especially helpful  :(

You are being very helpful by persevering

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #111 on: November 27, 2011, 05:28:52 pm »
Quote
dataMine[50]:
Running Lua Startup

It's informative rather than an error??
Ah - yes, that's normal - I assume this just flashes up briefly and then goes away...

Quote
Will the firefox extension help more??
Yes, FireBug is very good at displaying this sort of this - it'll display it nicely as text and marries up the request and response so you don't have to work out the "mess" that Wireshark shows.

I still think that there's an issue with the graph library not being loaded - everything else seems to be working. The Wireshark message you showed with the "channel" data is the request from the UI to Vera, and obviously Vera is returning something and I can't see why it would be incorrect. The only other suggestion I can make is to make sure the L_DataMine1.js file is up to date.

Again, FireBug will be able to confirm any issues in the UI quite quickly as it will show any javascrip errors (such as a function not being loaded).

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #112 on: November 27, 2011, 06:22:54 pm »
Quote
dataMine[50]:
Running Lua Startup

It's informative rather than an error??
Ah - yes, that's normal - I assume this just flashes up briefly and then goes away...

It stays there permanently

Quote
Will the firefox extension help more??
Yes, FireBug is very good at displaying this sort of this - it'll display it nicely as text and marries up the request and response so you don't have to work out the "mess" that Wireshark shows.

Downloaded and installed, what settings do I need to capture what we need to see??

I still think that there's an issue with the graph library not being loaded - everything else seems to be working. The Wireshark message you showed with the "channel" data is the request from the UI to Vera, and obviously Vera is returning something and I can't see why it would be incorrect. The only other suggestion I can make is to make sure the L_DataMine1.js file is up to date.

Again, FireBug will be able to confirm any issues in the UI quite quickly as it will show any javascrip errors (such as a function not being loaded).

Sounds good, give me a clue as to what parameters we want to look at and I'll try a capture

Just uploaded J_Datamine1.js, fixed the upload via developers fault which occurred when I made the change suggested in reply 67 from Florin, reverted back to the original file and can upload again

I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???

Just reuploaded all the "developer tab" files and it made no difference, it tries to draw a graph but there is nothing there, just blank white once the timer goes.

So near but so far :-(

Nick

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #113 on: November 27, 2011, 06:35:32 pm »
Quote
Downloaded and installed, what settings do I need to capture what we need to see??
There's a "Firebug" button down in the bottom right corner of Firefox - press this.

To view all the traffic between the UI and Vera, click on the "Console" tab. Open up data mine, select a channel to graph, and look in the console window for something that looks like this
Code: [Select]
GET http://192.168.2.232/port_3480/data_request?id=lr_dmData&channel1=1&channel2=0&channel3=0&channel4=0&start=0&stop=0&format=json
I want to see the response which will be displayed under this line if you click on it. It would be best to graph something that doesn't have too much data!

The other thing that you might see in the bottom right corner (in red) is an error message. If there are any script errors, then Firebug may report "1 Error". If you click on this, I think it will tell you the line number etc.

Let me know what you find with this and we'll see where we go next.

Quote
I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???
Yes - sorry, that was a typo.


Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #114 on: November 28, 2011, 07:57:15 am »

Hi, I'm a bit confused as to why the temperature data int the graph isn't being sub-sampled. Is it something to do with the interval?

Thanks, Alex

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #115 on: November 28, 2011, 08:14:23 am »
Quote
I'm a bit confused as to why the temperature data int the graph isn't being sub-sampled.
What makes you think it's being subsampled? DataMine can subsample data, but I don't think that's happening here. The only way to tell is to look at the raw TXT files in the dataMine directory - the first parameter is in seconds, so if there's a big jump between subsequent lines, then dataMine is graphing what is recorded.

What sort of sensor is this coming from? I am using a couple of the ST814 (I think that's the model) Temp/Humidity sensors, and this is about what I see from them. They only send out updates occasionally, and for humidity, only when there is a 5% change (I think it's 1 deg for temperature).

I am starting to use the One Wire sensors from EDS as they aren't bound by various limitations in Vera (e.g. the artificial 1 degree resolution that Vera imposes) and the plots are much nicer.

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #116 on: November 28, 2011, 08:17:24 am »
What sort of sensor is this coming from?

Rather simplistically I was expecting curves :) Yes it's a thermostat which is sending temperature sensor readings intermittently with a granularity of 1°C.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #117 on: November 28, 2011, 09:28:10 am »
Quote
Rather simplistically I was expecting curves
Ah - ok, that's something different. This is unfortunately the problem with Vera only providing sensor data at low resolution. Currently, dataMine assumes that the data stays constant between changes - this is effectively what Vera will show in the UI. It was necessary to do this in order to properly display status values (e.g. light switch on or off), but it's not (necessarily) correct for sensors.

One of the next things on my "todo" list is to change this. I want to add extra configuration parameters in the config window to allow things like this to be set on a per channel basis. It still won't be curves (at least not initially), but it won't look like the Manhattan skyline.

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #118 on: November 28, 2011, 09:36:42 am »
Quote
Rather simplistically I was expecting curves
It still won't be curves (at least not initially), but it won't look like the Manhattan skyline.

Great stuff :)

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #119 on: November 28, 2011, 10:21:31 am »
Quote
Downloaded and installed, what settings do I need to capture what we need to see??
There's a "Firebug" button down in the bottom right corner of Firefox - press this.

To view all the traffic between the UI and Vera, click on the "Console" tab. Open up data mine, select a channel to graph, and look in the console window for something that looks like this
Code: [Select]
GET http://192.168.2.232/port_3480/data_request?id=lr_dmData&channel1=1&channel2=0&channel3=0&channel4=0&start=0&stop=0&format=json
Lots of traffic, various errors which I don't think are DataMine, and very little traffic when asking for a graph


I want to see the response which will be displayed under this line if you click on it. It would be best to graph something that doesn't have too much data!

More to follow later when I have more time

The other thing that you might see in the bottom right corner (in red) is an error message. If there are any script errors, then Firebug may report "1 Error". If you click on this, I think it will tell you the line number etc.

No error there

Let me know what you find with this and we'll see where we go next.

Quote
I am assuming that when you said this " L_DataMine1.js file is up to date." it is a typo and that is J_ not L_ as I don't have an L version???
Yes - sorry, that was a typo.

Good, means I am starting to get to know what is there!

I had a major cockup last night, I replaced all the MIOS developer files with the latest ones, went to bed and checked it in the morning.

The /DataMine directory was empty, whatever I had done had broken it completely and it had deleted all the history that had been building up.

I went back and replaced these files with an older version and got it working again, data is logging now.

Question, is there a copy of a set of the Flotr files that definately work in a place I can get to them, I took them from the link that was on the original install hence they were not from your collection.

Can you put a copy in the depositry so I can get known good ones??

Somewhere among the 6 files that go into the plugins directory there is a faulty file in one of the various incarnations of the program, that is what broke things last night for me. Luckily I had all the various options saved and was able to rebuild.

My gut feel is t replace the Flotr files with known good ones from an installation that is drawing graphs OK



Nick