We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 245511 times)

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #90 on: November 26, 2011, 01:29:26 pm »
Quote
/what/datamine???

I see /overlay/www/cmh/plugins/datamine
The /overlay directory is holding the extra plugin files. What you should find in WinSCP is a directory called /dataMine - i.e. from the root directory.  This is the directory that ought to be linked into the USB drive - if this directory doesn't exist, then you're not going to get too far, and this would probably explain at least some of what's going on (such as your "logging 4 channels" going to 0).

Had failed to create that, that explains a lot

Have now created it and can't tell if it is mounted on the USB or it's on Vera.


Quote
How do I tell what is the stick and what is the Vera with WinSCp?
You can't directly see this in WinSCP - Linux makes things nice and seemless...
The command "mount -v" will do what you want (I think). You can run this command by using the WinSCP terminal (under Commmands | Open Terminal menu).

Tried the mount command, don't know if it worked

When I did this it actually showed that my system isn't linking to my USB drive, so I need to look further into this. This means everything is going into the Vera drive, which will ultimately fill up...  I'll take a look at this and try and work out what's happening - it did used to work, so presumably it means the USB isn't getting mounted on startup.

Not just me then :)

I now have it claiming to be logging and staying there when I press save instead of losing it, having somewhere to save it helps!

Not getting graphs yes, there is a yellow box that appears when I go to the graph tab and the channels that I selected on the config tab are there but I can't click anything that makes a graph appear

I can't seem to select logging interval in the config tab either.

The /datamine directory has 8 files in it and a "lost and found" directory which all looks like the other log location was

LuaUPnP.log is there dated a few mins ago

Looks like a file is being created with one of the devices I am logging as a file name, that's a good sign, there are a few pairs of values in it as well.

Still don't seem to be able to draw a graph though

Definate progress though, it was me failing to create the output directory that was causing a lot of the trouble

I still have

dataMine[50]:
Running Lua Startup

top right of the dashboard, something is unhappy

Definately nearer though

Nick


Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #91 on: November 26, 2011, 01:39:19 pm »
Quote
Not just me then :)
No - I've been working through this, and it's not quite as simple as I'd thought to get the custom-user-startup file to run on startup - I'll write that up in a separate post in a minute...

Quote
I can't seem to select logging interval in the config tab either.
You can't select logging interval - dataMine will log data when it changes - not at a specific intrval. So, you won't have any data logged until it has changed - if you're logging a switch status, then flick them on/off a few times, and you should get data logged.

Quote
I still have

dataMine[50]:
Running Lua Startup

top right of the dashboard, something is unhappy
Can you take a look in the log file and send me any entries when dataMine starts - there must be something wrong, and the log is the best (only?) place to work out what.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #92 on: November 26, 2011, 01:49:56 pm »
I've recently noticed that the custom-user-startup wasn't being executed on startup, and my dataMine data was being logged into the local Vera drive. This isn't good since a) it's not especially big, and b) it probably has limited write cycles.

So, after a lot of chasing, I've worked out that 2 more commands are required.
Code: [Select]
chmod +x /etc/init.d/custom-user-startup
/etc/init.d/custom-user-startup enable

The first command (chmod) sets the file permissions to make the file executable. I'm not 100% sure this is required, but I think it is.

The second command makes a symbolic link in the /etc/rc.d directory. This is what actually gets the file to run when Vera starts.

When Vera is running, you can check that the USB stick is mounted and mapped to the /dataMine directory by running the command -:
Code: [Select]
mount -v
On my system, the following is shown -:
Code: [Select]
/dev/scsi/host0/bus0/target0/lun0/part1 on /dataMine type vfat (rw)
One point to note is that if you find you also have this issue, you should copy all the files from your dataMine directory onto your PC before you remap the data to the USB stick. Otherwise, you'll loose access to the files since the USB stick will be mapped over the other files.

(I should add, I'm not a Linux expert so if there's someone out there who knows more about this than me, feel free to comment)

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #93 on: November 26, 2011, 01:53:21 pm »
Looking at Log file


11/26/11 18:10:10.592   luup_log:50: dataMine: Initialising Data Mining System <0x402>
50   11/26/11 18:10:10.726   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 18:10:10.727   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
06   11/26/11 18:10:10.728   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 2 now: 2 #hooks: 0 upnp: 0 v:0x825c78/NONE duplicate:1 <0x402>
06   11/26/11 18:10:10.729   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 2 now: 2 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
01   11/26/11 18:10:10.742   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>


Hopefully that means something to you

It is logging time and data for the items I am logging, just don't seem to have the graphing in there yet

Will look at the data mount stuff later, get it working first.

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #94 on: November 26, 2011, 02:28:24 pm »
So, this has the same error as in your previous log. It's also not logging the extra stuff I added earlier today, which is a "startup complete" message.

The only thing I can think of is that some of your other files are out of date (maybe the I_DataMine1.xml file). Can you make sure the latest files are loaded from the repository. 

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #95 on: November 26, 2011, 02:49:42 pm »
I hadn't updated the file at that stage, remember it was failing to upload via the developers option then.

Screenshot of my version of the graph tab attached

I have just SCPd it and will try again

OOps new message

Startup Lua failed

Graph tab now blank

Config Selections not coloured

Looks like there is a Lua crash report in there, can I send you the whole log file by e-mail, it is 2.5 meg

Looks like changing the Lua  file and the two Flotr files that I have just updated has crashed it somewhere, not sure how to restart it. The log extract below seems to be the crash

Nick (out of his depth)



2011-11-26 19:25:57 - LuaUPnP Terminated with Exit Code: 245



2011-11-26 19:25:57 - LuaUPnP crash

01 2011-11-26 17:53:7 caught signal 11 <0x400>
01 2011-11-26 18:9:55 caught signal 11 <0x400>
01 2011-11-26 18:10:29 caught signal 11 <0x400>
01 2011-11-26 18:12:10 caught signal 11 <0x400>
01 2011-11-26 18:30:40 caught signal 11 <0x400>
01 2011-11-26 18:41:40 caught signal 11 <0x400>
01 2011-11-26 18:43:55 caught signal 11 <0x400>
01 2011-11-26 18:58:24 caught signal 11 <0x400>
01 2011-11-26 19:22:55 caught signal 11 <0x400>
01 2011-11-26 19:25:57 caught signal 11 <0x400>
---------------exited-------------
  PID USER       VSZ STAT COMMAND
    1 root      1444 S    init       
    2 root         0 SW   [keventd]
    3 root         0 SWN  [ksoftirqd_CPU0]
    4 root         0 SW   [kswapd]
    5 root         0 SW   [bdflush]
    6 root         0 SW   [kupdated]
    8 root         0 SW   [mtdblockd]
   95 root         0 SWN  [jffs2_gcd_mtd4]
  119 root      1444 S    init       
  157 root      1692 S    syslogd -C256
  159 root      1436 S    klogd
  172 root       708 S    /sbin/hotplug2 --override --persistent --set-worker /lib/hotplug2/worker_fork.so --set-rules-file /etc/hot
  313 root         0 SW   [khubd]
  523 root         0 SW   [usb-storage-0]
  524 root         0 SW   [scsi_eh_0]
  593 root         0 SW   [kjournald]
  624 root         0 SW   [loop0]
  942 root      1456 S    crond -c /etc/crontabs -l 5
  948 root      1120 S    /usr/s


Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #96 on: November 26, 2011, 03:17:44 pm »
Still tinkering

Changed "L_DataMine1.lua" back to the original and it now gives me

dataMine[50]:
Running Lua Startup

rather than the other error and the devices are back in the yellow box again on the graph tab. Still no graph though

Hope this makes some sense

Log file shows at the time I swapped the file back and pressed refresh

11/26/11 20:01:22.077   luup_log:50: dataMine: Initialising Data Mining System <0x402>
02   11/26/11 20:01:22.080   ZWaveJobHandler::SetDeviceStatus XXX2 0/0/0/0 <0x803>
50   11/26/11 20:01:22.683   luup_log:50: dataMine: Watching: D[33] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 20:01:22.684   luup_log:50: dataMine: Watching: D[42] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/26/11 20:01:22.685   luup_log:50: dataMine: Watching: D[45] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50   11/26/11 20:01:22.686   luup_log:50: dataMine: Watching: D[27] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/26/11 20:01:22.687   luup_log:50: dataMine: Watching: D[25] S[urn:upnp-org:serviceId:SwitchPower1] V[Status] <0x402>
50   11/26/11 20:01:22.688   luup_log:50: dataMine: Watching: D[43] S[urn:micasaverde-com:serviceId:EnergyMetering1] V[Watts] <0x402>
50   11/26/11 20:01:22.689   luup_log:50: dataMine: Watching: D[47] S[urn:micasaverde-com:serviceId:SecuritySensor1] V[Tripped] <0x402>
06   11/26/11 20:01:22.690   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 7 now: 7 #hooks: 0 upnp: 0 v:0x827238/NONE duplicate:1 <0x402>
06   11/26/11 20:01:22.691   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 7 now: 7 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x402>
01   11/26/11 20:01:22.692   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>

Hope this makes sense??

Nick



Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #97 on: November 26, 2011, 05:34:23 pm »
One thing that I note in your picture is that you don't have any of the icons on the graph page - they are just showing up as some sort of default icon (at least that's how it looks). If that's the case, then the other problem may be that the other files (PNG, JS etc) aren't in the correct location /www/cmh/plugins/dataMine. If this wasn't correct, then you won't get a graph.
What browser are you using? I've tried the libraries on IE and Firefox, so there's a (small) possibility that there's an issue here as well.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #98 on: November 26, 2011, 05:49:27 pm »
I've now got a version of dataMine working that will graph "non numerical" data. I don't have a lot of this sort of data, so I'd like a bit of feedback before I release it to the repository. (refer to the files attached to this post).

Below is an example of what it looks like. It's worth noting that dataMine builds up a lookup table between the text and numbers, so it may do some strange stuff if you try and graph data that hasn't fully built up the lookup. At some stage, I'll post an explanation of how to change the default values, but for now I just want to see if it works.

So, if you have this sort of data, and are feeling game, then feel free to give this a go - I welcome any feedback.  One issue I have noticed is that if you try and plot this sort of data on the right hand y axis, it doesn't work. I don't know if this is something I'm doing wrong, or if it's a limitation of the library - I'll work this out over the next few days.

<< There's an error in this version as it doesn't correctly change graph modes. I don't seem to be able to remove the zip file below - I'll load another version in a new post >>
« Last Edit: November 27, 2011, 05:43:37 am by Chris »

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #99 on: November 26, 2011, 06:05:14 pm »
One thing that I note in your picture is that you don't have any of the icons on the graph page - they are just showing up as some sort of default icon (at least that's how it looks). If that's the case, then the other problem may be that the other files (PNG, JS etc) aren't in the correct location /www/cmh/plugins/dataMine. If this wasn't correct, then you won't get a graph.

Spotted one error on my part, the lack of a capital M in the directory name. I have changed it now and still get the same results.

How do I force things to restart, does the refresh symbol on the dashboard work for that???

What browser are you using? I've tried the libraries on IE and Firefox, so there's a (small) possibility that there's an issue here as well.

Using Chrome or Safari, both the same, tried IE6 and the spanner symbol doesn't do anything there, data logging is working nicely now, it's just the graphics missing

More progress, tried Chrome again now the directory name is right and it's giving me the custom symbols, allowing me to select channels, just failing to draw the graph!

Browser comes into it and Safari is bad news, one for the instructions page there, I normally use Chrome anyway, Vera just seemed a little happier with Safari when I was originally setting up and I stuck with it.

I'll try Firefox tomorrow and see if that is any happier drawing graphs, we are very close now, thanks for your patience is talking me through it all

Still got "running Lua Startup" at the top though

Nick




Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #100 on: November 26, 2011, 09:04:36 pm »
I have just uploaded the newer versions of the two files you mentioned, Chris. Have been getting that old "Startup Lua Failed" message again (before I uploaded new files. Interestingly I copied (replaced) the L_DataMine1.lua in etc/cmh-ludl and it shows a date/time stamp of 27/11/11 10.07am when it is 11.56am here - even when I refresh and restart dataMine the file remains unchanged. Also when I checked the log file I cannot find any reference to dataMine at all - strange.

Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)

Wondering if I should just start from scratch again...

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #101 on: November 27, 2011, 03:13:37 am »
Hi Chris

I tried Firefox, still no graphs but it tried to draw them.

Thinking aloud it's logging data in the /dataMine directory and that data is appearing ok, what I don't see are the "downsampled files" that you talk about. The dataMine directory is shown on the screen shot attached.

I'm sure we are on the home straight now, the data is being captured and the graph plugin is showing, it's just the last bit now

Many thanks for the help and all the good work, it looks just what I hoped Vera would do when I bought it!

Nick

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #102 on: November 27, 2011, 05:35:55 am »
@willstar
Quote
Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)
I'm not sure what's going on - I don't think this is an issue with dataMine itself (although I stand ready to be corrected). I think this must be something strange with the new Vera version you're running or something. I've certainly not had any problems loading updated software through the developers tab - I've not tried it directly with WinSCP.

@nickrwym
Quote
what I don't see are the "downsampled files" that you talk about
No - this isn't actually implemented yet. It's in there, but as I haven't worked out a couple of things, it's currently disabled - I've had more requests for other graph types than the historical data, so that's taken my time so far.

In your graph screen, do you see all the buttons with correct icons in them? The image you sent yesterday didn't look correct.
Are you still getting the error about Lua Startup with the same isue logged in the log file? Either your problem is associated with this (but as most of the software seems to be working, I'm not sure), or the other issue is that the flotr.js files aren't being loaded. These files should be in the same directory as the PNG icons - hence my question about the icons.



Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #103 on: November 27, 2011, 05:46:24 am »
I've found minor a bug in the version of the software that graphs text data. It doesn't correctly set the data into this mode - I'd done this manually during testing, and then forgot to add the functionality.

So, attached is an update - sam caveats apply - the software will need to run for a while to learn the different text data values.

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #104 on: November 27, 2011, 11:49:26 am »
@willstar
Quote
Now, having re-checked in etc/cmh-ludl there's no sign of the  L_DataMine1.lua file at all - nor in the overlay directory. Trying to copy it across seems to fail - the file doesn't appear in the directory (using winSCP)
I'm not sure what's going on - I don't think this is an issue with dataMine itself (although I stand ready to be corrected). I think this must be something strange with the new Vera version you're running or something. I've certainly not had any problems loading updated software through the developers tab - I've not tried it directly with WinSCP.


Win SCP copies it across fine, I was just having trouble loading it with the developer tab. It may have been a browser issue but it worked originally so I don't know.

Putting the old version back in made it work better than the new version so that one is in now.


@nickrwym
Quote
what I don't see are the "downsampled files" that you talk about
No - this isn't actually implemented yet. It's in there, but as I haven't worked out a couple of things, it's currently disabled - I've had more requests for other graph types than the historical data, so that's taken my time so far.

That's why I don't see them then!

In your graph screen, do you see all the buttons with correct icons in them? The image you sent yesterday didn't look correct.

Yes, that was the lack of a capital M in the middle of the directory name, my error
 
Are you still getting the error about Lua Startup with the same isue logged in the log file? Either your problem is associated with this (but as most of the software seems to be working, I'm not sure), or the other issue is that the flotr.js files aren't being loaded. These files should be in the same directory as the PNG icons - hence my question about the icons.

Yes, icons are there, it tried to draw a graph, allows you to select the data and then when you click the icon on the right it goes to the graph screen and I see a circling "wait" indicator, when that goes there is a lank screen, it's the same in Firefox as Chrome.

The data files are there for each of the items that I have selected to log, both in the pull down list and in the /dataMine directory as  a set of pairs of values, once the graphing is sorted I will be able to look at the data. Looking at it I am not convinced some are updating properly, the energy monitor is though.

Hopefully I am nearly there, still got the "Running Lua Startup" message though

Still not getting graphing but do have all the data selection and logging working.

Nick