We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287574 times)

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #75 on: November 17, 2011, 08:55:08 am »

Something I've noticed... every time I go to the the dataMine graphing tab the first time it fails to load. When I go to another tab and back to the graphing tab it's fine. Any ideas?

Offline utz

  • Sr. Member
  • ****
  • Posts: 274
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #76 on: November 17, 2011, 09:09:23 am »

Something I've noticed... every time I go to the the dataMine graphing tab the first time it fails to load. When I go to another tab and back to the graphing tab it's fine. Any ideas?

yes, a known bug in UI4. Happens with all js tabs.

Offline ajlennon

  • Full Member
  • ***
  • Posts: 154
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #77 on: November 17, 2011, 09:12:58 am »

Ah thanks. I've seen some comments about UI5 in passing. Is that still in beta?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #78 on: November 19, 2011, 09:07:37 am »
Appologies for not being too active here over the past week - I'm travelling a lot with work at the moment.

I've uploaded a new version of the Flotr libraries into the repository (http://code.mios.com/trac/mios_datamine). This fixes an anoying bug with the mouse tracking which meant the wrong point was listed if the background screen was scrolled down from the top.

I'm currently looking at a better way to graph binary status varibles by putting all binaries onto a single axis and vertically aligning them. This seems to be the most requested issue at the moment.

The other issue I want to look at is graphing of non-numeric data. This is more difficult since the Flotr package doesn't support non-numerics. This means somewhere converting the data into a numerical value that the user can understand... I'm open to suggestions on how to do this - conceptually, it's not difficult to code something, but anything that's done at graph time is going to slow down the response which will make the system less usable...

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #79 on: November 19, 2011, 11:48:25 am »
What about a simple mapping file?

<service id>,<variable name>,<value>,<new numerical value>

Display (the list of) <value> - <new numerical value> in the legend.
Log  <value> and <new numerical value>.
If a user doesn't like your default mapping, the user could change the mapping in the configuration file.

... and provide a method for graphing alphanumerical data that cannot be mapped to a discrete number of numerical values (e.g., earthquake region, weather condition, ...).  :)
« Last Edit: November 19, 2011, 11:58:13 am by Ap15e »

Offline chixxi

  • Hero Member
  • *****
  • Posts: 1036
  • Karma: +37/-14
Re: New dataMine graphing and logging plugin
« Reply #80 on: November 22, 2011, 04:14:01 am »
@Chris: Everything working fine here on Vera2 Firmware 1.1.1338. Thank you for this absolutely great plugin!
Developer of Plugins: Virtual Switch, Variable Container, Popcorn Hour Remote, Vacation Ghost. => PLUGINS HAVE BEEN UNPUBLISHED BY ME.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #81 on: November 25, 2011, 02:51:05 pm »
I've uploaded a new version to the repository. The L_DataMine.LUA and J_DataMine.JS files have been updated to add an offset when graphing multiple binary variables on a single channel.  I haven't completely tested this so any comments welcome.

I've also put an offset into the vertical axis to move traces away from the top/bottom of the chart as this often ended up with parts of the plot missing where it ran across the top of the plot. Again, any comments welcome about this.

Offline willstar

  • Sr. Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #82 on: November 26, 2011, 05:48:50 am »
Chris, repository as in here?  http://code.mios.com/trac/mios_datamine  I just couldn't get/see the updated files if this is the location.
Cheers,
David

I've uploaded a new version to the repository.

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #83 on: November 26, 2011, 06:05:21 am »
Help Please :-)

I am setting up Datamine and it's a big learning curve for me

As far as I can tell all the files are in the right place

DataMine seems to start OK

I can select the channels I want to log and they turn green

It says "recording 4 channels"

Come back to it later and it's now saying


Total Channels 0   
Channels Recording 0   


Go back into setup and the config tab and they are blank again, the channels are there but the green has gone

Top of the dashboard page is

dataMine[50]:
Running Lua Startup

which does not go away

The /etc/cmh-ludl/ directory shows all the files there with Izo on the end apart from the L_datamine1.lua file

I saw the bits on Izo earlier in the thread but am not sure what I am meant to do about it!

Any help appreciated

Thanks

Nick


Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #84 on: November 26, 2011, 08:52:00 am »
Following up on myself I've copied the log lines relating to DaraMine startup in case they give you some clues what is happening. Something is unhappy. Lines pasted at end of this message.

From the instructions I had trouble in this area

Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat.

mkdir /dataMine
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine
Modify the /etc/init.d.custom-user-startup file to add the line -:

mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine &

That init.d.custom file didn't seem to be there

Any help very welcome

thanks

Nick




11/26/11 10:46:27.139   luup_log:50: dataMine: Initialising Data Mining System <0x402>
06   11/26/11 10:46:27.141   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: DataDirectory was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402>
50   11/26/11 10:46:27.142   luup_log:50: dataMine: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x402>
06   11/26/11 10:46:27.143   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0x825aa0/NONE duplicate:0 <0x402>
06   11/26/11 10:46:27.145   Device_Variable::m_szValue_set device: 50 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x402>
01   11/26/11 10:46:27.147   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #85 on: November 26, 2011, 09:39:55 am »
Are the log entries you've pasted from the first time you ran dataMine, or does this same thing get logged each time? It looks like it's from the first time it was run as it's setting up some variables that should only get set the first time (ie. the data directory).

The fact that you got the "logging 4 variables" status shows that it was possible for it to write the config file. However, if the config file was located somewhere that didn't get remapped if Vera restarted, then this may explain why the status went back to 0 logged varibles.

Quote
01   11/26/11 10:46:27.147   LuImplementation::StartLua skipping device 50 implementation valid 1 <0x402>
This message is interesting - I don't know what it means as it's coming from the lua engine, but it doesn't look good!

Quote
That init.d.custom file didn't seem to be there
That's actually quite normal - you should just be able to create the file and put the appropriate line into it.

Quote
I saw the bits on Izo earlier in the thread but am not sure what I am meant to do about it!
Nothing - it looks like you've got the correct filenames otherwise you'd have a different problem.

If you look in the /dataMine directory, is there anything in there? Does it map to the USB stick? I think you should be able to remove the USB stick and put it into your PC to see the files - I've not tried this myself though!

It seems that all the main files are in the right place or you would get as far as you are, so I'd suggest looking at the USB stick and the mapping of the directory.

I've added a couple of new messages into the startup function. I'm not sure they will help, but they might. Can you grab the latest L_DataMine1.lua file and load this to your Vera and take a look at the logs.

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #86 on: November 26, 2011, 10:00:07 am »
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September

Am I missing something about getting the new file?

Struggling here, out of my depth rather with Linux.

THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening

How can I tell if it's writing to that rather than Vera's flash? I don't know enough about what I am looking for in Win Scp

Those log file extracts were a while back, I'll try and get some more

It seems that it thinks it's recording data until I press the refresh button on the dashboard display and then it goes away and shows zero again

Something is happening to show the items in the config file, for them to change colour, and for the same ones to show in the graph tab even though I can't seem to get graphs out of it

Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??

Nearly there I am sure but it's finding that last bit

Thanks for your help, much appreciated

Nick




Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #87 on: November 26, 2011, 10:19:42 am »
Quote
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September
Sorry - I didn't update the ZIP - just the code repository. Take a look here -: http://code.mios.com/trac/mios_datamine/browser

Quote
THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening
Ok - that's probably a good indicator that it's going to the USB stick...

Quote
Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??
Yes - the config file is the dataMineConfig.json file, and it should be in the same directory as the data is stored (ie. /dataMine). Is this file in the directory?

More log file dumps would be good...

Offline nickrwym

  • Full Member
  • ***
  • Posts: 139
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #88 on: November 26, 2011, 11:18:36 am »
Quote
I downloaded the zip again and the  L_DataMine1.lua file date is the same as in the original zip, 30th September
Sorry - I didn't update the ZIP - just the code repository. Take a look here -: http://code.mios.com/trac/mios_datamine/browser


Missing knowledge my end, have downloaded the new files from there but having trouble putting them into Vera with the developers route. Tried it in Safari and Chrome and it's not playing. Put them there in Safari in the first place.

Quote
THere is a USB stick in the Vera and the data light flashes from time to time so it looks like something is happening

Ok - that's probably a good indicator that it's going to the USB stick...



Something is, might be Vera logs rather than Datamine ones though.

How do I tell what is the stick and what is the Vera with WinSCp?


Quote
Does it write it's config and log output files in the same place, could it be that it's not finding the USB stick and hence not finding that place??
Yes - the config file is the dataMineConfig.json file, and it should be in the same directory as the data is stored (ie. /dataMine). Is this file in the directory?

/what/datamine???

I see /overlay/www/cmh/plugins/datamine

that's got all the PNG files etc in it but not the dataMineConfig.json file

the LuaUPnP file time is the same as the one I sent the extract from before

I'm struggling here

Nick

Done a file search on Vera with Win Scp and not found it anywhere, seems to suggest it can't find where to write things



More log file dumps would be good...
[/quote]

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #89 on: November 26, 2011, 11:53:39 am »
Quote
/what/datamine???

I see /overlay/www/cmh/plugins/datamine
The /overlay directory is holding the extra plugin files. What you should find in WinSCP is a directory called /dataMine - i.e. from the root directory.  This is the directory that ought to be linked into the USB drive - if this directory doesn't exist, then you're not going to get too far, and this would probably explain at least some of what's going on (such as your "logging 4 channels" going to 0).


Quote
How do I tell what is the stick and what is the Vera with WinSCp?
You can't directly see this in WinSCP - Linux makes things nice and seemless...
The command "mount -v" will do what you want (I think). You can run this command by using the WinSCP terminal (under Commmands | Open Terminal menu).

When I did this it actually showed that my system isn't linking to my USB drive, so I need to look further into this. This means everything is going into the Vera drive, which will ultimately fill up...  I'll take a look at this and try and work out what's happening - it did used to work, so presumably it means the USB isn't getting mounted on startup.