We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 287574 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #615 on: April 15, 2013, 11:54:33 am »
Sorry for the hassle - are you able to get the log file?  If you have WinSCP, then first restart Luup to make sure we view the log when there's an error, then in the terminal box in WinSCP type "cat /var/log/cmh/LuaUPnP.log | grep -i DataMine". Hopefully this would tell me where it's failing.

Chris

Offline Pshomectrl

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #616 on: April 15, 2013, 12:58:37 pm »
Hi Chris
I'm not firm with linux comands. Sorry.
But, i saved the Datamine Debug page just before i tried to reload the UI5.
I will attach a "Datamine Debug Page.txt" file. May be it helps?
The entries i can see in the "LuaUPnP.log" are realy newer than the ones from Datamine Debug.
Hope this file also helps you?
Otherwise, please explain how to get the info you need.
Thanks a lot for your work
Paul

Offline Pshomectrl

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #617 on: April 15, 2013, 01:12:03 pm »
Hi Chris.
May be, this is what you asked for:

/var/log/cmh$ cat /var/log/cmh/LuaUPnP.log | grep -i DataMine
02   04/15/13 18:59:51.940   JobHandler_LuaUPnP::FetchMissingFiles need to download D_DataMine1.json <0x2ad2c000>
02   04/15/13 18:59:51.940   JobHandler_LuaUPnP::m_sMissingFile_set D_DataMine1.json <0x2ad2c000>
01   04/15/13 18:59:52.014   Device_LuaUPnP::LoadDeviceDoc can't load /etc/cmh-lu//D_DataMine1.xml <0x2ad2c000>
02   04/15/13 18:59:52.029   JobHandler_LuaUPnP::m_sMissingFile_set /etc/cmh-lu//D_DataMine1.xml <0x2ad2c000>
01   04/15/13 18:59:52.030   JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 21/D_DataMine1.xml so device 21 is offline <0x2ad2c000>
09   04/15/13 18:59:59.483   JobHandler_LuaUPnP::Run device 21  room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x76aba0 upnp: 0 <0x2ad2c000>
03   04/15/13 18:59:59.500   JobHandler_LuaUPnP::Run ready 13 devices installation 35102692 missing /etc/cmh-lu//D_DataMine1.xml version *1.5.408*  allow_evt: 1 allow_ud: 1 upnp 0 netpnp: 1 reprovision: 0 changed: 0 pMem 0x817000 start 7389184 diff: 1093632 <0x2ad2c000>
09   04/15/13 19:00:18.692   JobHandler_LuaUPnP::Run device 21 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0x9a0ab0 upnp: 0 <0x2b93d000>
50   04/15/13 19:00:21.472   luup_log:21: dataMine: Initialising dataMine System (0.975) <0x2c271680>
06   04/15/13 19:00:21.473   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetDataDirectory was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.473   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.474   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountPoint was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.475   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetManualMount was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.475   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetTimeoutPeriod was: EMPTY now: 12 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.476   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetHistoryEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.476   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetUseBackup was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.477   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetEventsEnable was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.478   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0x9daf40/NONE duplicate:0 <0x2c271680>
06   04/15/13 19:00:21.478   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c271680>
50   04/15/13 19:00:21.479   luup_log:21: dataMine: Installing update of dataMine web application <0x2c271680>
50   04/15/13 19:00:25.829   luup_log:21: dataMine: ERROR: Startup state is 'uninitialised'! <0x2c271680>
06   04/15/13 19:00:25.830   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0x9daf40/NONE duplicate:1 <0x2c271680>
06   04/15/13 19:00:25.830   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2c271680>
50   04/15/13 19:00:25.830   luup_log:21: dataMine: Reinitialising configuration structure <0x2c271680>
50   04/15/13 19:00:25.831   luup_log:21: dataMine: Startup complete <0x2c271680>
01   04/15/13 19:00:41.074   LuaInterface::CallFunction_Timer-5 function deferredStartup failed [string "-- dataMine Plugin for Vera..."]:725: bad argument #1 to 'sort' (table expected, got nil) <0x2cc71680>
06   04/15/13 19:01:06.105   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: SetMountUUID was:  now: EC13-1B32 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2e471680>
09   04/15/13 19:01:30.726   JobHandler_LuaUPnP::Run device 21 dataMine graphing and logging room 0 type urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xe7a9f0 upnp: 0 <0x2ba0a000>
50   04/15/13 19:01:33.111   luup_log:21: dataMine: Initialising dataMine System (0.975) <0x2c33d680>
06   04/15/13 19:01:33.112   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelCnt was: 0 now: 0 #hooks: 0 upnp: 0 v:0xeb4ec0/NONE duplicate:1 <0x2c33d680>
06   04/15/13 19:01:33.112   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: ChannelRec was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 <0x2c33d680>
50   04/15/13 19:01:33.112   luup_log:21: dataMine: Mounting to UUID 'EC13-1B32' <0x2c33d680>
50   04/15/13 19:01:33.528   luup_log:21: dataMine: UUID = /dev/sda1: LABEL="VERA_PLS" UUID="EC13-1B32" <0x2c33d680>
50   04/15/13 19:01:33.543   luup_log:21: dataMine: Found UUID 'EC13-1B32' at '/dev/sda1' <0x2c33d680>
50   04/15/13 19:01:33.565   luup_log:21: dataMine: Mounted: /dev/sda1 on /dataMine type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) <0x2c33d680>
06   04/15/13 19:01:33.566   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountLocation was: EMPTY now: /dev/sda1 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c33d680>
06   04/15/13 19:01:33.566   Device_Variable::m_szValue_set device: 21 service: urn:cd-jackson-com:serviceId:DataMine1 variable: mountType was: EMPTY now: vfat #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 <0x2c33d680>
50   04/15/13 19:01:33.592   luup_log:21: dataMine: ERROR: Unable to open config file for read :: /dataMine/dataMineConfig.json <0x2c33d680>
01   04/15/13 19:01:33.608   LuaInterface::CallFunction_Startup-1 device 21 function initialise failed [string "-- dataMine Plugin for Vera..."]:725: bad argument #1 to 'sort' (table expected, got nil) <0x2c33d680>
01   04/15/13 19:01:33.609   LuImplementation::StartLua running startup code for 21 I_DataMine1.xml failed <0x2c33d680>

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #618 on: April 15, 2013, 01:43:38 pm »
That's the one - thanks.

Unfortunately I can't test this since I'm not at home. I've attached an updated Lua file. To load this from UI5, go to App, Develop Apps, Luup Files. Select the file, click "Restart Luup after upload" and hit Go.

Hopefully this will resolve the issue, but if not another log dump would be useful...

Cheers
Chris

Offline Pshomectrl

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #619 on: April 15, 2013, 01:56:06 pm »
Hi Chris
You are the man............
Thanks, seems this solved my problem.
And.......Chris what ever you do just this time, i hope, it makes fun. Enjoy.
Again, thanks for all your work.
Regards, Paul

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #620 on: April 15, 2013, 05:16:08 pm »
Great - thanks for the feedback. I'll commit this to apps...

Cheers
Chris

Offline ChrisAZ

  • Full Member
  • ***
  • Posts: 104
  • Karma: +1/-0
Re: New dataMine graphing and logging plugin
« Reply #621 on: April 15, 2013, 11:14:20 pm »
I started having the same problems this weekend. Tried the patch you posted and it's all working again. Thanks for the support.

Offline vascodp

  • Sr. Newbie
  • *
  • Posts: 41
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #622 on: April 17, 2013, 06:18:30 pm »

Also - is there a way to combine variables into a single graph plot?  Say for example I want to combine several lights in one room into a single plot - is that possible?  Alternatively, is it possible to plot all energy use over time in the "chart" tab (e.g. the total wattage used over time in a single variable to show total energy usage)?
This isn't something that can currently be done. I did think about making the donut graph selectable between categories and rooms, but I haven't done this. I'm happy to think about this once I've finished getting the history stuff integrated reliably (it's not so simple having to account for random Vera/Lua restarts when you're post processing data). The history may support some of what you're after in that it will derive usage stats on an hour/daily basis (and maybe month), and I then hope to support combining these in different ways (ie by category, room...). But this isn't currently planned for the raw data...

Cheers
Chris

@Chris, I'm starting to explore DataMine (great job !) and I'm looking for a way to monitor whole house and segmented (per device / device type, room level...) energy consumption over a period of time (eg: last day / week / month). Are you planning for such features ? Any clue on how to implement them at the meanwhile ?

Thanks

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: New dataMine graphing and logging plugin
« Reply #623 on: April 18, 2013, 04:46:54 am »
Chris,

since 3 or 4 days my vera became unstable and non-resonsive. I expect this to be because of datamine. i have to reload now 2 or 3 times a day else nothing works anymore. It got better since last evening when 0.977 was auto-updated.

Are more people experiencing this? Is this something you can explain?

2nd quesiton. There is now a section:
Disk Space Total (kb)
Disk Space Used (kb)
Disk Space Free (kb)
but it shows no values...

Cloudl it be my 256Mb stick is full? Or should that not stop vera to work. After reset (power physicaly) it works again for a short amount of time (hours).
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #624 on: April 18, 2013, 08:30:44 am »
@SanderL,
The disk space etc should show numbers. They are showing on my system, although I can't be 100% sure that they are updating at the moment (they should be!). If they aren't showing, then I wonder if your system is updated to the current Lua version (??).

I would doubt that your memory stick is full, and in any case this shouldn't have the effect your seeing. There's been very little change with dataMine in the past 3 or 4 weeks, so I would be surprised if your problems are related to DM software, but if you can send me the logs then I'm happy to take a look and see if there's anything showing in there...

Cheers
Chris

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #625 on: April 18, 2013, 08:39:02 am »
@Chris, I'm starting to explore DataMine (great job !) and I'm looking for a way to monitor whole house and segmented (per device / device type, room level...) energy consumption over a period of time (eg: last day / week / month). Are you planning for such features ? Any clue on how to implement them at the meanwhile ?

So, what I'm planning is usage per hour/day/week and month (from memory). This is largely implemented, but I've not spent a lot of time to test it out. When implemented, it will post process your current data, so it will work on historical data.  This will be implemented in the Lua on your Vera. In addition, the idea is for the GUI to then process this into the same energy categories as we've currently got.

That's the plan ;) Most of the data side is done, but I need to test it, and I'm currently away from home for work quite a bit  :( so I don't want to commit to a timescale right now. The other thing I also wanted to do at the same time is to migrate to a better file structure, since this generates a bunch more files....

Unfortunately I don't have any ideas on how to do this now - sorry.

Cheers
Chris

Offline jouked

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #626 on: April 18, 2013, 08:51:50 am »
What you're trying to do looks a lot like rddtool, which has the logic in it to automatically consolidate historical data. This reduces disk usage dramatically while also speeding up analysis work.

Check out http://oss.oetiker.ch/rrdtool/doc/rrdtool.en.html for more information on this.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #627 on: April 18, 2013, 09:55:19 am »
Hi,
Yes, I'm aware of RRD, but this can't really be integrated into dataMine. Also,  while this does ultimately achieve a similar result, there is a difference - with RRD, it only keeps the regular data - so you loose all the changes in between. With dataMine, you get all the real-time data, and also the hourly/daily etc stats, so hopefully the best of both worlds.

This is already coded up, so it just needs to be tested...

Cheers
Chris

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: New dataMine graphing and logging plugin
« Reply #628 on: April 19, 2013, 06:23:57 am »
@SanderL,
The disk space etc should show numbers. They are showing on my system, although I can't be 100% sure that they are updating at the moment (they should be!). If they aren't showing, then I wonder if your system is updated to the current Lua version (??).

I would doubt that your memory stick is full, and in any case this shouldn't have the effect your seeing. There's been very little change with dataMine in the past 3 or 4 weeks, so I would be surprised if your problems are related to DM software, but if you can send me the logs then I'm happy to take a look and see if there's anything showing in there...

Cheers
Chris

chris,

sent you a PM with logs.

Yes am running latest LUA (622)
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #629 on: April 19, 2013, 03:28:29 pm »
chris,

sent you a PM with logs.

Yes am running latest LUA (622)
Got it - thanks.

Question - do you have a Vera 2? This may explain why the diskspace doesn't work. I know that the fdisk command is not available on Vera 2, and this is not showing up in your logs (it should be between the -4- and -5-). It may be that the df command (which is used to get the disk space) may also not be available. I'll check this when I get a chance over the weekend.

Otherwise I don't see anything strange in your logs. As I said in my previous post, there's not been a lot of changes to dataMine in the past few weeks, so while I wouldn't rule anything out, I'd be surprised if there was a link here...

Cheers
Chris