We have moved at community.getvera.com

Author Topic: Power/Energy Monitoring Basics.  (Read 21207 times)

Offline John Bentley

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
Power/Energy Monitoring Basics.
« on: May 19, 2013, 12:23:46 am »
I have some basic Power/Energy monitoring requirements: To view, log, and graph present (real time) and historical wattage draw. Beyond this basic requirement I'd like also to be able to export wattage logs (for example to play with in Excel).

These are obviously common requirements. However, as a newbie who is evaluating whether to buy a Vera (lite) I'm not clear on how this is done. While I'm specifically interested in Appliance level monitoring, rather than Whole House level monitoring, the following considerations are, I think, level agnostic. After poking around the Power Management sub forum I have the following (probably wrong) picture in my head:

Firstly, you need hardware that is known to report Watts. The leading solution here, for appliance rather than whole home monitoring, is The Aerotec (aka "Aeon Labs") Smart Energy Switch.

Secondly, you need to install a plugin to support your hardware device, here a Aerotec (aka "Aeon Labs") Smart Energy Switch plugin (??).

Present (real time) wattage can be readily viewed in the Vera Web Interface > Devices tab, as illustrated in the FAQ (SmartSwitch) Controlling SmartSwitch From The Dashboard.

For the purposes of logging, viewing, and graphing historical wattage draw the Energy tab (on the Vera Web Interface) does not provide this functionality out of the box. Rather the Ergy Plugin is offered by Vera, on the Energy Tab, as something you sign up for and install. The Engy Plugin is a made by a third party. It is free to use. It logs to the cloud, not the Vera controller.

This might provide all the basic Appliance (or Whole Home) Power/Energy monitoring requirements you need.

There are alternative Power/Monitoring plug-ins. The YouLess EnergyMeter for example, like Ergy, stores logs in the cloud.

The leading alternative, however, is the dataMine Plugin. This both allows quite detailed interrogation possibilities, is also free, allows logging of variables in addition to energy/power variables (e.g. temperature from a temperature sensor), and logs locally (to a USB stick plugged into the Vera). These local logs can be copied to Excel (From the dataMine Manual "dataMine records data in CSV files (Comma Separated Value)").

In what ways is this picture wrong and incomplete? Of the three Energy Monitoring plugins (Ergy, YouLess, dataMine) is there a consensus around which one to go with? Have I missed a superior, or another leading, solution?

Edit01: Minor grammar.
« Last Edit: May 19, 2013, 03:17:02 am by John Bentley »

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: Power/Energy Monitoring Basics.
« Reply #1 on: May 19, 2013, 02:32:25 am »
Once again, John, you have the fundamentals in much greater focus than many who start out.  Both the hardware and software are, of course, key.  Let me start with my views on software (limited and to some extent biased, but then whose is not?)...

Software: The built-in energy monitoring functionality of Vera, does provide some primitive logging and graphing but relies on 'cloud' storage with MiCasaVerde.  It is to be avoided.  The Ergy plugin has raised many questions - some about data protection - and, as you say, is also in the 'cloud'.  I'm looking for the relevant thread and will edit this post when I find it.  The other thing about the custom solutions is that they are generally about just the power.  The real value of datamining is to correlate multiple measurements, for example: power with temperature or with house occupancy.  My quick recommendation: Datamine - every time.  Local logging to USB, functional, fabulously well supported (although only data logging and plotting - you will have to export to do fancy calculations).

Hardware: Although you are rightly interested in Appliance level monitoring - which can be done with a number of power-monitoring modules - I would urge a keen interest in whole-house data.  I started with clamp-on measurements, but I think I'm right in saying these measure VA rather than Watts, so if you have significant inductive loads you will get the wrong answer - seems to be a problem in particular at low loads.   I switched to pulse counting data loggers which bring the added bonus of using the accuracy of your existing household meter  along with the capability to remote-read (and log) your actual meter reading.  A proper reading, taken regularly, can give a better overall picture than individual instantaneous power snapshots.  If you want the ultimate in HA data integration, go and browse http://www.bwired.nl/.  I use the NorthQ data logger.

I'm sure others have a wealth of experience to share.  I look forward to their advice.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline John Bentley

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
Re: Power/Energy Monitoring Basics.
« Reply #2 on: May 19, 2013, 06:55:22 am »
Thank you again akbooer. That's helpful.

I'm glad to know I'm not missing any obvious alternative solution.

What you've written would be enough for me to want a hardware meter properly wired for whole-house level monitoring, to complement appliance level monitoring. Only at the whole-house level do you, of course, get the full and interesting data. The clamp on meters, yes, I wouldn't expect to be accurate (I'm glad to have my intuitions confirmed by your experience). However, since I'm renting, whole-house monitoring is something I will have to do without.

On the energy monitoring plugins: I remain vehemently ambivalent about cloud solutions in general, and for this specific use.

For example, on the plus side micasaverde's cloud account mechanism looks like a clever way to simplify secure remote access for most users (I installed SSH myself once and I wouldn't wish that upon Satan). Cloud hosted services can also be more readily updated by the devs rather than having end users update and reconfigure upgrades. On the minus side, to take the specific example to hand, sending appliance energy data out to the cloud (as Ergy does) creates a greater security attack surface (even if the data is properly secured in transit).

In the end unless there is a good reason to use the cloud I generally try to avoid it. That's my (partially unjustified) bias.

I've discovered a bit more detail about the Ergy Plugin. The Echo Labs website, www.ergyenergy.com clarifies most things:
Being a software developer myself means two things. Firstly, I appreciate that for all the effort they've put into what looks like a sweet product it's reasonable they'll want to recoup costs by offering a subscription service. Secondly, that we end user/developers should be able to come up with a free, competing, and possibly open source solution ourselves.

Which is exactly what Chris, in offering us the dataMine plug-in, looks like he has done. There seem to be two considerations in evaluating it's use:
  • Copying the data from the USB stick attached to the Vera.
  • The polling frequency.
For copying data you could simply unplug the USB stick and plug it into your Desktop. For an automated process the dataMine Manual, "Advanced Topics > Backing up dataMine data" P20, has a procedure by vespaman. That seems readily adaptable for windows. So while there might be a bit of work to do that's not a roadblock.

On polling frequency the manual, "Advanced Topics > Under the hood" P19, mentions
Quote
dataMine works by looking for changes in device variables ? it is not periodically logging data based on time, it simply logs changes. This is important to note when thinking about how a graph is rendered ? for status type data (eg. on/off), the graph needs to account for this to avoid diagonal lines.

I'd imagine I'd prefer the option to set a constant polling interval. In practice, however, is the variable-change polling no big deal (because you just infer data values and/or variables change frequently enough)?

A forth option, apart from Ergy, YouLess, and dataMine, would be to get at the raw vera logs themselves (either to use the raw logs directly or with a view to developing another plugin). Looking at the micasaverde FAQ for the plugin for The Aerotec (aka "Aeon Labs") Smart Energy Switch I see that there is a "Poll this node at most once every" setting. So presumably the raw, low level, vera energy logs exist somewhere at the desired polling frequency (time based rather than change based).

Would The Aerotec (aka "Aeon Labs") Smart Energy Switch be logging to the vera log detailed at http://wiki.mios.com/index.php/Logs ?

Edit01: Minor grammar; Added "at the desired polling frequency (time based rather than change based)."
« Last Edit: May 19, 2013, 07:02:23 am by John Bentley »

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: Power/Energy Monitoring Basics.
« Reply #3 on: May 19, 2013, 08:12:39 am »
Quote
since I'm renting, whole-house monitoring is something I will have to do without.

Not really.  The data logger power monitors are battery powered and have stick-on sensors (easily removable) which count the installed meter's own LED pulse (or, if you're old-fashioned, the bar on the rotating disc).  Unless, of course, you're only renting part of a whole house.

Quote
micasaverde's cloud account mechanism looks like a clever way to simplify secure remote access for most users
Exactly so, a good use of the cloud.  You may (or may not) be surprised that there are still people who punch holes in their own firewalls to avoid using this (heaven help them).

Quote
In practice, however, is the variable-change polling no big deal (because you just infer data values and/or variables change frequently enough)?

The philosophy here has (I think) been to minimize logged data volume and avoid polling by using event-driven callbacks.  Easy enough to interpolate (or take nearest neighbour) is subsequently required.

Quote
get at the raw vera logs themselves

Not sure how that particular module shows up in the logs.  Probably MUCH easier to simply write a scene which polls the values you want on a specified schedule.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Chris

  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: Power/Energy Monitoring Basics.
« Reply #4 on: May 19, 2013, 08:13:46 am »
On polling frequency the manual, "Advanced Topics > Under the hood" P19, mentions
Quote
dataMine works by looking for changes in device variables ? it is not periodically logging data based on time, it simply logs changes. This is important to note when thinking about how a graph is rendered ? for status type data (eg. on/off), the graph needs to account for this to avoid diagonal lines.

I'd imagine I'd prefer the option to set a constant polling interval. In practice, however, is the variable-change polling no big deal (because you just infer data values and/or variables change frequently enough)?
One of the next things I'm working on is statistics, which will collate all the data over a certain time period - 1 hour, 1 day, 1 week, 1 month. This may do what you're after. This is pretty much all coded, but as I had to change the data format, this has taken priority (and the day job keeps shipping me abroad!).


A forth option, apart from Ergy, YouLess, and dataMine, would be to get at the raw vera logs themselves (either to use the raw logs directly or with a view to developing another plugin). Looking at the micasaverde FAQ for the plugin for The Aerotec (aka "Aeon Labs") Smart Energy Switch I see that there is a "Poll this node at most once every" setting. So presumably the raw, low level, vera energy logs exist somewhere at the desired polling frequency (time based rather than change based).
There are no "raw vera logs" as such - this is one of the things  dataMine does - it produces a log. If you wanted, you can certainly interrogate the dataMine logs yourself. dataMine logs the data as it comes in - I've always had the philosophy to record the raw data - you can always sub sample etc, but you can never go back to raw data. Changing the polling period won't necessarily do what you want either - for an energy monitoring device such as a whole house device, it might (maybe!), however if you wanted to log other energy devices (sockets, lights etc), then they will change when you change the state of the device. For these devices, you actually only get updates when they change state... Basically, different devices will end up with different sampling rates.

My suggestion would be that if dataMine doesn't do what you want once the stats package is included, either let me know what you're after, and if it's a useful suggestion that would be of general benefit, I'll look at implementing it. Or, you could use the dataMine data yourself - either by access the files, or exporting the data.

Cheers
Chris

Offline John Bentley

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
Re: Power/Energy Monitoring Basics.
« Reply #5 on: May 19, 2013, 10:29:38 am »
WHOLE-HOUSE MONITORING

Akbooer thanks.

The data logger power monitors are battery powered and have stick-on sensors (easily removable) which count the installed meter's own LED pulse (or, if you're old-fashioned, the bar on the rotating disc). 

Ah thanks! That corrects my false assumption that you had to wire in the whole-house meter for accuracy. A good Youtube video "NorthQ Power Reader" details how this works by merely externally sticking on the power reader. The video also show why we could expect this to be rather accurate. As it turns out I'm in units and the power meter is downstairs in a common area (quite apart from potentially requiring negotiation with others it might be out of the 15 metre wireless range). So in the end I'm not so interested in it. However, this information is good to be in the thread for the interest of others.


GRAPHING HISTORICAL DATA

Chris, thanks very much for your info and your efforts on the plug in. There is a lot of work already done, I can see!

One of the next things I'm working on is statistics, which will collate all the data over a certain time period - 1 hour, 1 day, 1 week, 1 month. This may do what you're after.

At a quick skim of your manual already shows some good looking graphing functionality. If that requires further coding to better handle choosing time intervals then I would very much look forward to that. It probably would sufficiently handle the historical energy monitoring I'm after.

I'd suggest also intervals 1 qtr, 6 month, 1 year, all the data (and perhaps you already had that in mind).


DISPLAYING REAL TIME (PRESENT VALUES)

For real time data the dashboard tab (in progress) looks good too. I'd just suggest that this be robust enough to handle merely the sum of application level meters (when whole-house metering is unimplemented). It looks like your design might already take that into account.


DATA EXPORT

If you wanted, you can certainly interrogate the dataMine logs yourself.

Yeah, that might be a good option. Once I get the logs in Excel then I can use all the advanced features to play with the data.

but you can never go back to raw data.

What did you mean by that?


POLLING INTERVAL

Quote
if you wanted to log other energy devices (sockets, lights etc), then they will change when you change the state of the device. For these devices, you actually only get updates when they change state... Basically, different devices will end up with different sampling rates.

Do you mean:

* While The Aerotec (aka "Aeon Labs") Smart Energy Switch polls at a regular interval, other devices do not. Therefore when coding dataMine the most universal approach was to log at state-change; or
* No device, including The Aerotec (aka "Aeon Labs") Smart Energy Switch, polls at regular intervals (if this is so what does "Poll this node at most once every" do?).

?

Was akbooer right that
Quote
The philosophy here has ... been to minimize logged data volume and avoid polling by using event-driven callbacks.

?

It probably doesn't matter if, as akbooer suggested, it is "Easy enough to interpolate (or take nearest neighbour)" when looking at state-changed data.


THE RAW VERA LOGS APPROACH

There are no "raw vera logs" as such - this is one of the things  dataMine does - it produces a log. ...

You mean the raw vera logs, as described at  http://wiki.mios.com/index.php/Logs, simply aren't in a variable interrogating friendly format and that even with akbooer's suggestion ....

Not sure how that particular module shows up in the logs.  Probably MUCH easier to simply write a scene which polls the values you want on a specified schedule.

... there would be so much effort needed to write a log parser that you'd be better off starting from the other end: generating the logs you want (as you've done with dataMine) ?