We have moved at community.getvera.com

Author Topic: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)  (Read 13958 times)

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #15 on: December 08, 2014, 11:33:59 am »
Overall are you happy with what you describe above to report to SEG for both GEM and other Vera energy reporting, e.g. smart plugs?

Yup, completely.  I look at the data in SEG daily after I make significant changes in the house.

I report data to SEG from 2 different sources:

  • Weather, House-Temperature & Vera Performance data from my Vera
    These streams use the technique above to push that to my SEG Account.
  • Brultech GEM and ECM-1240 Energy Data
    These streams are published using btmon.py running on a RaspberryPi.

I moved from using Vera as the conduit for the Energy streams in order to reduce the load on Vera (I have a lot of stuff on Vera that needed to be more real-time than the energy monitoring stuff)

Is SEG "smart enough" to allow the user to avoid double counting the smart plug loads which are also included in the whole-house-meter loads?

SEG has no knowledge of the relationship between the [data] Streams you register with it.  It sees Devices (think "Main Panel", "SubPanel", "Vera") and the Streams that come from those Devices (think "Upstairs Temperature", "Grid Energy", "Solar Energy", "Office Energy", "CPU Usage")

Once you've connected the Devices and established the Streams, you use the SEG UI to build Dashboards to display the UI for those Streams (you pick the UI/Graph styles, as well as what Streams to display in those graphs).  It's a fairly advanced graphical Portal builder for "energy" data. 

You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system.  I do this to "rollup" certain low level stats into more useful stuff.

eg. TotalEnvironmentalEnergy = MainPanel/AC Energy + SubPanel/Furnace Energy

So this is where, indirectly, relationships are being captured.... by how you build Graphs, and how you build these Computed Streams.  All of this is done in a UI, and is easy to do.

I am not a scripter (but can follow along and fill in basic blanks, etc.)  Do you expect this is an appropriate solution or will I be jumping off a clifF?

The hardest part of my setup is the bymon.py running on Pi.  Since you need to setup the Pi, btmon.py, deal with Serial Ports (for attachment to ECM-1240) and Wifi (for dealing with my GEM) and script it all to start on reboot (etc).

Not impossible, but it would take little searching to pull the parts together.


That said, if you've got a Vera, and it's not doing too much else, then you could just run the Brultech Plugin and it'll do the work of acquiring the data for you (esp if you've got the GEM on WiFi and/or Ethernet).  You'd then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG.  That's all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id's)

Offline fullmoonguru

  • Full Member
  • ***
  • Posts: 207
  • Karma: +0/-3
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #16 on: December 08, 2014, 11:53:08 am »
I find SEG really confusing.  It looks very flexible and powerful and I'm sure I could get it dialed in given enough time with it, but I wouldn't call it an intuitive UI.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #17 on: December 08, 2014, 12:16:33 pm »

...You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system.  I do this to "rollup" certain low level stats into more useful stuff.

...That said, if you've got a Vera, and it's not doing too much else, then you could just run the Brultech Plugin and it'll do the work of acquiring the data for you (esp if you've got the GEM on WiFi and/or Ethernet).  You'd then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG.  That's all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id's)

Guessed, thanks! Those were the answers I was hoping to receive...

As to calculated streams on SEG:  Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".  In thinking about this, I suppose one could just to the upper tier rollup manually from each device, vs as a two-tier roleup.

As to Vera vs PI and complexity:  I have an ISY with Zwave module and find it to be VERY reliable/stable.  However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN).  Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data.  Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc.  I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-).  In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS:  exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Finally, you mention GEM wifi or Ethernet.  Are these options in the GEM one buys, or ad-ons/hacks to do after the fact.  I have not purchased yet.  I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

Thanks again!

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #18 on: December 08, 2014, 12:53:56 pm »
As to calculated streams on SEG:  Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Both Add and Subtract, of any Stream with the same Units of measure.  In my case, I have some streams that are "rolled up" at the source, based upon where I've clamped...  sometimes I re-roll them up from the component streams, in SEG, since there might be errors in measurement that I'm interested in seeing.

I'll post a series of screen-captures shortly to illustrate these bits...

Quote
Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".

Yep, Calculations are just another Stream, so you can composite new Calculations based upon others.  Sometimes this limits the granularity of the result (to 30 minutes) but in general that's not an issue.

Quote
As to Vera vs PI and complexity:  I have an ISY with Zwave module and find it to be VERY reliable/stable.  However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN).  Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data.  Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc.  I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-).  In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS:  exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Take a look at the stuff that Teken is doing on the Brultech forums.  He has ISY gear.   My Vera is a Vera3, so it has more memory than a VeraLite, but the same as the newer VeraEdge. 

When running a reasonable load, Vera is good at Lighting/Z-Wave control (etc).... but when it runs out of steam, you're hosed.  They don't currently have a "bigger" (and, in all honesty, less-buggy) option for power-users so I paired back my Vera3 to running the critical stuff, and have moved everything else to Pi's (and now, via openHAB, I'm expanding/migrating to my Mac)

Quote
Finally, you mention GEM wifi or Ethernet.  Are these options in the GEM one buys, or ad-ons/hacks to do after the fact.  I have not purchased yet.

You buy them as part of the GEM when you order it.  Mine is a 1st Gen GEM, which I bought with ZigBee, since I already had ECM-1240 units with the ZigBee module (and the related ZB <-> Ethernet Bridge).  At the time the ZB Modules were socketed, so I replaced mine with a RovingNetworks RN-XV module (now part of MicroChip).

In the latest models, these are simply options that you buy with the GEM.  I like WiFi here, since I don't have cabling for Ethernet anywhere near my GEM (and I don't want to either, since my GEM is right next to my SubPanel)

In my setup, I have a WiFi rPi which is USB-Serial attached to a ECM-1240, reading the Main-Panel.  This is on one side of the garage and it's using WiFi to get access to the GEM, which is next to the Sub-Panel on the other side of the Garage.  The rPi aggregates the data and phones it up to SEG (as well as PVOutput, for the Solar bits)

Quote
I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

You mean the DashBox.  It still appears to be somewhat under development.  It's performing the role of what I'm using a Pi + SEG for, albeit using "locally-stored" data, which will be of interest to some folks.   I may do this sometime, to replace the Pi and make my solution more  consumer-friendly... handy if you ever want to sell you house with this stuff as "features"  ;)

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #19 on: December 08, 2014, 12:56:42 pm »
A few more SEG images, showing the default Stream-list, a default graph for one of the Streams, as well as custom dashboard tabs I've built for Energy Overview/Detail and Temperature monitoring.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #20 on: December 08, 2014, 01:12:54 pm »
guessed, very cool and super helpful.  Thanks very much!!

PS:  It was after reading Taken's "book" on ISY / GEM integration that prompted me to look at Vera for energy reporting ;-)

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #21 on: December 18, 2014, 07:08:05 pm »
Hi again

I apologize in advance for my inexperience.  I turned on my Vera today for the first time, added the Nest plugin, got the Sonos plug-in running, got the ISY manual loading plugin running (needed to learn about WinSCP for that) and am now trying to climb the hill described in this thread...

In Vera I added a smart plug and Nest T-stats (via the nest plugin) and have created a "Site" and a couple of "Devices" at SEG and see how to get the token numbers.  BUT: I am not sure how/where to write the script guessed provided on the prior page.  I'm not a programmer and don't know Vera, so my questions are VERY basic...

  • Where do I write your script (with the edits relating to my SEG/Vera devices)
  • Are all the parts of your script used for for a single device,e.g. a smart plug, or just part of the script (which part)
  • Would you mind giving a basic step-by-step for setting up a single smart plug, for example, to report to SEG?  I will then use that to expand on my own as much as possible

THANKS YOU VERY MUCH AND SORRY TO BE SUCH A NEWBIE
« Last Edit: December 19, 2014, 11:33:11 pm by ccclapp »

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #22 on: December 19, 2014, 02:10:18 pm »
@ccclapp,

  • 1) The script belongs in the Startup Lua section of Vera.
    In UI5, this is under:

        Apps -> Develop Apps -> Edit Startup Lua

    Don't forget to confirm the changes, and Save/Reload Vera to force this to startup.

  • 2) The Script just does a pick-n-pull of any set of "State Variables" you want to send to SEG.
    This can include any combination of things really, from any Vera Device, but typically it's Power, Energy & Temperature related values.

    You need only know the correct ServiceId/StateVariable, and Device#, that you want to Monitor.  It's a list, so it can be as many of these as you want.  The code monitors for changes in the value, and just sends it over to SEG.

    Note that SEG may not want you sending 100's of values/second, and it will take your "SEG Device" offline if you sent at a high rate.  If you're using the Brultech Power Monitor Plugin, set the GEM to refresh data every 60-90s seconds (the default is 5s, mine is 90s)

  • 3) Step by Step...
    Not totally step-by-step, but the basic process is:

    a) Modify the script to include YOUR SiteId, as well as the list of DeviceId/ServiceId/StateVariable combinations
    Save this, and Vera will start pushing data to SEG, on your SiteID.

    b) Login to SEG in a browser, and navigate to:
        My Things > Tools > Discoveries

    and set it running to discover your stuff.  After a few minutes, it'll see your feed data and present you with a UI to include it.
    At this point, the Feeds will become a Device/Stream combination.  You'll have a Stream, within the [SEG] Device, for each thing you setup in the Vera/SEG startup script.

    If you're not seeing anything here, you can also try:
        My Things > Tools > API Trace

    and this will let you see what Vera is sending over.  After a few minutes, if you still see nothing, then either Vera isn't sending stuff OR, more likely, you're using the wrong Token for SiteID.


    c) Edit the Streams to add any metadata that you want.
    This might include better Title/Description(s), along with Units-of-measure (in cases where SEG wasn't able to introspect it from the Stream)


    At this point you'll have basic graphs and the data will start accumulating.  From there, once you have some data, it's probably a good point to build a simple one-sensor dashboard.


Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #23 on: December 19, 2014, 02:17:38 pm »
...wonderful.  Thanks guessed!!

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #24 on: February 18, 2015, 02:54:49 pm »
do you guys know it the script should work in UI7 on vera edge?  i copied/pasted it in, edited per instructions , but don't see the connection at SEG.   possibly i made an error and will post screenshots, but first figured I'd inquire if it should work with UI7

thanks!

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #25 on: February 18, 2015, 03:05:43 pm »
do you guys know it the script should work in UI7 on vera edge?  i copied/pasted it in, edited per instructions , but don't see the connection at SEG.   possibly i made an error and will post screenshots, but first figured I'd inquire if it should work with UI7

It's a fairly simple script, so there's no reason it shouldn't work on all Lua-enabled MiOS engines.  That said, I've not tested this specific item on UI7, so it can always be a dice-roll.

Double check what you C&P'd, as well as any errors reported in /var/log/cmh/LuaUPnP.log since it'll tend to show up there if there's a typo.

Also, when looking at SEG, there's a "Trace" mode in their User-toolbox that can be used to see _what_ is being pushed to SEG.  The most common problem, if you're not seeing data there, is using an incorrect token in the push.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #26 on: February 20, 2015, 01:34:06 pm »
Hi Guessed

Two things:

#1 I want you to know I linked to your script on the ISY forums in hopes it would help us figure out how to transmit from ISY to SEG.  I hope that is ok with you.  Here is the post:

http://forum.universal-devices.com/topic/12682-energy-monitoring-circuits-plugs-interface/?p=132278

#2 My Vera was having issues the other day.  That is why I did not connect to SEG.  Your script works fine in UI7.  I will separately post a few observations/questions.  Once I better understand those, I will try to write another post in this thread spelling out a newbie step-by-step for this from the info you and others have provided above (to help consolidate the info in these 6 pages for other beginners, like myself)...

Thanks!

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #27 on: February 20, 2015, 02:21:41 pm »
Yeah, it's public content/posting here, so feel free to point to it.

Feel free also to create a break-out thread that just handles SEG, as needed.  I buried it here because it's the thread I've been using for all updates on my energy monitoring, and curbing energy usage, discussions.

As long as it remains a free-to-use option, with open source-code disclosure, I'm ok with others using it anyway they want ;)

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #28 on: February 20, 2015, 02:25:03 pm »

Feel free also to create a break-out thread that just handles SEG, as needed. 

Either is fine with me.  Would you prefer the SEG linking info be in a separate thread.  If you don't, I'd stay here.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #29 on: February 20, 2015, 03:01:55 pm »
Probably easier for users to find if it's a new Thread.  Feel free to start a new one, and you can just add a link back to here for reference (and link a post here, to the new thread also, so people can stumble across either)

That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)