We have moved at community.getvera.com

Author Topic: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)  (Read 13957 times)

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #30 on: February 20, 2015, 03:40:36 pm »
That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)

That sounds perfect... I'm banging my head against a brick wall on this, but will withhold comments until the new thread is started.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #31 on: February 20, 2015, 04:12:11 pm »
That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)

That sounds perfect... I'm banging my head against a brick wall on this, but will withhold comments until the new thread is started.

I was too but then got it. Once I did I saw that Guessed's instructions were perfect. I had to pick from a few posts to pull it together and again once I did I saw the simplicity of what he created. I'm on my way up to the ski mountains at the moment so it might be Sunday or beginning of the weekfor a new thread. In the meantime, I'm sure your basic questions would be answered here

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #32 on: February 27, 2015, 07:28:30 am »

I'm on my way up to the ski mountains at the moment so it might be Sunday or beginning of the weekfor a new thread. In the meantime, I'm sure your basic questions would be answered here

sorry I was delayed in doing this this week. I will try to work on it today or this weekend?

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #33 on: March 11, 2015, 04:08:09 pm »
I forked it out this morning.  Feel free to use extend the discussion here.

As a side-note, I also made an openHAB version of the script, for those that are looking at that route.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #34 on: March 20, 2015, 12:47:42 am »
Vera SEG Step-by-Step Guide

The following is an (overly detailed) beginners step-by-step guide for linking Vera devices to SEG.  100% of this comes from @Guessed?s script and his instructions in this thread.  I am doing this because as a beginner it took me a while to understand what he succinctly wrote.  In case others are like me, this may be helpful:

INTRODUCTION: 

Purpose: Monitor energy use and other stream data over time on a great, flexible and easy to use website accessible from all devices.  SEG (Smart Energy Groups) is such a site.  Thanks to @Guesseds script we can easily send all stream data from Vera to SEG.  Stream date includes things like (i) energy data from smart plugs, home circuit panel or individual circuits, (ii) environmental conditions like indoor/outdoor temp, wind, humidity (iii) pretty much anything else that changes over time. 

Examples:

Below are examples of the energy monitoring products.

http://www.amazon.com/Aeon-Labs-DSC06106-ZWUS-Z-Wave-Energy/dp/B007UZH7B8/ref=sr_1_1?ie=UTF8&qid=1426806219&sr=8-1&keywords=zwave+smart+plug
http://www.amazon.com/Aeon-Labs-Aeotec-Z-Wave-DSC25-ZWUS/dp/B00H5U40L8/ref=sr_1_2?ie=UTF8&qid=1426806219&sr=8-2&keywords=zwave+smart+plug
http://www.amazon.com/Aeon-Labs-Aeotec-Z-Wave-DSB28-ZWUS/dp/B00FKJBUX2/ref=sr_1_2?ie=UTF8&qid=1426806166&sr=8-2&keywords=zwave+energy+monitor
http://brultech.com/store/

In my case I have about 10 smart plugs connected to AV and PC equipment, sump pump and other high energy users I want to monitor at the plug level.  I also have the Aeon HEM on my panel and have a GEM on order to grab all circuits.  I also pick up indoor temperature and humidity from my NEST t-stats and outdoor temp, wind, etc. from the Underground Weather plug-in.

Context:  Going into this I was using 1,600 KWH/month 20,000 KWH/year and DO NOT have electric heat or use AC.  Thats a huge #!! By using a kill-a-watt meter and smart plugs to map every device?s on/standby electric draw and then Vera (and an ISY) to control everything  I have cut this almost in half over the past month and have more to do.

INSTRUCTIONS:

On Vera

1)  Copy into a text file to edit @Guesseds script in post #5, page 1 of this thread, http://forum.micasaverde.com/index.php/topic,31212.msg175678.html#msg175678

In your pasted text file, edit only this highlighted portion of the text, as follows:

SEE ATTACHED IMAGE FOR EXACT LINES TO EDIT

 
2)  Where is says <yourSEGSiteIdHere>", remove the < and > symbols and put your own SEG Site ID token between the quotation marks.  (In the SEG instructions below I?ll describe how to set up SEG and get that token, or if you want see/do SEG step #1, below and then continue with the Vera Steps.)

3) The list after the words SEG_DEVICES = { are to be edited as follows:

- The concept is you will have one line for each Vera data stream you want to send to SEG.  The example in the script includes three streams (3 lines).  Note: a single Vera device could have more than one stream and thus more than one lines, e.g. Nest has temp and humidity.  Smart plugs have WATTS and KWH. You decide what to report and make a line for each.  Each line will create a stream item at SEG (we?ll get to SEG in a moment).

NOTE:  That you are about to do is fully editable in Vera.  Therefore you may want to start with 1 or 2 lines (for 1 or 2 streams), as described below and then add additional lines once it is working for you.

3(a) Using the 1st line of the script SEG Devices table as the example, where the text says: stream='t_upstairs', you replace the text inside the single quotes with whatever you want to SEG to identify that stream by.  This does not have to be the name of the stream on SEG, you can change the SEG stream name, but not the SEG stream ID.  You are creating the stream ID here.  I don?t know if SEG likes spaces, caps, etc, so dont use them.

- For energy and temp streams (and maybe others, but I dont know) a prefix in the stream ID tells SEG to use WATTS, KWH or Degrees.  You can edit that at SEG so don?t need to use prefix.  Prefixes are e_ for energy (Watts), p_ for power (Watt Hours) and t_ for temp (Degrees).

3(b) Where the text says deviceId=335 you put the Vera device #.  THIS IS NOT the ZWAVE NODE #.  In UI7 the Device # is seen at the top of the Advanced tab on the device (just below the green header)

3(c) I am skipping to the end of the text line where it says serviceVar="CurrentTemperature"}, (Ill come back to the middle part below).  You will replace the text between the double quotes as follows:  On Vera go to the Advanced tab for the device.  Go to the Variables column.  Find the variable you want to stream, e.g. WATTS, KWH, CurrentTemperature, CurrentLevel, WindSpeed, or whatever variable you want that Vera lists for that device.  Copy/paste the variable field label name.

3(d) Now moving backwards in the line from the script text, where it says serviceId='urn:upnp-org:serviceId:TemperatureSensor1' replace the text between the single quotes as follows:  On Vera hover your mouse over the service variable label you used in step 4, above.  When you hover you will see the service ID text to use.   

Examples are: 
urn:upnp-org:serviceId:TemperatureSensor1,
urn:upnp-micasaverde-com:HumiditySensor1,
urn:micasaverde-com:serviceId:Weather1.

NOTE on steps 3(c) & 3(d):  I have NOT yet properly configured my Vera/SEG for the Weather Underground Plugin streams and possibly not all service variables and corresponding service IDs are supported by the script.  More likely I made a typo...TBD

Here is a link to the Vera Wiki discussing Service Variables and Service ID:  http://wiki.micasaverde.com/index.php/Luup_Variables#Device_category_3

3(e) On Vera, the final step is to paste your edited script into the Startup Lua, as follows:  (i) copy the text file you just edited.  On Vera navigate to Apps ->Develop Apps -> Edit Startup Lua.  Paste the text into the box. Click Go.  Enter ctl-F5 to restart the Vera engine, now incorporating this script.  You are now done on the Vera side.

On SEG

1) Open a free account here: https://smartenergygroups.com/ . Then click My Things -> My Sites and create a site.  At the bottom of the new site info box you it says API Details and below that Site Token.  The number below is what you want.   This is the number you put in Vera Step #2 above where the script says says ?<yourSEGSiteIdHere>", (As stated above remove the < and > symbols and put your own SEG Site Token between the quotation marks).

If you previously created a SEG Site find the token by clicking on your site thumbnail image and then clicking the pencil and paper icon (the Edit icon) Under API Detail is your Site Token. (Be sure you are getting the Site Token, not a Device or Stream token).

2)  On SEG create a new Device.  Name it anything, but be sure to make the Node Name vera (or you can change the node name, but also edit the Vera script changing {node='vera',   to the name you use on SEG).  Make Device Type be Display.

3) Now you are ready for SEG to receive the streams from Vera.  To do this on SEG click Tools on the right of every screen, then Discoveries -> Discover.  Wait a few minutes and let it run.  It is looking for the stream data Vera will send.

NOTE:  Per @Guessed, Vera only sends data when it changes (or SEG only logs it when it changes).  Either way, I recommend on Vera select Poll Now for the device(s) you are sending to Seg.  You may then want to change the data on Vera, e.g. turn smart switch on/off and Poll Now each time to have changing data for SEG.    I don?t know if you need to keep Discover running on SEG until it picks up all your streams, or just the 1st.  In any event you can repeat Discover.  To keep Discover running I log into SEG on another browser tab and click on my vera Device in SEG to see if any streams have been created (while Discover continues in the other browser tab).

4) Once 1 or more streams show up in SEG, you can click see more detail and time periods and also to edit them.  Mine usually default to Kilo eg KW and KWH (vs W and WH).  To get to W WH edit the stream and select None if Kilo is selected.  Also, on the edit screen set the Type of Data to correspond to the Vera Service Variable you picked, e.g. WATTS. 

Just remember, both SEG and the Vera script can be changed if things aren't perfect.

On SEG there are lots of things you can do with the data as @Guessed describes and shows in screenshots earlier in this thread.

That's all for now

Have fun!

Thanks @Guessed!!!!!!!!!!!!!!!!!!!!!
« Last Edit: March 20, 2015, 12:55:18 am by ccclapp »

Offline scyto

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #35 on: March 22, 2015, 11:25:38 pm »
1) Why did you choose display and not energy meter on the SEG site.

2) I would recommend that for energy devices on SEG one defines both the energy and power for a device as such my Aeon SmartStrip looks like this
Code: [Select]
SEG_DEVICES = {
    {node='vera', stream='p_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"}
   }

3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY.  On the Vera it rounds up to 8 in the UI - does this mean the script wont send anything because it didn't detect a change (PS turning off the strip made no difference either).

At this point i have the 2 streams appearing in SEG, but no data. It also shows only as being online in SEG 15 minutes ago...
« Last Edit: March 22, 2015, 11:27:58 pm by scyto »

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #36 on: March 23, 2015, 12:24:44 am »
1) Why did you choose display and not energy meter on the SEG site.

I moved to pushing Energy data, directly, using BTMon.py to improve Vera stability.  It's very reliable, so I only needed to push infrequently changing aux-data from Vera. 

I've recently stopped pushing any data from Vera, and now push all of my SEG data directly from openHAB.

Quote
2) I would recommend that for energy devices on SEG one defines both the energy and power for a device as such my Aeon SmartStrip looks like this
Code: [Select]
SEG_DEVICES = {
    {node='vera', stream='p_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"}
   }

Yup, different (but related) quantities.  Both are quite useful, depending upon what you're trying to analyze/fix.

Quote
3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY. 

Rev 1.  That script is fairly simple, and only pushes data-changes.  The openHAB one pushes on change, but also every 2 minutes.  This can help with some graphing tools, as some don't handle data streams of vastly differing period.

The script itself is agnostic to what's generating the data, and on what frequency.  Always room for tuning..  ;-)

Quote
On the Vera it rounds up to 8 in the UI - does this mean the script wont send anything because it didn't detect a change (PS turning off the strip made no difference either).

At this point i have the 2 streams appearing in SEG, but no data. It also shows only as being online in SEG 15 minutes ago...

At some point, tiny values for Wh and/or W are going to be in the level of error (typically 1-3% for a GEM, depending upon the CT used) so I wouldn't sweat that resolution too much.

... esp when your Daily consumption is going to be measured, and billed, in KWh, not Wh  8)


For debug, use the SEG "Tools" menu, and enable Trace for a bit.  It'll give you more information about how it's seeing the data stream you're sending.

It's very easy to get the Stream-name incorrect, and not have it pickup the stream.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #37 on: March 23, 2015, 12:42:48 pm »

Quote
3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY. 

Rev 1.  That script is fairly simple, and only pushes data-changes.  The openHAB one pushes on change, but also every 2 minutes.  This can help with some graphing tools, as some don't handle data streams of vastly differing period.

The script itself is agnostic to what's generating the data, and on what frequency.  Always room for tuning..  ;-)

Guessed

I wish I was not such an idiot and knew how to do basic scripting  ???
Would providing the line(s) to tell Vera to poll and send current reading to SEG every X seconds/min, be a modest task to request from you?  It would be a huge help for me/us  and I/we would be extremely appreciative!!

This would greatly add to the functionality.  I had primarily looked at RAW data at SEG.  When I look at 30 min data, I see there are huge gaps in the streams when devices are off, asleep or constant.  In my HA energy conservation endeavors, I hope to maximize those periods.  I didn't realize this until mentioned above.

Fingers crossed...

Thanks




Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #38 on: March 23, 2015, 12:55:22 pm »
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #39 on: March 23, 2015, 01:08:44 pm »
Guessed, that's too bad, but I totally understand.  Are you willing to give the next guy a brief description of the conceptual steps required for that code update?  I would imagine it would be more feasible for someone to try to pick up where you left if they had a simple roadmap (or at least some clues), as you likely have the dual knowledge of SEG and Vera better than anyone else. 

Thanks you very much for writing your existing script!!  It is the reason I bought a Vera (as plug reporting to SEG is not possible directly from an ISY). 
« Last Edit: March 23, 2015, 01:10:38 pm by ccclapp »

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #40 on: March 23, 2015, 01:29:17 pm »
For someone that knows Lua, it's not that complex.

It's basically:

a) Add a new, parameterless, function that can be used with luup.call_timer()
Put a loop in there that goes over all the values in the Array, and builds the complete "message" to send to SEG.  This will be similar to the existing one, except it'll build the entire list, not just the piece that changed.
This will use a for loop

The guts of this function will be of the form (syntax & output not checked):
Code: [Select]
    local segData = ""
    for k, v in pairs(SEG_DEVICES) do
        segData = segData .. string.format("(node %s ? (%s %s))", v.node, v.stream, luup.variable_get(v.deviceId, v.serviceId, v.serviceVar))
    end

    segData = string.format("(site %s %s)", SEG_SITE, segData)
    result, status = http.request(SEG_URL, segData)
b) Set  this new function to run under luup.call_timer() every "nn" seconds.

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #41 on: March 23, 2015, 01:33:18 pm »
Thanks Guessed.  Hopefully there is an Heir Apparent in the wings...

Offline scyto

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #42 on: March 23, 2015, 04:32:37 pm »
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.

Understand, I am trying to convince Ben at Brultech to get their DashBox to pull the data from ISY over IP REST interface - not sure I am getting far there...

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #43 on: March 23, 2015, 04:36:54 pm »
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.

Understand, I am trying to convince Ben at Brultech to get their DashBox to pull the data from ISY over IP REST interface - not sure I am getting far there...

I have a Dash on the way.  As I understand it, with an ISY that will send to SEG fine.  The missing link has always been for plugs (or low cost HON eg Aeon).  I've got a lot of high users on 1-2 circuits, thus plug monitoring/reporting is critical, (at least for me and someone else without a GEM/Dash).

Offline scyto

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #44 on: March 24, 2015, 06:51:38 pm »
The dash will send the data collected by a GEM or ECM to SEG.
It can also send the data to the ISY.

I am not ware of anything that sends from ISY to SEG but ben just wrote a proto-script in his forum he things could do it fi there is data in ISY.

alex