We have moved at community.getvera.com

Author Topic: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)  (Read 13964 times)

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
I just moved it over to SEG and disabled the Vera side of things.  Both my GEM and ECM1240 are running through a RPi with btmon, which in turn is shuffling the data over to SEG.  Only took a few hours to do all of it, including a significant GEM SW upgrade.

If I want the data in Vera, I'll write a SEG hookup to bring it back, or talk to RRD on the RPi.  In the yrs that I've run this stuff, I've only ever done analysis/tuning, not automation/alerting, and SEG is night and day better for analytics.

It's similar in concept to running a DashBox, just more command line steps to get I setup.

EDIT: Add hyperlinks
EDIT: Originally part of the "Adventures in reducing baseline power ..." thread, but broken out for clarity.
« Last Edit: March 11, 2015, 12:29:09 pm by guessed »

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #1 on: May 04, 2014, 02:59:55 am »
I just moved it over to SEG and disabled the Vera side of things.  Both my GEM and ECM1240 are running through a RPi with btmon, which in turn is shuffling the data over to SEG. 

Excuse the ignorance... what is SEG?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #2 on: May 04, 2014, 10:07:28 am »
Excuse the ignorance... what is SEG?
Oops, sorry about that.  I'd linked it in the other thread but not here. 

SEG is Smart Energy Groups.  There are a number of energy portals, and this one is one of the more flexible ones.

You register for a free account, get a site key, and start pushing RESTful stream of data to them.  This contains data for streams of information for one or more devices.  information in units like kWh, W, Temp, Volts, Humidity, CO2, Tank Levels, Pulse Counts (etc) all in one or more named data feeds that are using your key.

On the receiving end, you just tell SEG to discover and it tells you what's being poked at it, and enters the relevant device-stream names into it's catalog (that can then be further augmented, via UI or API, with the units information)

At that point, it starts collecting the data, and automatically doing rollups for various periods.  It'll provide a set of per stream graphs by default, with "real-time", day, month yr type graphs, but then you can totally customize the portal with a set of widgets on configurable dashboards that you create.

All of this is done via the UI, but there's a backing API for doing it programmatically as well (as well as getting data both in and out of the service)


And "btmon.py" is a bridge program that takes feeds in from the Brultech devices, and puts them into one or more configurable targets, either locally (RRD, SQLLite, MySQL) or remotely (SEG, PlotWatt, etc)

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #3 on: May 04, 2014, 10:35:54 am »
Very cool.  Thanks for the detailed explanation.
Something else to investigate!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #4 on: May 04, 2014, 10:40:38 am »
In fact, excellent

No need for any Vera power-related app, then.  I'll just throw all power data from DataYours to SEG and cross that one off the list.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #5 on: May 12, 2014, 09:04:40 pm »
For those interested in pushing data to SEG for analysis, here's a snippet of Lua that can be used in your Vera Startup.

It's not a plugin, and it's not resilient (if SEG goes down, the data isn't accumulated at Vera) but it works reasonably well (I use it for supplementary data like Room & Outdoor temperatures)

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.


Code: [Select]
local http = require('socket.http')
http.TIMEOUT = 5

local SEG_SITE = "<yourSEGSiteIdHere>"
local SEG_URL = "http://api.smartenergygroups.com/api_sites/stream"
local segLog = function (text) luup.log('SEG Logger: ' .. (text or "empty")) end

SEG_DEVICES = {
    {node='vera', stream='t_upstairs',   deviceId=335, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_downstairs', deviceId=384, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_outside',    deviceId=318, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"}
  }

local function findStream(deviceId, serviceId, serviceVar)
  -- TODO: Change from a simple scan to something that'll support more entries efficiently.
  for k, v in pairs(SEG_DEVICES) do
    if (v.deviceId == deviceId and
        v.serviceId == serviceId and
        v.serviceVar == serviceVar) then
      return v.node, v.stream
    end
  end

  return nil, nil
end

local function initWatch()
  for k, v in pairs(SEG_DEVICES) do
    luup.variable_watch('segWatch', v.serviceId, v.serviceVar, v.deviceId)
  end
end

function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
  local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)

  if (nodeName == nil or streamName == nil) then
    segLog(string.format("Node not found for %s,%s,%s", deviceId, serviceId, serviceVar))
    return
  end

  -- TODO: Encoding
  segData = string.format("(site %s (node %s ? (%s %s)))", SEG_SITE, nodeName, streamName, newValue)
  segLog(segData)

  result, status = http.request(SEG_URL, segData)
end

initWatch()


You need only substitute in your SEG Site Id, and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track, and the script will watch them for changes and push the values across to SEG.

In my case, I have a SEG Device registered with the label "vera", and beneath that I have SEG Streams for "t_upstairs", "t_downstairs" and "t_outside".   In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

Since the data is only pushed when a value changes, you may have to wait a little while before you see the data in SEG (after loading the code into Vera's Startup method, and restarting Vera)


NOTE: When naming your streams for SEG, use prefixes of
  • t_ - temperature data
  • e_ - energy data (Wh, kWh, etc)
  • p_ - power data (W, kW, etc)

When using SEG's "Discoveries" feature, it automatically decodes the Stream names, and applies the correct typing information if the streams are named using these prefixes.

This tool is on http://smartenergygroups.com/tools
    MyThings -> tools -> Discoveries

When using the Discovery feature, you may need to leave SEG on that page for some time, as I'm only sending data over as it changes.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #6 on: June 03, 2014, 02:44:03 pm »
Quote
And here's what that looks like

Ooh, that's Google Charts, but what produced it? SEG??
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #7 on: June 03, 2014, 02:46:27 pm »
Quote
And here's what that looks like

Ooh, that's Google Charts, but what produced it? SEG??
Straight out of SEG, they have an option/menu they display on each Graph to Save it.  Formats include PNG, JPEG, PDF and SVG.

Saves taking screenshots ;)

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #8 on: June 03, 2014, 02:50:46 pm »
oh, and I enabled @watou's option on the Nest T-Stats to get finer grained data.  It was a bit chunky before I did that...

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #9 on: June 17, 2014, 09:40:35 pm »
HI Guessed. Great thread ...thanks for sharing and introducing me to SmartEnergyGroups!
SEG is great stuff, and run by a lad in Sydney.  Always happy to promote enterprising companies from home ;)

I find myself using it for a variety of measurement/graphing/comparison stuff now, esp given how simply it ingests data.

Quote
There is another way you could improve your fuel economy, perhaps without changing your driving style ;-)
http://www.discovery.com/tv-shows/mythbusters/videos/dimpled-car-minimyth.htm
That's hilarious!  I was laughing a ton as I watched, so thankyou!

Those guys are nuts.  Every time I see them I'm reminded of this little "mishap" and wish there was video of it  8)
    http://www.wired.com/2011/12/mythbusters-cannonball-map/


On a related note, just noticed that electricity rates have climbed to 33-39c/kWh back home, which is nearly as bad as Hawaii!  No wonder Solar PV installs went nuts down there...

Offline fullmoonguru

  • Full Member
  • ***
  • Posts: 207
  • Karma: +0/-3
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #10 on: July 11, 2014, 08:02:20 am »
Guessed, I am trying to use this script, but I'm pretty weak in this area so I could use some clarification.

Quote
You need only substitute in your SEG Site Id

So first I create a device on SEG.  That gives me a token number and a stream name which I then use as the site ID and teh stream name in the script?  And is that site ID with the < > brackets, or without?  I am assuming without.

Quote
and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track
Quote
In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

So in the advanced settings tab for my energy monitor I have the deviceId under id, the ServiceId info is under device_type, and the State Variable would be Watts.  I suppose I would set another stream up for KWH.

But my serviceId looks like this:

Code: [Select]
urn:schemas-micasaverde-com:device:PowerMeter:1
I have a script pushing this data to Plotwatt with this serviceId:

Code: [Select]
urn:micasaverde-com:serviceId:EnergyMetering1

I tried that too but nothing working so far.  I'm not getting an error from the script, just nothing on SEG.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #11 on: July 11, 2014, 10:43:27 am »
For SEG, the Site Id looks like the attached image.  Once logged in, it's under:

   My Things -> Sites -> <Site Image> -> [Edit]

The string doesn't have <> characters around it, use it as displayed when you select the pane above.



This is the serviceId:
     urn:micasaverde-com:serviceId:EnergyMetering1

The other string you had was the device type, which can't be used.  If you want to validate, just "hover" the mouse over the Service Variable name/label in the Device's Advanced tab.  It'll tooltip the ServiceID for the Service Variable.



Once you set the Vera-end of this "running" with the SEG Site ID, you can goto SEG:

    My Things -> Tools -> Discoveries

and then press (Discover).  Give it a few minutes and it'll create ALL of the relevant Devices for you, which you can then just go and fill in the blanks (like UoM, range, etc)

Offline fullmoonguru

  • Full Member
  • ***
  • Posts: 207
  • Karma: +0/-3
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #12 on: July 15, 2014, 11:41:27 pm »
The proper site ID did it.  Thanks!

Offline helraiser

  • Jr. Member
  • **
  • Posts: 93
  • Karma: +2/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #13 on: September 16, 2014, 11:18:15 am »
Yup, same here. site-id is actually the token key that's generated. The info here made it really easy to set up and get data into SEG. Now to figure out how to forecast my monthly bills.

Thanks for the 'fo!

Offline ccclapp

  • Sr. Newbie
  • *
  • Posts: 44
  • Karma: +1/-0
Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
« Reply #14 on: December 07, 2014, 04:57:20 pm »

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.


Hi Guessed and others

I am adding a veralite to my ISY HA setup, in part because of my frustration in trying to get energy data from ISY to SEG or any other good energy monitoring site.  In my case I want track circuits AND also individual smart plugs on individual devices.

  • Overall are you happy with what you describe above to report to SEG for both GEM and other Vera energy reporting, e.g. smart plugs?
  • Is SEG "smart enough" to allow the user to avoid double counting the smart plug loads which are also included in the whole-house-meter loads?
  • I am not a scripter (but can follow along and fill in basic blanks, etc.)  Do you expect this is an appropriate solution or will I be jumping off a clifF?

 Thanks for posting this info and your work to make it straightforward for others!!
« Last Edit: December 07, 2014, 05:00:14 pm by ccclapp »