The Vera Community forums have moved!

General => Power Management => Topic started by: guessed on May 04, 2014, 12:57:05 am

Title: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on May 04, 2014, 12:57:05 am
I just moved it over to SEG (http://smartenergygroups.com/) and disabled the Vera side of things.  Both my GEM and ECM1240 are running through a RPi with btmon (http://lancet.mit.edu/mwall/projects/power/btmon-howto.html), which in turn is shuffling the data over to SEG.  Only took a few hours to do all of it, including a significant GEM SW upgrade.

If I want the data in Vera, I'll write a SEG hookup to bring it back, or talk to RRD on the RPi.  In the yrs that I've run this stuff, I've only ever done analysis/tuning, not automation/alerting, and SEG is night and day better for analytics.

It's similar in concept to running a DashBox, just more command line steps to get I setup.

EDIT: Add hyperlinks
EDIT: Originally part of the "Adventures in reducing baseline power ... (http://forum.micasaverde.com/index.php/topic,9998.0.html)" thread, but broken out for clarity.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: akbooer on May 04, 2014, 02:59:55 am
I just moved it over to SEG and disabled the Vera side of things.  Both my GEM and ECM1240 are running through a RPi with btmon, which in turn is shuffling the data over to SEG. 

Excuse the ignorance... what is SEG?
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on May 04, 2014, 10:07:28 am
Excuse the ignorance... what is SEG?
Oops, sorry about that.  I'd linked it in the other thread but not here. 

SEG is Smart Energy Groups (http://smartenergygroups.com/).  There are a number of energy portals (http://lancet.mit.edu/mwall/projects/power/service-review-2013/), and this one is one of the more flexible ones.

You register for a free account, get a site key, and start pushing RESTful stream of data to them.  This contains data for streams of information for one or more devices.  information in units like kWh, W, Temp, Volts, Humidity, CO2, Tank Levels, Pulse Counts (etc) all in one or more named data feeds that are using your key.

On the receiving end, you just tell SEG to discover and it tells you what's being poked at it, and enters the relevant device-stream names into it's catalog (that can then be further augmented, via UI or API, with the units information)

At that point, it starts collecting the data, and automatically doing rollups for various periods.  It'll provide a set of per stream graphs by default, with "real-time", day, month yr type graphs, but then you can totally customize the portal with a set of widgets on configurable dashboards that you create.

All of this is done via the UI, but there's a backing API for doing it programmatically as well (as well as getting data both in and out of the service)


And "btmon.py" is a bridge program that takes feeds in from the Brultech devices, and puts them into one or more configurable targets, either locally (RRD, SQLLite, MySQL) or remotely (SEG, PlotWatt, etc)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: akbooer on May 04, 2014, 10:35:54 am
Very cool.  Thanks for the detailed explanation.
Something else to investigate!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: akbooer on May 04, 2014, 10:40:38 am
In fact, excellent

No need for any Vera power-related app, then.  I'll just throw all power data from DataYours to SEG and cross that one off the list.

Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on May 12, 2014, 09:04:40 pm
For those interested in pushing data to SEG for analysis, here's a snippet of Lua that can be used in your Vera Startup.

It's not a plugin, and it's not resilient (if SEG goes down, the data isn't accumulated at Vera) but it works reasonably well (I use it for supplementary data like Room & Outdoor temperatures)

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.


Code: [Select]
local http = require('socket.http')
http.TIMEOUT = 5

local SEG_SITE = "<yourSEGSiteIdHere>"
local SEG_URL = "http://api.smartenergygroups.com/api_sites/stream"
local segLog = function (text) luup.log('SEG Logger: ' .. (text or "empty")) end

SEG_DEVICES = {
    {node='vera', stream='t_upstairs',   deviceId=335, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_downstairs', deviceId=384, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"},
    {node='vera', stream='t_outside',    deviceId=318, serviceId='urn:upnp-org:serviceId:TemperatureSensor1', serviceVar="CurrentTemperature"}
  }

local function findStream(deviceId, serviceId, serviceVar)
  -- TODO: Change from a simple scan to something that'll support more entries efficiently.
  for k, v in pairs(SEG_DEVICES) do
    if (v.deviceId == deviceId and
        v.serviceId == serviceId and
        v.serviceVar == serviceVar) then
      return v.node, v.stream
    end
  end

  return nil, nil
end

local function initWatch()
  for k, v in pairs(SEG_DEVICES) do
    luup.variable_watch('segWatch', v.serviceId, v.serviceVar, v.deviceId)
  end
end

function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
  local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)

  if (nodeName == nil or streamName == nil) then
    segLog(string.format("Node not found for %s,%s,%s", deviceId, serviceId, serviceVar))
    return
  end

  -- TODO: Encoding
  segData = string.format("(site %s (node %s ? (%s %s)))", SEG_SITE, nodeName, streamName, newValue)
  segLog(segData)

  result, status = http.request(SEG_URL, segData)
end

initWatch()


You need only substitute in your SEG Site Id, and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track, and the script will watch them for changes and push the values across to SEG.

In my case, I have a SEG Device registered with the label "vera", and beneath that I have SEG Streams for "t_upstairs", "t_downstairs" and "t_outside".   In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

Since the data is only pushed when a value changes, you may have to wait a little while before you see the data in SEG (after loading the code into Vera's Startup method, and restarting Vera)


NOTE: When naming your streams for SEG, use prefixes of

When using SEG's "Discoveries" feature, it automatically decodes the Stream names, and applies the correct typing information if the streams are named using these prefixes.

This tool is on http://smartenergygroups.com/tools
    MyThings -> tools -> Discoveries

When using the Discovery feature, you may need to leave SEG on that page for some time, as I'm only sending data over as it changes.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: akbooer on June 03, 2014, 02:44:03 pm
Quote
And here's what that looks like

Ooh, that's Google Charts, but what produced it? SEG??
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on June 03, 2014, 02:46:27 pm
Quote
And here's what that looks like

Ooh, that's Google Charts, but what produced it? SEG??
Straight out of SEG, they have an option/menu they display on each Graph to Save it.  Formats include PNG, JPEG, PDF and SVG.

Saves taking screenshots ;)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on June 03, 2014, 02:50:46 pm
oh, and I enabled @watou's option on the Nest T-Stats (http://forum.micasaverde.com/index.php/topic,10508.msg177844.html#msg177844) to get finer grained data.  It was a bit chunky before I did that...
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on June 17, 2014, 09:40:35 pm
HI Guessed. Great thread ...thanks for sharing and introducing me to SmartEnergyGroups!
SEG is great stuff, and run by a lad in Sydney.  Always happy to promote enterprising companies from home ;)

I find myself using it for a variety of measurement/graphing/comparison stuff now, esp given how simply it ingests data.

Quote
There is another way you could improve your fuel economy, perhaps without changing your driving style ;-)
http://www.discovery.com/tv-shows/mythbusters/videos/dimpled-car-minimyth.htm
That's hilarious!  I was laughing a ton as I watched, so thankyou!

Those guys are nuts.  Every time I see them I'm reminded of this little "mishap" and wish there was video of it  8)
    http://www.wired.com/2011/12/mythbusters-cannonball-map/


On a related note, just noticed that electricity rates have climbed to 33-39c/kWh back home, which is nearly as bad as Hawaii!  No wonder Solar PV installs went nuts down there...
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: fullmoonguru on July 11, 2014, 08:02:20 am
Guessed, I am trying to use this script, but I'm pretty weak in this area so I could use some clarification.

Quote
You need only substitute in your SEG Site Id

So first I create a device on SEG.  That gives me a token number and a stream name which I then use as the site ID and teh stream name in the script?  And is that site ID with the < > brackets, or without?  I am assuming without.

Quote
and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track
Quote
In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

So in the advanced settings tab for my energy monitor I have the deviceId under id, the ServiceId info is under device_type, and the State Variable would be Watts.  I suppose I would set another stream up for KWH.

But my serviceId looks like this:

Code: [Select]
urn:schemas-micasaverde-com:device:PowerMeter:1
I have a script pushing this data to Plotwatt with this serviceId:

Code: [Select]
urn:micasaverde-com:serviceId:EnergyMetering1

I tried that too but nothing working so far.  I'm not getting an error from the script, just nothing on SEG.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on July 11, 2014, 10:43:27 am
For SEG, the Site Id looks like the attached image.  Once logged in, it's under:

   My Things -> Sites -> <Site Image> -> [Edit]

The string doesn't have <> characters around it, use it as displayed when you select the pane above.



This is the serviceId:
     urn:micasaverde-com:serviceId:EnergyMetering1

The other string you had was the device type, which can't be used.  If you want to validate, just "hover" the mouse over the Service Variable name/label in the Device's Advanced tab.  It'll tooltip the ServiceID for the Service Variable.



Once you set the Vera-end of this "running" with the SEG Site ID, you can goto SEG:

    My Things -> Tools -> Discoveries

and then press (Discover).  Give it a few minutes and it'll create ALL of the relevant Devices for you, which you can then just go and fill in the blanks (like UoM, range, etc)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: fullmoonguru on July 15, 2014, 11:41:27 pm
The proper site ID did it.  Thanks!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: helraiser on September 16, 2014, 11:18:15 am
Yup, same here. site-id is actually the token key that's generated. The info here made it really easy to set up and get data into SEG. Now to figure out how to forecast my monthly bills.

Thanks for the 'fo!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on December 07, 2014, 04:57:20 pm

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.


Hi Guessed and others

I am adding a veralite to my ISY HA setup, in part because of my frustration in trying to get energy data from ISY to SEG or any other good energy monitoring site.  In my case I want track circuits AND also individual smart plugs on individual devices.


 Thanks for posting this info and your work to make it straightforward for others!!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on December 08, 2014, 11:33:59 am
Overall are you happy with what you describe above to report to SEG for both GEM and other Vera energy reporting, e.g. smart plugs?

Yup, completely.  I look at the data in SEG daily after I make significant changes in the house.

I report data to SEG from 2 different sources:


I moved from using Vera as the conduit for the Energy streams in order to reduce the load on Vera (I have a lot of stuff on Vera that needed to be more real-time than the energy monitoring stuff)

Is SEG "smart enough" to allow the user to avoid double counting the smart plug loads which are also included in the whole-house-meter loads?

SEG has no knowledge of the relationship between the [data] Streams you register with it.  It sees Devices (think "Main Panel", "SubPanel", "Vera") and the Streams that come from those Devices (think "Upstairs Temperature", "Grid Energy", "Solar Energy", "Office Energy", "CPU Usage")

Once you've connected the Devices and established the Streams, you use the SEG UI to build Dashboards to display the UI for those Streams (you pick the UI/Graph styles, as well as what Streams to display in those graphs).  It's a fairly advanced graphical Portal builder for "energy" data. 

You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system.  I do this to "rollup" certain low level stats into more useful stuff.

eg. TotalEnvironmentalEnergy = MainPanel/AC Energy + SubPanel/Furnace Energy

So this is where, indirectly, relationships are being captured.... by how you build Graphs, and how you build these Computed Streams.  All of this is done in a UI, and is easy to do.

I am not a scripter (but can follow along and fill in basic blanks, etc.)  Do you expect this is an appropriate solution or will I be jumping off a clifF?

The hardest part of my setup is the bymon.py running on Pi.  Since you need to setup the Pi, btmon.py, deal with Serial Ports (for attachment to ECM-1240) and Wifi (for dealing with my GEM) and script it all to start on reboot (etc).

Not impossible, but it would take little searching to pull the parts together.


That said, if you've got a Vera, and it's not doing too much else, then you could just run the Brultech Plugin and it'll do the work of acquiring the data for you (esp if you've got the GEM on WiFi and/or Ethernet).  You'd then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG.  That's all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id's)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: fullmoonguru on December 08, 2014, 11:53:08 am
I find SEG really confusing.  It looks very flexible and powerful and I'm sure I could get it dialed in given enough time with it, but I wouldn't call it an intuitive UI.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on December 08, 2014, 12:16:33 pm

...You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system.  I do this to "rollup" certain low level stats into more useful stuff.

...That said, if you've got a Vera, and it's not doing too much else, then you could just run the Brultech Plugin and it'll do the work of acquiring the data for you (esp if you've got the GEM on WiFi and/or Ethernet).  You'd then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG.  That's all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id's)

Guessed, thanks! Those were the answers I was hoping to receive...

As to calculated streams on SEG:  Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".  In thinking about this, I suppose one could just to the upper tier rollup manually from each device, vs as a two-tier roleup.

As to Vera vs PI and complexity:  I have an ISY with Zwave module and find it to be VERY reliable/stable.  However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN).  Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data.  Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc.  I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-).  In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS:  exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Finally, you mention GEM wifi or Ethernet.  Are these options in the GEM one buys, or ad-ons/hacks to do after the fact.  I have not purchased yet.  I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

Thanks again!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on December 08, 2014, 12:53:56 pm
As to calculated streams on SEG:  Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Both Add and Subtract, of any Stream with the same Units of measure.  In my case, I have some streams that are "rolled up" at the source, based upon where I've clamped...  sometimes I re-roll them up from the component streams, in SEG, since there might be errors in measurement that I'm interested in seeing.

I'll post a series of screen-captures shortly to illustrate these bits...

Quote
Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".

Yep, Calculations are just another Stream, so you can composite new Calculations based upon others.  Sometimes this limits the granularity of the result (to 30 minutes) but in general that's not an issue.

Quote
As to Vera vs PI and complexity:  I have an ISY with Zwave module and find it to be VERY reliable/stable.  However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN).  Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data.  Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc.  I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-).  In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS:  exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Take a look at the stuff that Teken is doing on the Brultech forums.  He has ISY gear.   My Vera is a Vera3, so it has more memory than a VeraLite, but the same as the newer VeraEdge. 

When running a reasonable load, Vera is good at Lighting/Z-Wave control (etc).... but when it runs out of steam, you're hosed.  They don't currently have a "bigger" (and, in all honesty, less-buggy) option for power-users so I paired back my Vera3 to running the critical stuff, and have moved everything else to Pi's (and now, via openHAB, I'm expanding/migrating to my Mac)

Quote
Finally, you mention GEM wifi or Ethernet.  Are these options in the GEM one buys, or ad-ons/hacks to do after the fact.  I have not purchased yet.

You buy them as part of the GEM when you order it.  Mine is a 1st Gen GEM, which I bought with ZigBee, since I already had ECM-1240 units with the ZigBee module (and the related ZB <-> Ethernet Bridge).  At the time the ZB Modules were socketed, so I replaced mine with a RovingNetworks RN-XV module (now part of MicroChip).

In the latest models, these are simply options that you buy with the GEM.  I like WiFi here, since I don't have cabling for Ethernet anywhere near my GEM (and I don't want to either, since my GEM is right next to my SubPanel)

In my setup, I have a WiFi rPi which is USB-Serial attached to a ECM-1240, reading the Main-Panel.  This is on one side of the garage and it's using WiFi to get access to the GEM, which is next to the Sub-Panel on the other side of the Garage.  The rPi aggregates the data and phones it up to SEG (as well as PVOutput, for the Solar bits)

Quote
I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

You mean the DashBox.  It still appears to be somewhat under development.  It's performing the role of what I'm using a Pi + SEG for, albeit using "locally-stored" data, which will be of interest to some folks.   I may do this sometime, to replace the Pi and make my solution more  consumer-friendly... handy if you ever want to sell you house with this stuff as "features"  ;)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on December 08, 2014, 12:56:42 pm
A few more SEG images, showing the default Stream-list, a default graph for one of the Streams, as well as custom dashboard tabs I've built for Energy Overview/Detail and Temperature monitoring.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on December 08, 2014, 01:12:54 pm
guessed, very cool and super helpful.  Thanks very much!!

PS:  It was after reading Taken's "book" on ISY / GEM integration that prompted me to look at Vera for energy reporting ;-)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on December 18, 2014, 07:08:05 pm
Hi again

I apologize in advance for my inexperience.  I turned on my Vera today for the first time, added the Nest plugin, got the Sonos plug-in running, got the ISY manual loading plugin running (needed to learn about WinSCP for that) and am now trying to climb the hill described in this thread...

In Vera I added a smart plug and Nest T-stats (via the nest plugin) and have created a "Site" and a couple of "Devices" at SEG and see how to get the token numbers.  BUT: I am not sure how/where to write the script guessed provided on the prior page.  I'm not a programmer and don't know Vera, so my questions are VERY basic...


THANKS YOU VERY MUCH AND SORRY TO BE SUCH A NEWBIE
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on December 19, 2014, 02:10:18 pm
@ccclapp,

Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on December 19, 2014, 02:17:38 pm
...wonderful.  Thanks guessed!!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on February 18, 2015, 02:54:49 pm
do you guys know it the script should work in UI7 on vera edge?  i copied/pasted it in, edited per instructions , but don't see the connection at SEG.   possibly i made an error and will post screenshots, but first figured I'd inquire if it should work with UI7

thanks!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on February 18, 2015, 03:05:43 pm
do you guys know it the script should work in UI7 on vera edge?  i copied/pasted it in, edited per instructions , but don't see the connection at SEG.   possibly i made an error and will post screenshots, but first figured I'd inquire if it should work with UI7

It's a fairly simple script, so there's no reason it shouldn't work on all Lua-enabled MiOS engines.  That said, I've not tested this specific item on UI7, so it can always be a dice-roll.

Double check what you C&P'd, as well as any errors reported in /var/log/cmh/LuaUPnP.log since it'll tend to show up there if there's a typo.

Also, when looking at SEG, there's a "Trace" mode in their User-toolbox that can be used to see _what_ is being pushed to SEG.  The most common problem, if you're not seeing data there, is using an incorrect token in the push.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on February 20, 2015, 01:34:06 pm
Hi Guessed

Two things:

#1 I want you to know I linked to your script on the ISY forums in hopes it would help us figure out how to transmit from ISY to SEG.  I hope that is ok with you.  Here is the post:

http://forum.universal-devices.com/topic/12682-energy-monitoring-circuits-plugs-interface/?p=132278

#2 My Vera was having issues the other day.  That is why I did not connect to SEG.  Your script works fine in UI7.  I will separately post a few observations/questions.  Once I better understand those, I will try to write another post in this thread spelling out a newbie step-by-step for this from the info you and others have provided above (to help consolidate the info in these 6 pages for other beginners, like myself)...

Thanks!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on February 20, 2015, 02:21:41 pm
Yeah, it's public content/posting here, so feel free to point to it.

Feel free also to create a break-out thread that just handles SEG, as needed.  I buried it here because it's the thread I've been using for all updates on my energy monitoring, and curbing energy usage, discussions.

As long as it remains a free-to-use option, with open source-code disclosure, I'm ok with others using it anyway they want ;)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on February 20, 2015, 02:25:03 pm

Feel free also to create a break-out thread that just handles SEG, as needed. 

Either is fine with me.  Would you prefer the SEG linking info be in a separate thread.  If you don't, I'd stay here.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on February 20, 2015, 03:01:55 pm
Probably easier for users to find if it's a new Thread.  Feel free to start a new one, and you can just add a link back to here for reference (and link a post here, to the new thread also, so people can stumble across either)

That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: akbooer on February 20, 2015, 03:40:36 pm
That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)

That sounds perfect... I'm banging my head against a brick wall on this, but will withhold comments until the new thread is started.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on February 20, 2015, 04:12:11 pm
That way, folks just interested in SEG for Data Capture, they'll have an easier experience ;)

That sounds perfect... I'm banging my head against a brick wall on this, but will withhold comments until the new thread is started.

I was too but then got it. Once I did I saw that Guessed's instructions were perfect. I had to pick from a few posts to pull it together and again once I did I saw the simplicity of what he created. I'm on my way up to the ski mountains at the moment so it might be Sunday or beginning of the weekfor a new thread. In the meantime, I'm sure your basic questions would be answered here
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on February 27, 2015, 07:28:30 am

I'm on my way up to the ski mountains at the moment so it might be Sunday or beginning of the weekfor a new thread. In the meantime, I'm sure your basic questions would be answered here

sorry I was delayed in doing this this week. I will try to work on it today or this weekend?
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on March 11, 2015, 04:08:09 pm
I forked it out this morning.  Feel free to use extend the discussion here.

As a side-note, I also made an openHAB version of the script (https://groups.google.com/forum/#!category-topic/openhab/rules/GPBI8AHFedQ), for those that are looking at that route.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 20, 2015, 12:47:42 am
Vera SEG Step-by-Step Guide

The following is an (overly detailed) beginners step-by-step guide for linking Vera devices to SEG.  100% of this comes from @Guessed?s script and his instructions in this thread.  I am doing this because as a beginner it took me a while to understand what he succinctly wrote.  In case others are like me, this may be helpful:

INTRODUCTION: 

Purpose: Monitor energy use and other stream data over time on a great, flexible and easy to use website accessible from all devices.  SEG (Smart Energy Groups) is such a site.  Thanks to @Guesseds script we can easily send all stream data from Vera to SEG.  Stream date includes things like (i) energy data from smart plugs, home circuit panel or individual circuits, (ii) environmental conditions like indoor/outdoor temp, wind, humidity (iii) pretty much anything else that changes over time. 

Examples:

Below are examples of the energy monitoring products.

http://www.amazon.com/Aeon-Labs-DSC06106-ZWUS-Z-Wave-Energy/dp/B007UZH7B8/ref=sr_1_1?ie=UTF8&qid=1426806219&sr=8-1&keywords=zwave+smart+plug
http://www.amazon.com/Aeon-Labs-Aeotec-Z-Wave-DSC25-ZWUS/dp/B00H5U40L8/ref=sr_1_2?ie=UTF8&qid=1426806219&sr=8-2&keywords=zwave+smart+plug
http://www.amazon.com/Aeon-Labs-Aeotec-Z-Wave-DSB28-ZWUS/dp/B00FKJBUX2/ref=sr_1_2?ie=UTF8&qid=1426806166&sr=8-2&keywords=zwave+energy+monitor
http://brultech.com/store/

In my case I have about 10 smart plugs connected to AV and PC equipment, sump pump and other high energy users I want to monitor at the plug level.  I also have the Aeon HEM on my panel and have a GEM on order to grab all circuits.  I also pick up indoor temperature and humidity from my NEST t-stats and outdoor temp, wind, etc. from the Underground Weather plug-in.

Context:  Going into this I was using 1,600 KWH/month 20,000 KWH/year and DO NOT have electric heat or use AC.  Thats a huge #!! By using a kill-a-watt meter and smart plugs to map every device?s on/standby electric draw and then Vera (and an ISY) to control everything  I have cut this almost in half over the past month and have more to do.

INSTRUCTIONS:

On Vera

1)  Copy into a text file to edit @Guesseds script in post #5, page 1 of this thread, http://forum.micasaverde.com/index.php/topic,31212.msg175678.html#msg175678

In your pasted text file, edit only this highlighted portion of the text, as follows:

SEE ATTACHED IMAGE FOR EXACT LINES TO EDIT

 
2)  Where is says <yourSEGSiteIdHere>", remove the < and > symbols and put your own SEG Site ID token between the quotation marks.  (In the SEG instructions below I?ll describe how to set up SEG and get that token, or if you want see/do SEG step #1, below and then continue with the Vera Steps.)

3) The list after the words SEG_DEVICES = { are to be edited as follows:

- The concept is you will have one line for each Vera data stream you want to send to SEG.  The example in the script includes three streams (3 lines).  Note: a single Vera device could have more than one stream and thus more than one lines, e.g. Nest has temp and humidity.  Smart plugs have WATTS and KWH. You decide what to report and make a line for each.  Each line will create a stream item at SEG (we?ll get to SEG in a moment).

NOTE:  That you are about to do is fully editable in Vera.  Therefore you may want to start with 1 or 2 lines (for 1 or 2 streams), as described below and then add additional lines once it is working for you.

3(a) Using the 1st line of the script SEG Devices table as the example, where the text says: stream='t_upstairs', you replace the text inside the single quotes with whatever you want to SEG to identify that stream by.  This does not have to be the name of the stream on SEG, you can change the SEG stream name, but not the SEG stream ID.  You are creating the stream ID here.  I don?t know if SEG likes spaces, caps, etc, so dont use them.

- For energy and temp streams (and maybe others, but I dont know) a prefix in the stream ID tells SEG to use WATTS, KWH or Degrees.  You can edit that at SEG so don?t need to use prefix.  Prefixes are e_ for energy (Watts), p_ for power (Watt Hours) and t_ for temp (Degrees).

3(b) Where the text says deviceId=335 you put the Vera device #.  THIS IS NOT the ZWAVE NODE #.  In UI7 the Device # is seen at the top of the Advanced tab on the device (just below the green header)

3(c) I am skipping to the end of the text line where it says serviceVar="CurrentTemperature"}, (Ill come back to the middle part below).  You will replace the text between the double quotes as follows:  On Vera go to the Advanced tab for the device.  Go to the Variables column.  Find the variable you want to stream, e.g. WATTS, KWH, CurrentTemperature, CurrentLevel, WindSpeed, or whatever variable you want that Vera lists for that device.  Copy/paste the variable field label name.

3(d) Now moving backwards in the line from the script text, where it says serviceId='urn:upnp-org:serviceId:TemperatureSensor1' replace the text between the single quotes as follows:  On Vera hover your mouse over the service variable label you used in step 4, above.  When you hover you will see the service ID text to use.   

Examples are: 
urn:upnp-org:serviceId:TemperatureSensor1,
urn:upnp-micasaverde-com:HumiditySensor1,
urn:micasaverde-com:serviceId:Weather1.

NOTE on steps 3(c) & 3(d):  I have NOT yet properly configured my Vera/SEG for the Weather Underground Plugin streams and possibly not all service variables and corresponding service IDs are supported by the script.  More likely I made a typo...TBD

Here is a link to the Vera Wiki discussing Service Variables and Service ID:  http://wiki.micasaverde.com/index.php/Luup_Variables#Device_category_3

3(e) On Vera, the final step is to paste your edited script into the Startup Lua, as follows:  (i) copy the text file you just edited.  On Vera navigate to Apps ->Develop Apps -> Edit Startup Lua.  Paste the text into the box. Click Go.  Enter ctl-F5 to restart the Vera engine, now incorporating this script.  You are now done on the Vera side.

On SEG

1) Open a free account here: https://smartenergygroups.com/ . Then click My Things -> My Sites and create a site.  At the bottom of the new site info box you it says API Details and below that Site Token.  The number below is what you want.   This is the number you put in Vera Step #2 above where the script says says ?<yourSEGSiteIdHere>", (As stated above remove the < and > symbols and put your own SEG Site Token between the quotation marks).

If you previously created a SEG Site find the token by clicking on your site thumbnail image and then clicking the pencil and paper icon (the Edit icon) Under API Detail is your Site Token. (Be sure you are getting the Site Token, not a Device or Stream token).

2)  On SEG create a new Device.  Name it anything, but be sure to make the Node Name vera (or you can change the node name, but also edit the Vera script changing {node='vera',   to the name you use on SEG).  Make Device Type be Display.

3) Now you are ready for SEG to receive the streams from Vera.  To do this on SEG click Tools on the right of every screen, then Discoveries -> Discover.  Wait a few minutes and let it run.  It is looking for the stream data Vera will send.

NOTE:  Per @Guessed, Vera only sends data when it changes (or SEG only logs it when it changes).  Either way, I recommend on Vera select Poll Now for the device(s) you are sending to Seg.  You may then want to change the data on Vera, e.g. turn smart switch on/off and Poll Now each time to have changing data for SEG.    I don?t know if you need to keep Discover running on SEG until it picks up all your streams, or just the 1st.  In any event you can repeat Discover.  To keep Discover running I log into SEG on another browser tab and click on my vera Device in SEG to see if any streams have been created (while Discover continues in the other browser tab).

4) Once 1 or more streams show up in SEG, you can click see more detail and time periods and also to edit them.  Mine usually default to Kilo eg KW and KWH (vs W and WH).  To get to W WH edit the stream and select None if Kilo is selected.  Also, on the edit screen set the Type of Data to correspond to the Vera Service Variable you picked, e.g. WATTS. 

Just remember, both SEG and the Vera script can be changed if things aren't perfect.

On SEG there are lots of things you can do with the data as @Guessed describes and shows in screenshots earlier in this thread.

That's all for now

Have fun!

Thanks @Guessed!!!!!!!!!!!!!!!!!!!!!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: scyto on March 22, 2015, 11:25:38 pm
1) Why did you choose display and not energy meter on the SEG site.

2) I would recommend that for energy devices on SEG one defines both the energy and power for a device as such my Aeon SmartStrip looks like this
Code: [Select]
SEG_DEVICES = {
    {node='vera', stream='p_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"}
   }

3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY.  On the Vera it rounds up to 8 in the UI - does this mean the script wont send anything because it didn't detect a change (PS turning off the strip made no difference either).

At this point i have the 2 streams appearing in SEG, but no data. It also shows only as being online in SEG 15 minutes ago...
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on March 23, 2015, 12:24:44 am
1) Why did you choose display and not energy meter on the SEG site.

I moved to pushing Energy data, directly, using BTMon.py to improve Vera stability.  It's very reliable, so I only needed to push infrequently changing aux-data from Vera. 

I've recently stopped pushing any data from Vera, and now push all of my SEG data directly from openHAB.

Quote
2) I would recommend that for energy devices on SEG one defines both the energy and power for a device as such my Aeon SmartStrip looks like this
Code: [Select]
SEG_DEVICES = {
    {node='vera', stream='p_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_AeonSmartStrip', deviceId=115, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"}
   }

Yup, different (but related) quantities.  Both are quite useful, depending upon what you're trying to analyze/fix.

Quote
3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY. 

Rev 1.  That script is fairly simple, and only pushes data-changes.  The openHAB one pushes on change, but also every 2 minutes.  This can help with some graphing tools, as some don't handle data streams of vastly differing period.

The script itself is agnostic to what's generating the data, and on what frequency.  Always room for tuning..  ;-)

Quote
On the Vera it rounds up to 8 in the UI - does this mean the script wont send anything because it didn't detect a change (PS turning off the strip made no difference either).

At this point i have the 2 streams appearing in SEG, but no data. It also shows only as being online in SEG 15 minutes ago...

At some point, tiny values for Wh and/or W are going to be in the level of error (typically 1-3% for a GEM, depending upon the CT used) so I wouldn't sweat that resolution too much.

... esp when your Daily consumption is going to be measured, and billed, in KWh, not Wh  8)


For debug, use the SEG "Tools" menu, and enable Trace for a bit.  It'll give you more information about how it's seeing the data stream you're sending.

It's very easy to get the Stream-name incorrect, and not have it pickup the stream.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 23, 2015, 12:42:48 pm

Quote
3) Whats the point in only sending energy changes, doesn't it need to send a reading for every device on a poll frequency?  Is there enough data fidelity if it is once in a blue moon or does it assume that the rate holds even over the time?  My data is steady state, as such i have not had a reading show up and i can't turn the strip off as my PC, Vera and other essential devices on it!  How can i boot strap a reading (both channels showed up).  Also i notice that the energy *is* chaging eg. 8.1 watt then 8.4 watts then 8.7 watts in the ISY. 

Rev 1.  That script is fairly simple, and only pushes data-changes.  The openHAB one pushes on change, but also every 2 minutes.  This can help with some graphing tools, as some don't handle data streams of vastly differing period.

The script itself is agnostic to what's generating the data, and on what frequency.  Always room for tuning..  ;-)

Guessed

I wish I was not such an idiot and knew how to do basic scripting  ???
Would providing the line(s) to tell Vera to poll and send current reading to SEG every X seconds/min, be a modest task to request from you?  It would be a huge help for me/us  and I/we would be extremely appreciative!!

This would greatly add to the functionality.  I had primarily looked at RAW data at SEG.  When I look at 30 min data, I see there are huge gaps in the streams when devices are off, asleep or constant.  In my HA energy conservation endeavors, I hope to maximize those periods.  I didn't realize this until mentioned above.

Fingers crossed...

Thanks



Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on March 23, 2015, 12:55:22 pm
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 23, 2015, 01:08:44 pm
Guessed, that's too bad, but I totally understand.  Are you willing to give the next guy a brief description of the conceptual steps required for that code update?  I would imagine it would be more feasible for someone to try to pick up where you left if they had a simple roadmap (or at least some clues), as you likely have the dual knowledge of SEG and Vera better than anyone else. 

Thanks you very much for writing your existing script!!  It is the reason I bought a Vera (as plug reporting to SEG is not possible directly from an ISY). 
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on March 23, 2015, 01:29:17 pm
For someone that knows Lua, it's not that complex.

It's basically:

a) Add a new, parameterless, function that can be used with luup.call_timer()
Put a loop in there that goes over all the values in the Array, and builds the complete "message" to send to SEG.  This will be similar to the existing one, except it'll build the entire list, not just the piece that changed.
This will use a for loop

The guts of this function will be of the form (syntax & output not checked):
Code: [Select]
    local segData = ""
    for k, v in pairs(SEG_DEVICES) do
        segData = segData .. string.format("(node %s ? (%s %s))", v.node, v.stream, luup.variable_get(v.deviceId, v.serviceId, v.serviceVar))
    end

    segData = string.format("(site %s %s)", SEG_SITE, segData)
    result, status = http.request(SEG_URL, segData)
b) Set  this new function to run under luup.call_timer() every "nn" seconds.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 23, 2015, 01:33:18 pm
Thanks Guessed.  Hopefully there is an Heir Apparent in the wings...
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: scyto on March 23, 2015, 04:32:37 pm
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.

Understand, I am trying to convince Ben at Brultech to get their DashBox to pull the data from ISY over IP REST interface - not sure I am getting far there...
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 23, 2015, 04:36:54 pm
@ccclapp ,
I'm not actively developing new code for Vera.  My time is spent with openHAB which, for me, is a better fit for whole-house automation/measurement/tuning needs.

I'm sure someone who still uses Vera will chime in with aid.

Understand, I am trying to convince Ben at Brultech to get their DashBox to pull the data from ISY over IP REST interface - not sure I am getting far there...

I have a Dash on the way.  As I understand it, with an ISY that will send to SEG fine.  The missing link has always been for plugs (or low cost HON eg Aeon).  I've got a lot of high users on 1-2 circuits, thus plug monitoring/reporting is critical, (at least for me and someone else without a GEM/Dash).
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: scyto on March 24, 2015, 06:51:38 pm
The dash will send the data collected by a GEM or ECM to SEG.
It can also send the data to the ISY.

I am not ware of anything that sends from ISY to SEG but ben just wrote a proto-script in his forum he things could do it fi there is data in ISY.

alex
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on March 24, 2015, 07:00:40 pm
I think you're correct.  GEM can nor report all channels directly to ISY but ISY doesn't report ANYTHING to SEG.

Also, most relevant to this thread is the fact that on any device ONLY GUESSED'S SCRIPT HANDLES PLUGS.  For me that's a key missing link...
Any chance you can/would take a stab at adding the polling/frequency edit to his script as he described in response to your comment about the impact of the lack of it?  I wish I could, but can't script :-(
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: scyto on March 25, 2015, 03:47:39 pm
Any chance you can/would take a stab at adding the polling/frequency edit to his script as he described in response to your comment about the impact of the lack of it?  I wish I could, but can't script :-(

Me neither, I can read one and make educated guess, actually figuring out where the braces go and what should be between them - no chance :-)  - unless I go pick up a book on lua scripting (my tip, download notepad++ on windows - makes reading these files a breeze)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: celbrandt on April 15, 2015, 07:04:08 am
I have now changed the script to send KWH data to SEG every 30 minutes. I have added a new parameter to each device I want to report to SEG, and modified the script to be used in a scene instead of under "startup-lua". This way I can easyli adjust how often values should be send to SEG, and if a device is asleep for several hours and hence has not updated the KWH for a long time nothing will be send to SEG untill the KWH is updated by the device.

The reason I need the extra paramenter is because SEG needs the delta KWH hence I need to know the prevous send value in order to calucalte the delta.

I have created a new paramenter for each device with the same serviceId as the original KWH (under the advanced tab->new variable"

then created a new new scene running every 30 min doing nothing but running this LUA script ( you can of course keep the exisiting script in start up lua for watts devices that should report watts in "real time" to SEG)

local http = require('socket.http')
http.TIMEOUT = 5
 
local SEG_SITE = "15e38bcc3f0d3d7"
local SEG_URL = "http://api.smartenergygroups.com/api_sites/stream"
local CALC_SID="urn:micasaverde-com:serviceId:EnergyMetering1"
 
SEG_DEVICES = {
   {node='vera', stream='e_beggekwh',   deviceId=3, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"},
   {node='vera', stream='e_northqt15',   deviceId=7, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"},
}

local i=1
while SEG_DEVICES do
    local v=SEG_DEVICES
    local oldCalc1 = luup.variable_get(CALC_SID, "KWHold", v.deviceId )
    local newCalc1 = luup.variable_get(CALC_SID, "KWHnew", v.deviceId)
    local oldCalc=tonumber(oldCalc1)
    local newCalc=tonumber(newCalc1)
    if (newCalc>oldCalc) then 
      luup.variable_set(CALC_SID, "KWHold", newCalc, v.deviceId)
      segData = string.format("(site %s (node %s ? (%s %s)))",  SEG_SITE, v.node, v.stream, newCalc-oldCalc)
      http.request(SEG_URL, segData)
    end
    i=i+1
 end


I have only had the veralite for 1 month and this is my first script written i LUA, (and the first script I have programmed for the last 3 years), hence please bear with me if it could have been done smarter, but I had to reuse as much as possible from guessed original script :-)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: gerardosamara on April 16, 2015, 04:28:14 am
Great stuff ... I have setup 3 streams ( temperature , energy watts and Kwh ) and they are displayed correctly on the SEG site ( Thanks Guessed)

But i noticed that there are some disconnections of the streams as per alert received from the SEG site below .Is it a known issue ?

Recent Device Events

@ your device dsb05 is missing, last known value was: 21.0 C 7 hours ago

@ your device dsb05 is back! with value: 21.0 C  9 hours ago

@  your device dsb05 is missing, last known value was: 22.0 C 4 hours ago

@ your device dsb05 is back! with value: 22.0 C  17 hours ago

Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on April 16, 2015, 02:03:35 pm
The two cases where I've seen SEG Report like that correlate to the client(s) "reporting too fast" and "reporting too slow".

In my case, I had a few other things reporting across the same entity and/or there was a volatile value in the list of stuff I sent over to SEG.

With energy data, unless it's some sort of daily-summary, I'd expect you'd get a reasonable level of volatility/change, so it's possible that it's too fast and SEG is throttling.

Quickest way to find out, esp if there's a [relatively] constant pattern, is to look at the Tools/Trace output (that's a SEG Menu/Tool) and see if your device is reporting (or not) during these periods.

For reference, my on-schedule Energy-based samples are sent to SEG ~5 minutes, with data collections every 90s.  My on-change openHAB transmissions are based upon change of the source data (in my case Nest-openHAB, polling every minute, and Weather, polling every 30 minutes)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: gerardosamara on April 16, 2015, 06:01:50 pm
OK @Guessed,

The decision to have on or off status of a stream is based on the missing period parameter in the API details ( 5000 Seconds )

As per the API trace , it is confirmed that the missing requests from Vera are the cause of these events due to no change reported to SEG within the API missing period of 5000 Seconds :

- The wall plug is always making energy report to SEG within the 5000 s period of time
- The 4in1sensor (DSB)  does not report temperature report  sometimes to SEG within the 5000 s period of time.

I was able to force a temperature report by changing the location of the DSB05 multisensor and to have then the status back on line for this device.

Conclusion : The root cause is sometimes too slow data , but this is consistent with the behavior of the devices which sends report only in case of change of value ( temperature or watts/kwh ).

Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: guessed on April 16, 2015, 07:11:06 pm
Some posts back, I pushed what was an almost-complete mod to the existing code to make it push data periodically. 

It only needed to be wrapped in a function declaration, and scheduled/re-scheduled via luup.call_timer() and it would provide both "periodic" as well as the existing "change-only" push to SEG.

That's one option. 

The next is to try the Scene-implemented version posted above, with the Scene acting as the timer. 

The third it to include "something" in the SEG list that does change often enough... Outdoor Temp, or anything really... it doesn't have to be data that SEG actually graphs  ;)
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: ccclapp on April 16, 2015, 07:22:06 pm
I have now changed the script to send KWH data to SEG every 30 minutes. I have added a new parameter to each device I want to report to SEG, and modified the script to be used in a scene instead of under "startup-lua". This way I can easyli adjust how often values should be send to SEG, and if a device is asleep for several hours and hence has not updated the KWH for a long time nothing will be send to SEG untill the KWH is updated by the device.

The reason I need the extra paramenter is because SEG needs the delta KWH hence I need to know the prevous send value in order to calucalte the delta.

I have created a new paramenter for each device with the same serviceId as the original KWH (under the advanced tab->new variable"

then created a new new scene running every 30 min doing nothing but running this LUA script ( you can of course keep the exisiting script in start up lua for watts devices that should report watts in "real time" to SEG)



Hi  celbrandt

I'm traveling for a bit and can't try your script, but wanted to thank you for your effort in adding periodic poling/reporting to guessed' script.

Is it working well?

Thanks again I look forward to this!
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: celbrandt on April 19, 2015, 03:26:23 pm
Hi CCCLAP

my script seems to Work flawlessly.  The last three days, KwH consumption for the last hours has been reported for 3 different stream every hour without any glitches.

But we still owe it all to Guessed. Without any LUA experience I could not have done it without all the hard work done initially by Guessed !



Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: bhunt99 on May 01, 2015, 10:33:31 am
hoping someone can help me troubleshoot this. I initially set up the original script to send energy and power data from an aeon labs smart switch hooked up to my washing machine. In SEG I created the dummy device, did the discovery, and it discovered the two streams and connected them to my device. Great so far.
I then tried to add streams for 2 home energy monitors. 2 monitors - power and energy for each. I'm not having any luck picking them up in SEG. When I do an api trace I almost never see anything at all in there, but the 2 streams for my washing machine device do regularly update. I tried to manually create another device thinking maybe I need to do that first, but I get a message about 'permalink already used'.

Here's my script
Code: [Select]
local http = require('socket.http')
http.TIMEOUT = 5
 
local SEG_SITE = "478c91a17fca017"
local SEG_URL = "http://api.smartenergygroups.com/api_sites/stream"
local segLog = function (text) luup.log('SEG Logger: ' .. (text or "empty")) end
 
SEG_DEVICES = {
    {node='vera', stream='p_washing_machine',   deviceId=19, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_washing_machine', deviceId=19, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"},
    {node='vera', stream='p_hem1', deviceId=28, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_hem1', deviceId=28, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"},
    {node='vera', stream='p_hem2', deviceId=37, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="Watts"},
    {node='vera', stream='e_hem2', deviceId=37, serviceId='urn:micasaverde-com:serviceId:EnergyMetering1', serviceVar="KWH"}
  }
 
local function findStream(deviceId, serviceId, serviceVar)
  -- TODO: Change from a simple scan to something that'll support more entries efficiently.
  for k, v in pairs(SEG_DEVICES) do
    if (v.deviceId == deviceId and
        v.serviceId == serviceId and
        v.serviceVar == serviceVar) then
      return v.node, v.stream
    end
  end
 
  return nil, nil
end
 
local function initWatch()
  for k, v in pairs(SEG_DEVICES) do
    luup.variable_watch('segWatch', v.serviceId, v.serviceVar, v.deviceId)
  end
end
 
function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
  local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)
 
  if (nodeName == nil or streamName == nil) then
    segLog(string.format("Node not found for %s,%s,%s", deviceId, serviceId, serviceVar))
    return
  end
 
  -- TODO: Encoding
  segData = string.format("(site %s (node %s ? (%s %s)))", SEG_SITE, nodeName, streamName, newValue)
  segLog(segData)
 
  result, status = http.request(SEG_URL, segData)
end
 
initWatch()
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: bhunt99 on May 01, 2015, 02:13:01 pm
I think I have it figured out now and I was just thinking of things on the SEG side in wrong way. I was expecting a device in SEG for each device in Vera. Now I set up 1 device in SEG called Vera w/ multiple streams coming from the different physical devices I have and everything seems to be working.

Is that the correct way to set it up on the SEG side?
Site = Vera
Device = Vera
Stream 1, 2, 3, 4, 5 etc. = p_hem1, e_hem1, p_hem2, e_hem2, p_washing_machine, etc.
Title: Re: Energy/Temperature Publication using SmartEnergyGroups.com (SEG)
Post by: gerardosamara on October 07, 2015, 10:11:24 am
Hum , I received a mail from SG this morning ,


Smart Energy Groups has been free for the majority of our members now for a long time now, and we need your support in order to continue.

This means that Smart Energy Groups is transitioning to a paid service in order for it to continue.

When you log in next, you will be prompted to select from our Community or Premium plans. This will set you up with an account and all the other necessary things, like your credit card details etc.


EDIT : After checking the SEG site , the free subscription still exists but with only 3 streams