We have moved at community.getvera.com

Author Topic: ALTUI & DataStorage Providers  (Read 10878 times)

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
ALTUI & DataStorage Providers
« on: January 19, 2016, 07:39:36 pm »
ALTUI & DataStorage Providers

DataStorage Providers are an alternative to the thingspeak integration that ALTUI provides. they are optional external module which can be used to receive the notification of a variable change and do whatever you want with it ( storage in another cloud, in a DB etc ). The user can choose which underlying DataStorage provider he wants to chose when he selects the Variable Push option.

You can implement DataStorage Providers as a LUA plugin, a piece of code in openLuup, or even something completely external like a node.js app running on some kind of Ux box ( like a raspberry ) as long as you can call a http GET and have a handler to be called in return in a url via a GET request

The high level summary on how it works is:
- ALTUI load DataProviders config from its variable
- or ALTUI UPNP action is called to register a new data provider
- the User configures a watch on a variable by specifying the provider he wants to use and the parameters required by this provider
- when the variable changes, ALTUI calls the DataStorage provider with the same parameters as a LUA watch + the DataProvider specific parameters

Lets see the details :

DataStorage Declaration

ALTUI device has a new variable called DataStorageProviders to keep this data provider configuration persistent accross reload/reboot

Thingspeak is allways provided by ALTUI but additional DataStorage can be dynamically declared by a UPNP action "RegisterDataProvider" requiring 3 parameters.
Code: [Select]
        <action>
            <name>RegisterDataProvider</name>
            <argumentList>
                <argument>
                    <name>newName</name>
                    <direction>in</direction>
                    <relatedStateVariable>Name</relatedStateVariable>
                </argument>
                <argument>
                    <name>newUrl</name>
                    <direction>in</direction>
                    <relatedStateVariable>Url</relatedStateVariable>
                </argument>
                <argument>
                    <name>newJsonParameters</name>
                    <direction>in</direction>
                    <relatedStateVariable>JsonParameters</relatedStateVariable>
                </argument>
            </argumentList>
        </action>

-Name : must be a unique name. ( thingspeak with the built in ALTUI thingspeak integration )
-Url : a URL that will be called as a callback when a variable change ( variable for which the user has selected a data push for this provider )
-JSON parameter:  a string representation of a JSON object specifying the user-interface parameters that the user must fill in.


That JSON is an indexed array of parameter objects. each parameter object defines a particular parameter

For thingspeak for instance, in LUA such an object would be like this. note it is a true indexed array. order and indexes matters.
Code: [Select]
{
[1] = { ["key"]= "channelid", ["label"]="Channel ID", ["type"]="number" },
[2] = { ["key"]= "readkey", ["label"]="Read API Key", ["type"]="text" },
[3] = { ["key"]= "writekey", ["label"]="Write API Key", ["type"]="text" },
[4] = { ["key"]= "fieldnum", ["label"]="Field Number", ["type"]="number", ["default"]=1 },
[5] = { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url" , ["default"]="//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true"}
}

in the UPNP action you would use the JSON stringified format of that object so something like:
in JSON
Code: [Select]
[{"type":"number","key":"channelid","label":"Channel ID"},{"type":"text","key":"readkey","label":"Read API Key"},{"type":"text","key":"writekey","label":"Write API Key"},{"default":1,"type":"number","key":"fieldnum","label":"Field Number"},{"default":"//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true","type":"url","key":"graphicurl","label":"Graphic Url"}]key is internal , label is what the user sees in the dialog box, type is a HTML 5 INPUT Type, default is a default value

By convention, the parameter named "graphicurl" is reserved to be a url with some {n} placeholders inside which are replaced at the time of displaying a graphic in an iFRAME
whose src attribute is that url where the {n} are replaced by the value of the n-th parameter of the array


When a variable change happens

if a DataStorage Provider was selected for that variable, ALTUI will call the url passed as part of the datastorage declaration with the following parameters:
Code: [Select]
-http://url...?lul_device=<n>&lul_variable=<n>&old=<n>&new=<n>&lastupdate=<n>&key=value&...  for all parameters provided on the DataStorage declaration
-http verb = GET

Caveat: calling UPNP action using http as documented by MCV requires to url encode the parameters. example, for a datastorage provider defined as:
Code: [Select]
[{"type":"text","key":"toto","label":"To To"},{"default":"//www.google.com/{0}","type":"url","key":"graphicurl","label":"Graphic Url"}]
The url to call the UPNP action to register the provider would be:
Code: [Select]
http://192.168.1.16/port_3480/data_request?id=action&output_format=json&DeviceNum=216&serviceId=urn:upnp-org:serviceId:altui1&action=RegisterDataProvider&newName=toto&newUrl=http%3A%2F%2F192.168.1.16%2Fport_3480%2Fdata_request%3Fid%3Dlr_ALTUI_Handler%26command%3DdataPush&newJsonParameters=%5B%7B%22type%22%3A%22text%22%2C%22key%22%3A%22toto%22%2C%22label%22%3A%22To%20To%22%7D%2C%7B%22default%22%3A%22%2F%2Fwww.google.com%2F%7B0%7D%22%2C%22type%22%3A%22url%22%2C%22key%22%3A%22graphicurl%22%2C%22label%22%3A%22Graphic%20Url%22%7D%5D

In return, when receiving the parameters back from ALTUI when it calls
Code: [Select]
http://url...?lul_device=<n>&lul_variable=<n>&old=<n>&new=<n>&lastupdate=<n>&key=value&...remember that the parameter values will be urlencoded so you need to urldecode them ( replace + with space etc )
For example ALTUI is calling this hypothetical "toto" data storage provider by
Code: [Select]
http://url?&toto=This+is+a+text&graphicurl=%2F%2Fwww.google.com%2F%7B0%7D
EDIT : Examples of Data Providers.
« Last Edit: January 26, 2016, 10:25:06 am by amg0 »

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #1 on: January 20, 2016, 03:39:56 am »
This looks great! Can't wait to give it a go with DataYours, it will be a much better interface than I have at the moment for defining new variables to watch.

Will let you know how it goes! (and probably have a few questions.)

Thanks again.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #2 on: January 20, 2016, 01:06:52 pm »
I added http://emoncms.org/ integration as a new (built-in) DataProvider in Version V>= 1128


NOTE : new parameter "ifheight" to set the height of the iframe to display a graphic ( if graphicurl field is present )

Code: [Select]

{
[1] = { ["key"]= "nodeid", ["label"]="Node ID", ["type"]="number" ,["default"]=1},
[2] = { ["key"]= "feedid", ["label"]="Feed ID", ["type"]="number" },
[3] = { ["key"]= "inputkey", ["label"]="Input Key name", ["type"]="text" },
[4] = { ["key"]= "readwritekey", ["label"]="Read/Write API Key", ["type"]="text" },
[5] = { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url", ["ifheight"]=460, ["default"]="http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}"}
}
« Last Edit: January 20, 2016, 01:10:34 pm by amg0 »

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #3 on: January 21, 2016, 05:24:54 am »
Just about to start trying this out.
  • What happens if the URL for the storage provider can't be reached?
  • What's the timeout?
  • Surely this will slow things down a lot?

Also, I believe you can use the routines located in the socket.url module to do the url encode/decoding without having to write your own:

Code: [Select]
local url = require "socket.url"

local x = "space percent%ampersand&plus+"
local y = url.escape (x)
local z = url.unescape(y)

print("encoded", y)
print("decoded", z)

gives:
Code: [Select]
encoded space%20percent%25ampersand%26plus%2b
decoded space percent%ampersand&plus+
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #4 on: January 21, 2016, 05:57:00 am »
Just about to start trying this out.
  • What happens if the URL for the storage provider can't be reached?
  • What's the timeout?
  • Surely this will slow things down a lot?

Also, I believe you can use the routines located in the socket.url module to do the url encode/decoding without having to write your own:

Code: [Select]
local url = require "socket.url"

local x = "space percent%ampersand&plus+"
local y = url.escape (x)
local z = url.unescape(y)

print("encoded", y)
print("decoded", z)

gives:
Code: [Select]
encoded space%20percent%25ampersand%26plus%2b
decoded space percent%ampersand&plus+

Cool, I have to try these routines.

regarding call from ALTUI to external url , I prepare the data, then I call it in a delayed routine so I hope and do not believe we should be impacted by the time it takes
Code: [Select]
luup.call_delay("myhttpget",1,url)
then I used it pretty much all default ( 10s I think )
Code: [Select]
function myhttpget(url)
debug(string.format("myhttpget(%s)",url))
local response_body = {}
local response, status, headers = http.request{
method="GET",
url=url,
-- source = ltn12.source.string(data),
sink = ltn12.sink.table(response_body)
}
debug("https Response=" .. json.encode({res=response,sta=status,hea=headers}) )
return response
end

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #5 on: January 22, 2016, 06:34:08 am »
I see the latest build includes a "callback" parameter in the DataStorageProviders structure.

Code: [Select]
{
  datayours = {
    callback = "",
    parameters = {},
    url = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&format=altui"
  },
  emoncms = {
    callback = "sendValueToStorage_emoncms",
    parameters = {{
        default = 1,
        key = "nodeid",
        label = "Node ID",
        type = "number"
      },{
        key = "feedid",
        label = "Feed ID",
        type = "number"
      },{
        key = "inputkey",
        label = "Input Key name",
        type = "text"
      },{
        key = "readwritekey",
        label = "Read/Write API Key",
        type = "text"
      },{
        default = "http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}",
        ifheight = 460,
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  },
  thingspeak = {
    callback = "sendValueToStorage_thingspeak",
    parameters = {{
        key = "channelid",
        label = "Channel ID",
        type = "number"
      },{
        key = "readkey",
        label = "Read API Key",
        type = "text"
      },{
        key = "writekey",
        label = "Write API Key",
        type = "text"
      },{
        default = 1,
        key = "fieldnum",
        label = "Field Number",
        type = "number"
      },{
        default = "//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  }
}

Is this a local callback routine we can provide instead of constructing a URL for saving the data, or is this only for your built-in data providers?

DataYours as a data storage provider is now working in a development version - it required very few changes, so this is an easy API to use.  I'm moving on to the plotting aspect now.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #6 on: January 22, 2016, 07:15:07 am »
I see the latest build includes a "callback" parameter in the DataStorageProviders structure.

Code: [Select]
{
  datayours = {
    callback = "",
    parameters = {},
    url = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&format=altui"
  },
  emoncms = {
    callback = "sendValueToStorage_emoncms",
    parameters = {{
        default = 1,
        key = "nodeid",
        label = "Node ID",
        type = "number"
      },{
        key = "feedid",
        label = "Feed ID",
        type = "number"
      },{
        key = "inputkey",
        label = "Input Key name",
        type = "text"
      },{
        key = "readwritekey",
        label = "Read/Write API Key",
        type = "text"
      },{
        default = "http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}",
        ifheight = 460,
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  },
  thingspeak = {
    callback = "sendValueToStorage_thingspeak",
    parameters = {{
        key = "channelid",
        label = "Channel ID",
        type = "number"
      },{
        key = "readkey",
        label = "Read API Key",
        type = "text"
      },{
        key = "writekey",
        label = "Write API Key",
        type = "text"
      },{
        default = 1,
        key = "fieldnum",
        label = "Field Number",
        type = "number"
      },{
        default = "//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  }
}

Is this a local callback routine we can provide instead of constructing a URL for saving the data, or is this only for your built-in data providers?

DataYours as a data storage provider is now working in a development version - it required very few changes, so this is an easy API to use.  I'm moving on to the plotting aspect now.

yes indeed , this is for "embedded" providers ( thingspeak, emoncms ) as I could not figure out how a function name and function pointer could be used by an external module , nor passed as parameters of a UPNP action. 
There is no persistency of these type of providers, meaning they need to re-register at each lua restart as I have to have also another datastructure which holds the mapping between the name and the function itself.
All my attempts with loadstring() and things like that to change context were failures probably due to my incompetency in lua :-)
Code: [Select]
local DataProvidersCallbacks={} -- map names to functions in the local context, only for embedded providers registered within this module in LUA

Alexis

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #7 on: January 22, 2016, 07:36:33 am »
yes indeed , this is for "embedded" providers ( thingspeak, emoncms ) as I could not figure out how a function name and function pointer could be used by an external module , nor passed as parameters of a UPNP action. 

Yes, I'd thought about that problem.  It seems to me that it could be overcome if the callback parameter was actually the name of a module you could use with require() with a known function name entry point, say DataStorageProvider.  So in the Lua code you'd do something like:

Code: [Select]
local usermodule = require (userSuppliedName)
usermodule.DataStorageProvider (...)

However, probably not worth the effort since I doubt many people will want to write local storage providers.

For openLuup, it's not an issue since a local HTTP call never gets to the TCP stack and is converted directly into a call to the request handler anyway.  For DataYours this is particularly effective since it then issues a UDP request, possibly to a remote system, which is much lower overhead and doesn't have a handshake which can timeout. (It also doesn't guarantee receipt, but over my LAN I've logged literally millions of UDP packets for DataYours, none of which have ever been 'lost'.)


3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #8 on: January 22, 2016, 10:07:32 am »
Success!  See attached screen capture of DataYours plotting a graph as a DataStorage Provider in AltUI.

You'll see that the graphicurl is parameter-less, but it would be great if there were some substitutions which were available for deviceNo, serviceId, variableName, etc., and the url could then be built automatically.

DataYours, running DataWatcher, should be installed on the local machine, but can send the data to a remote DataCache.  Also, in this case the graphicurl points to the local machine, but there's no reason this cannot be remote as well.

-----

Edit: added a second version with manual parameters for the device/service/ etc.  It's these for which it would be great to have automatic substitutions.
« Last Edit: January 22, 2016, 11:04:57 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #9 on: January 22, 2016, 11:21:17 am »
Success!  See attached screen capture of DataYours plotting a graph as a DataStorage Provider in AltUI.

You'll see that the graphicurl is parameter-less, but it would be great if there were some substitutions which were available for deviceNo, serviceId, variableName, etc., and the url could then be built automatically.

DataYours, running DataWatcher, should be installed on the local machine, but can send the data to a remote DataCache.  Also, in this case the graphicurl points to the local machine, but there's no reason this cannot be remote as well.

-----

Edit: added a second version with manual parameters for the device/service/ etc.  It's these for which it would be great to have automatic substitutions.

Great stuff.

one undocumented feature here to control the height of the graphic IFrame. this is for emoncsm. see the graphicurl field definition. it contains a 'ifheight' attribute. if it is present, I use this for the IFrame height
Code: [Select]
{
[1] = { ["key"]= "nodeid", ["label"]="Node ID", ["type"]="number" ,["default"]=1},
[2] = { ["key"]= "feedid", ["label"]="Feed ID", ["type"]="number" },
[3] = { ["key"]= "inputkey", ["label"]="Input Key name", ["type"]="text" },
[4] = { ["key"]= "readwritekey", ["label"]="Read/Write API Key", ["type"]="text" },
[5] = { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url", ["ifheight"]=460, ["default"]="http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}"}
}

For your substitution idea,  I ll give some thoughts

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #10 on: January 24, 2016, 06:35:04 am »
Really appreciate the flag for variables using the DataStorage Provider facility in build 1151 - thanks!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #11 on: January 24, 2016, 10:39:37 am »
Really appreciate the flag for variables using the DataStorage Provider facility in build 1151 - thanks!
and if you hover over it , ( not on mobiles ) it tells you the provider name

Offline rutger

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
Re: ALTUI & DataStorage Providers
« Reply #12 on: February 28, 2016, 08:38:21 am »
Is it possible to modify the default fields which are being sent with the request?
If I'm correct these fields are sent by default:
  • lul_service
  • lastupdate
  • lul_variable
  • old
  • lul_device

I'd like to add the name for example.

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3174
  • Karma: +209/-8
Re: ALTUI & DataStorage Providers
« Reply #13 on: February 29, 2016, 02:55:20 am »
Is it possible to modify the default fields which are being sent with the request?
If I'm correct these fields are sent by default:
  • lul_service
  • lastupdate
  • lul_variable
  • old
  • lul_device

I'd like to add the name for example.

Currently the parameters passed on the data watch notification url are:
lul_device, lul_service, lul_variable,old, new, lastupdate, provider_params

provider_params being the data provider specific parameters passed at the time of the registration via the UPNP call. UPNPregisterDataProvider()


What name would you like to add ?name of the device owning the variable  ?

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +290/-70
  • "Less is more"
Re: ALTUI & DataStorage Providers
« Reply #14 on: February 29, 2016, 05:59:21 am »
Isn't that what the newJsonParameters variable is for?

When registering DataYours (see DataYours - a Data Storage Provider for AltUI)  I use the following Lua code:
Code: [Select]
local newJsonParameters = {
    {
        default = "unknown",
        key = "target",
        label = "Metric Name",
        type = "text"
      },{
        default = "/data_request?id=lr_render&target={0}&hideLegend=true&height=250&from=-y",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }
    }
  local arguments = {
    newName = "datayours",
    newUrl = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&target={0}",
    newJsonParameters = json.encode (newJsonParameters),
  }

  luup.call_action ("urn:upnp-org:serviceId:altui1", "RegisterDataProvider", arguments, AltUI)

When selecting a variable for storage through the AltUI interface, the "Metric Name" label parameter appears on the menu to be filled in with the desired name.  In my case, I use this to indicate the name of the file to be stored and the maximum storage time of the archive.  But it could be whatever you like for your own provider code.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.