Vera - Smarter Home Control Forum

Advanced => Programming => Plugins & Plugin Development => Topic started by: ustredna on January 16, 2018, 02:14:03 am

Title: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: ustredna on January 16, 2018, 02:14:03 am
Hi friend,
i'm new in VERA. Vera is very good controller for me, but i need help how can i parse online JSON data from my heating machine cloud.

json data>

http://www.stokercloud.dk/dev/getjsondriftdata.php?mac=hampl


i need read this some JSON data and store into variable as temperature, status, ...
and i need automatically refresh this data each 5 minutes
how can i write easy lua script? is possible run this script automatically each 5 minutes in scenes scheduler?

how make this?


many tnx for your help.
Best regards
Peter

 
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: akbooer on January 16, 2018, 06:29:40 am
i need read this some JSON data and store into variable as temperature, status, ...
and i need automatically refresh this data each 5 minutes

This is very easy if you use  a JSON decoder.  Fortunately, Vera comes with one these days called dkjson.  For example:
Code: [Select]
local json = require "dkjson"

local j = [[
{"501":"34.1 °C flow","502":"0 % Power","503":"24 °C Shaft","504":"0 kW Power","505":"18.2 kg Hopper","506":"0 Lx Light",
"507":"Error no pellets ","508":"16/01-2018 12:22:30","521":"0 °C Return","522":"0 liter/hour Flow","524":"0 °C External temperature",
"525":"54 °C Temperature DHW","526":"15.7 % Actual oxygen","527":"0 % Target oxygen","528":"850 Gram auger/6 min","530":"10.115 kg Today",
"531":"0 kg/m2 Today","532":"0 °C Smoke temp.","533":"65 °C Target boiler temp.","534":"56 °C Target DHW temp.","542":" ","584":"Nitra ",
"585":"86 % humidity","586":"1005 hPa pressure","587":"6.2 m/s wind speed","588":"http://openweathermap.org/img/w/13d.png ",
"589":"-2 °C air temperature","591":"130 null","592":"0.0 °C T5","alarm":"1"}
]]

local data = json.decode(j)

results in the following Lua data structure:
Code: [Select]
{
  ["501"] = "34.1 °C flow",
  ["502"] = "0 % Power",
  ["503"] = "24 °C Shaft",
  ["504"] = "0 kW Power",
  ["505"] = "18.2 kg Hopper",
  ["506"] = "0 Lx Light",
  ["507"] = "Error no pellets ",
  ["508"] = "16/01-2018 12:22:30",
  ["521"] = "0 °C Return",
  ["522"] = "0 liter/hour Flow",
  ["524"] = "0 °C External temperature",
  ["525"] = "54 °C Temperature DHW",
  ["526"] = "15.7 % Actual oxygen",
  ["527"] = "0 % Target oxygen",
  ["528"] = "850 Gram auger/6 min",
  ["530"] = "10.115 kg Today",
  ["531"] = "0 kg/m2 Today",
  ["532"] = "0 °C Smoke temp.",
  ["533"] = "65 °C Target boiler temp.",
  ["534"] = "56 °C Target DHW temp.",
  ["542"] = " ",
  ["584"] = "Nitra ",
  ["585"] = "86 % humidity",
  ["586"] = "1005 hPa pressure",
  ["587"] = "6.2 m/s wind speed",
  ["588"] = "http://openweathermap.org/img/w/13d.png ",
  ["589"] = "-2 °C air temperature",
  ["591"] = "130 null",
  ["592"] = "0.0 °C T5",
  alarm = "1"
}


Quote
how can i write easy lua script? is possible run this script automatically each 5 minutes in scenes scheduler?

The easiest way to do this is simply to put your Lua code into a scene and schedule that scene to run every 5 minutes.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: ustredna on January 16, 2018, 06:36:02 am
ok,tnx

and how can i readJSON online direct from HTTP link?

Can you write me a simple LUA script with write procedure into device variable sample... i use Multistring Plugin for showing data
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: akbooer on January 16, 2018, 08:56:11 am
The Luup function luup.inet.wget() to make and receive HTTP requests is documented here

http://wiki.micasaverde.com/index.php/Luup_Lua_extensions#function:_wget
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on January 16, 2018, 01:13:29 pm
There's a plugin called SiteSensor that will do this for you. You can configure it to query a remote API for a JSON response, and tell it what response fields go into variables that you can access from scenes and Lua scripting.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: ustredna on January 17, 2018, 06:53:10 am
Hi,
this plugin is super, but i need show this variables as device variables in device list and show this readed variable in imperihome panel on wall tablet.
imperihome don't see this device variables.

any reason for me?
tnx friends.


Peter
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on January 17, 2018, 08:22:33 am
I don't use Imperihome, so I can't speak to that. In order for the device variables to be populated, you need to specify the reference equations in the configuration. You then should be able to see your individual values on the Vera side by going into the settings/control page for the SiteSensor device, where they appear next to the bracketed numbers (i.e. [1] through [8]). You should see your parsed values there. You can also confirm this by going into the Advanced tab, then Variables, and looking at Value1 through Value8. In either case, if you don't have the expected values, you have an incomplete or incorrect configuration.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 19, 2018, 12:05:07 pm
@rigpapa, great plugin! Just trying to use it with openweathermap, got the last query succeeded, that's a good performance for me  :)
But now two points,
1. I selected Server response is handled as: Text is this right?
Because I got this as result on the web page: (here's an extract)
...
09:00:00"},{"dt":1529496000,"main":{"temp":296.314,"temp_min":296.314,"temp_max":296.314,"pressure":842.84,"sea_level":1032.69,"grnd_level":842.84,"humidity":29,"temp_kf":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01d"}],"clouds":{"all":0},"wind":{"speed":0.72,"deg":322},"rain":{},"sys":{"pod":"d"},"dt_txt":"2018-06-20 12:00:00"},{"dt":1529506800,"main":{"temp":294.398,"temp_min":294.398,"temp_max":294.398,"pressure":842.32,"sea_level":1031.73,"grnd_level":842.32,"humidity":36,"temp_kf":0},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10d"}],"clouds":{"all":44},"wind":{"speed":0.61,"deg":324.502},"rain":{"3h":0.13},"sys":{"pod":"d"},"dt_txt":"2018-06-20
...
2.  What is the correct syntax to read for example, let's say the "pressure" and "Main" variable? I put "pressure" on the Value Expressions, but I do not understand this part, so nothing appears next to the [1]...

I see your example, but not able to adapt it: { "errCode": 0, "type": { "name": "Normal", "class": "apiobject" } }

In fact I just want to use these variables the same way I use them with the WeatherUnderground plugin. To compare...

Thanks for your time.
 
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on June 19, 2018, 12:07:58 pm
You need to select JSON response type. That will give you access to the expression fields to extract data from the JSON response. I can't help you with the expressions as posted because you only posted a fragment. Post the whole thing (or email it to me, on my profile) and I'll give you some guidance.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 19, 2018, 12:15:13 pm
Sorry, I have just modified my post in the same time!
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on June 19, 2018, 12:57:26 pm
OK. Got your email.

First, before we really start digging deep, you're doing a forecast request, rather than a current conditions request. Is that intended? If not, your request URL should look more like this:

http://api.openweathermap.org/data/2.5/weather?id=xxxxxxxx&APPID=yyyyyy

And from your second email, you always need to have the protocol ("http://" or "https://") at the front of the request URL.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 20, 2018, 02:51:23 am
Ok, I use both, Current condition for the outside blind control, because this is critical to me when there is thunderstorm or high wind to put it up. And I use forecast for Heat Pump management, winter time. For example when day+1 forecast is Sunny at 6:00AM the next morning Heat Pump will stay Off, Sun is enough to warm the house.
But if you agree, we are in summer time, so lets start with Current conditions.

Ok did the changes and now I have the correct answer:
08:53:28: SiteSensor: Response HTTP status 200, body="{\"coord\":{\"lon\":7.38,\"lat\":46.26},\"weather\":[{\"id\":800,\"main\":\"Clear\",\"description\":\"clear sky\",\"icon\":\"01d\"}],\"base\":\"stations\",\"main\":{\"temp\":293.15,\"pressure\":1023,\"humidity\":60,\"temp_min\":293.15,\"temp_max\":293.15},\"visibility\":10000,\"wind\":{\"speed\":2.6,\"deg\":90},\"clouds\":{\"all\":0},\"dt\":1529475600,\"sys\":{\"type\":1,\"id\":6005,\"message\":0.003,\"country\":\"CH\",\"sunrise\":1529465903,\"sunset\":1529522751},\"id\":ID,\"name\":\"CityName\",\"cod\":200}"

So now what is the syntax to get for example "main" as [1] -> "Clear" ? If I can get this, I will understand!
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on June 20, 2018, 08:11:44 am
OK. You've actually chosen a slightly more difficult example for two reasons: (1) "main" is used in two places in the response, so it adds a little potential confusion; and (2) the one you want to use is contained in array of one element, which complicates the reference syntax a little.

Let's start with a simpler example and build on it. I'll use "temp", which is the current temperature. You navigate through the subkeys in the response using the key "response" as the root. So, "temp" is found underneath "main", so we refer to it as response.main.temp. As OpenWeatherMap.org returns it, the temperature is in degrees Kelvin, so we need to convert to Celsius by simply subtracting 273.15, so now response.main.temp - 273.15, or Fahrenheit with response.main.temp*9/5-459.97. You can add the "round" function to clean that up a bit: round(response.main.temp*9/5-459.97,1) rounds the value to one decimal place (second argument to round is number of decimals).

The humidity and pressure keys can also be found under the "main" subkey, so they are response.main.humidity and response.main.pressure.

Now, let's get back to your example--digging out the word "Clear". That is also under a subkey called "main", but it's a subkey of "weather", and you may notice that "weather" is an array. So we need to refer to main through weather as an array element. That can be either response.weather[1].main or last(response.weather).main. The former chooses the first element in the array, and the latter chooses the last, but since it's an array of one element, the effect is the same, so you can use either.

I've put this entire recipe up in the web site documentation at https://www.toggledbits.com/sitesensor, and the full expression syntax and function library for SiteSensor are documented with LuaXP here: https://www.toggledbits.com/luaxp

When looking at the forecast response, it returns an array for different times of day upcoming. You will need to sort out which one you want to use, but I believe they are returned in ascending time order, so the first would be the soonest. Give it a try and see how you do, and if it gives you trouble, post in the SiteSensor plugin support thread (http://forum.micasaverde.com/index.php/topic,50440.0.html) and we'll get it sorted.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 20, 2018, 09:52:15 am
WOW, great! I just followed your example, everything is working great, just one error with humidity:
15:45:23: SiteSensor: Response HTTP status 200, body="{\"coord\":{\"lon\":7.38,\"lat\":46.26},\"weather\":[{\"id\":701,\"main\":\"Mist\",\"description\":\"mist\",\"icon\":\"50d\"}],\"base\":\"stations\",\"main\":{\"temp\":294.73,\"pressure\":1018,\"humidity\":37,\"temp_min\":279.15,\"temp_max\":303.15},\"visibility\":10000,\"wind\":{\"speed\":6.7,\"deg\":260},\"clouds\":{\"all\":20},\"dt\":1529499300,\"sys\":{\"type\":1,\"id\":6005,\"message\":0.0049,\"country\":\"CH\",\"sunrise\":1529465906,\"sunset\":1529522754},\"id\":2660503,\"name\":\"Grimisuat\",\"cod\":200}"
15:45:23: SiteSensor: Eval #1: "last(response.weather).main"=("string")"Mist"
15:45:23: SiteSensor: Eval #2: "response.name"=("string")"Grimisuat"
15:45:23: SiteSensor: Eval #3: "round(response.main.temp * 9 / 5 - 459.97,1)"=("number")70.5
15:45:23: SiteSensor: Failed to execute `"response.humidity"', { location=18, __source="luaxp", message="Subreference not found: humidity", type="evaluation" }
15:45:23: SiteSensor: Eval #4: "response.humidity"=("nil")nil
15:45:23: SiteSensor: Eval #5: "round(response.main.pressure * 0.029529988,2)"=("number")30.06
15:45:23: SiteSensor: Eval trip expression: "response.cod == 200"=("boolean")true
15:45:23: SiteSensor: Next activity in 7200 seconds
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 20, 2018, 10:15:48 am
Was even able to add:
16:13:23: SiteSensor: Eval #6: "response.wind.speed"=("number")6.7
16:13:23: SiteSensor: Eval #7: "response.wind.deg"=("number")260

And to convert from m/s to km/h: round(response.wind.speed * 18 /5)

So everybody should be able to play with this now,  ;) thanks again for this great plugin!
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 20, 2018, 10:43:30 am
Ok, for me the error was: (forget the main)

16:42:05: SiteSensor: Eval #3: "response.main.temp - 273.15"=("number")22.25
16:42:05: SiteSensor: Eval #4: "response.main.humidity"=("number")38
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on June 20, 2018, 10:44:57 am
15:45:23: SiteSensor: Failed to execute `"response.humidity"', { location=18, __source="luaxp", message="Subreference not found: humidity", type="evaluation" }

The expression should be response.main.humidity, because humidity lives underneath the main subkey in the response.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 21, 2018, 04:59:53 am
More complicated, because sometimes datas are missing, like for the wind direction (deg):
10:57:00: SiteSensor: Failed to execute `"response.wind.deg"', { location=18, __source="luaxp", message="Subreference not found: deg", type="evaluation" }

{"coord":{"lon":7.36,"lat":46.23},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"base":"stations","main":{"temp":290.14,"pressure":1018,"humidity":50,"temp_min":277.15,"temp_max":298.15},"visibility":10000,"wind":{"speed":0.5},"clouds":{"all":20},"dt":1529569200,"sys":{"type":1,"id":6005,"message":0.0028,"country":"CH","sunrise":1529552328,"sunset":1529609161},"id":2658576,"name":"Sion","cod":200}
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on June 21, 2018, 10:22:15 am
OK. For this one, you'll need a minor update. Download the attached Lua file and upload it to your Vera through the Apps > Develop apps > Luup files uploader. This update allows references to non-existent subkeys to put null into the result stack rather than immediately throwing an error.

Once that's done, you can use the if() function like this: if( response.wind==null, "no wind data", response.wind.deg )

The if() function takes three arguments: a condition expression (boolean), an expression to evaluate when the condition is true, and an expression to evaluate when it's false (it's modeled after the Excel function of the same name). In this case, if OpenWeatherMap does not provide wind data, response.wind will evaluate to null, so the "no wind data" string will be the expression result. If it is provided (i.e. response.wind is not null) then the expression returns the value of response.wind.deg.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: chris66 on June 22, 2018, 04:01:52 am
@rigpapa, thanks very much, never seen so good support! Will try this after Holidays, a nice day/holidays?  8) Chris
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: cranb on September 30, 2018, 01:29:56 pm
Hello,

I have set up the sitesensor plugin to parse the data from an openweathermap URL. Works great. As i am looking for a replacement for the Wunderground weather plugin, would it be possible to put the data from the sitesensor into a device? I am looking for a setup similar to the Wunderground plugin which consist of a number of devices (temp, hum) that get their data from the plugin. See attached screenshot.


Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on September 30, 2018, 01:38:42 pm
Hello,

I have set up the sitesensor plugin to parse the data from an openweathermap URL. Works great. As i am looking for a replacement for the Wunderground weather plugin, would it be possible to put the data from the sitesensor into a device? I am looking for a setup similar to the Wunderground plugin which consist of a number of devices (temp, hum) that get their data from the plugin. See attached screenshot.

The data that your 8 expressions produce is stored in your SiteSensor sensor, and can be retrieved into Lua, Reactor, PLEG, etc. Use service urn:toggledbits-com:serviceId:SiteSensor1, and the variable names are Value1 through Value8.
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: cranb on September 30, 2018, 04:13:52 pm
Thanks for your quick answer! I am able to get the values in Pleg and use them in for example the multistring plugin. But what i really want is to create a separate device to display that value and that is where my knowledge is not sufficient at the moment  :(  Could you provide some explanation / steps on how to achieve this?
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: rigpapa on September 30, 2018, 06:39:53 pm
Thanks for your quick answer! I am able to get the values in Pleg and use them in for example the multistring plugin. But what i really want is to create a separate device to display that value and that is where my knowledge is not sufficient at the moment  :(  Could you provide some explanation / steps on how to achieve this?

If you just want to display one of the values (and given that it's weather data, I'm guessing that's not the case... you want temp, humidity, etc.), you can put an expression into the "MessageExpr" state variable in your SiteSensor, and it will display that expression result on its dashboard card. Since the data is probably already one of the existing 8 expressions, you can set MessageExpr to expr[n] where n is 1 to 8 and it will show the value from that expression.

If you need to display the result of more than one expression result in as many devices, I don't have a built-in solution for you at the moment. I can add the ability to create virtual devices of specific types to the future feature list. In the meanwhile, if it's only about display, I'm thinking rstrouse's excellent Virtual Panel plugin may be a solution for you: http://forum.micasaverde.com/index.php/topic,31128.0.html
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: cranb on October 02, 2018, 02:33:31 am
Indeed i was thinking of a structure in where site-sensor is the mother and the child-devices are temperature, humidity, etc. A bit like the current WU-ground plugin where the child devices get their data from the plugin.

I am going to have a further look at the panel plugin. Already have it installed to control my IR TV through broadlink but did not think of it in this case.

Thanks for the tips and the work on your great plugins!!
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: phoenixaus on January 22, 2019, 08:15:30 am
Hi,

Sorry for a Newbie question, but I am trying to achieve exactly what the subject of this thread is!!

I'm using SiteSensor to retrieve a temperature reading from my local Aircon system. I have configured the device correctly, as I am getting the measured Temp reading coming through correctly.

However, I'm trying to push the value via a scene running to a virtual Temp Sensor, but I can't seem to get it to work.

Any ideas?


Code: [Select]
local bedtemp=luup.variable_get("urn:toggledbits-com:serviceId:SiteSensor1","Value1",78)
luup.variable_set("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", bedtemp,85)
return true

78 is the device id for the sitesensor
85 is the device id for the temp sensor.

85 has a variable called CurrentTemperature , so I thought this would work. When I run the scene it says success, but then Vera gives me the error in Lua scenes.
Any hints and tips greatly appreciated!
Steve
Title: Re: need help with parsing JSON data from HTTP and store variables to virtual device
Post by: cranb on February 11, 2019, 06:18:55 am
Hi,

Do you know the Virtual Sensor plugin from rigpapa? The last update of the plugin allows you to create a virtual sensor that can get data from another source.

I use it in combination with Sitesensor to get data from openweathermap into a temp and humidity sensor. Works like a charm!