We have moved at community.getvera.com

Author Topic: Unable to access https API  (Read 546 times)

Offline jwwilhelm

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-4
Unable to access https API
« on: January 04, 2019, 10:20:08 pm »
I have used Lua scripting and the socket.http module to access current weather conditions from the NOAA website.  Last month, NOAA shutdown the legacy website and has launched a new API that requires https as well as a user-agent header.  Unfortunately socket.http does not support https (just coerces url to http).  I have come across the following examples of ways to make https requests from Lua but have not been successful for the reasons indicated for each.  To simplify the testing, I am just testing them here against a popular test page.

This one works, but I cannot add a custom user-agent header using this format.

local curl, response, http = luup.inet.request( "https://httpbin.org/get", nil, nil, nil,nil )
luup.log("status=".. http)
luup.log("response= ".. response)

This format allows a custom user-agent header, but does not work.

local curl, response, http = luup.inet.request{
    url = "https://httpbin.org/get"
}
luup.log("status=".. http)
luup.log("response= ".. response)

Same for this one.
local http = require("ssl.https")
local res, code, headers, status = http.request{
url="https://httpbin.org/get"
}
luup.log("status=".. status)
luup.log("response= ".. res)

Any idea what I might be doing wrong such that these approaches aren't working?  Any other approaches that I should be using instead?

Offline rafale77

  • Community Beta
  • Hero Member
  • ******
  • Posts: 1749
  • Karma: +101/-27
  • HA ≠ IoT as a blue sky is cloudless.
Re: Unable to access https API
« Reply #1 on: January 04, 2019, 11:36:14 pm »
Without knowing what you are running this on, it is difficult to know. I suspect that you are running a very old vera and firmware which does not support the SSL version required for that website. I have had similar problems initially on openLuup and had to upgrade the lua socket.
openLuup (79 devices, 141 scenes, 19 apps) master to VeraPlus (142 zwave nodes, 8 Zigbee nodes, 221 devices,  20 scenes , 2 apps) +  Hubitat (15 Zigbee nodes) + Home-Assistant (API Integrations). Bridged to Siri and Alexa. Homewave. VeraPlus ExtRooted and mios server independent.

Offline jwwilhelm

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-4
Re: Unable to access https API
« Reply #2 on: January 05, 2019, 12:02:32 pm »
I'm running this on a Vera Edge with firmware version 1.7.4.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: Unable to access https API
« Reply #3 on: January 05, 2019, 12:18:57 pm »
You need to use the LuaSec library, which should already be installed.

It has the same request API as the http library, but uses https.

Code: [Select]
local https = require "ssl.https"

local status = https.request (...)

Even using an HTTPS url, the normal library does not use SSL.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jwwilhelm

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-4
Re: Unable to access https API
« Reply #4 on: January 05, 2019, 03:39:41 pm »
Thanks for confirming I'm heading the right direction.  I believe that library you are recommending is the same library as the third option I tried above, which does not work ("failed to test code" error in Vera luup test).  I believe I need to use the general form, rather than the simple form you show in your suggestion, for the NOAA site so that I can force a user-agent header.  Am I messing up the syntax somehow?

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: Unable to access https API
« Reply #5 on: January 05, 2019, 04:01:21 pm »
Thanks for confirming I'm heading the right direction.  I believe that library you are recommending is the same library as the third option I tried above, which does not work ("failed to test code" error in Vera luup test).  I believe I need to use the general form, rather than the simple form you show in your suggestion, for the NOAA site so that I can force a user-agent header.  Am I messing up the syntax somehow?

The reason your code isn't working is that the return values can be nil, and in your log calls you are trying to concatenate them as strings. You can't concatenate nil, and you need to guard against it. Make sure you are looking in the Luup log file. This kind of error will be exposed there.

That said, there's something odd going on with that site that is making https return those nil values. If I change the URL to my own web site, it works fine. Here's your code, modified. Works on my sandbox Vera3:
Code: [Select]
local http = require("ssl.https")
local res, code, headers, status = http.request{
url="https://www.toggledbits.com/reactor"
}
luup.log("status=".. tostring(status)) -- use tostring() to guard nil
luup.log("response= ".. ( res or "nothing")) -- use logical operator to guard nil

This also requires a bit more work to get the actual response...

Code: [Select]
local http = require("ssl.https")
local ltn12 = require "ltn12"
r = {} -- init empty table
local res, code, headers, status = http.request{
url="https://www.toggledbits.com/reactor",
sink = ltn12.sink.table( r )
}
luup.log("status=".. tostring(status))
luup.log("response= ".. table.concat( r, "" ) )
« Last Edit: January 05, 2019, 04:05:22 pm by rigpapa »
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline jwwilhelm

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-4
Re: Unable to access https API
« Reply #6 on: January 06, 2019, 09:42:08 pm »
Fantastic.  It works and I don't even need to provide a special user-agent header.  Thanks for the help all and especially rigpapa. 

In case others are interested, I have posted the basic request code here that returns a JSON object containing all of the current weather conditions.  This may be a good replacement for the the WU plugin now that WU is discontinuing their free API.  In my case, this comes from MSP airport.  You can look up your nearest weather station using a different endpoint in the same API.  https://forecast-v3.weather.gov/documentation

Code: [Select]
local http = require("ssl.https")
local ltn12 = require "ltn12"
r = {} -- init empty table
local res, code, headers, status = http.request{
url="https://api.weather.gov/stations/KMSP/observations/current",
sink = ltn12.sink.table( r )
}
luup.log("status=".. tostring(status))
luup.log("response= ".. table.concat( r, "" ) )

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: Unable to access https API
« Reply #7 on: January 06, 2019, 10:00:25 pm »
For those of you that don't want to do Lua, the SiteSensor plugin will make requests from an API that returns JSON, and you can pick and store elements of the response in state variables on the device.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.