We have moved at community.getvera.com

Author Topic: How best to handle HTTP posts - not a standard IO method  (Read 23990 times)

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
How best to handle HTTP posts - not a standard IO method
« on: June 24, 2012, 06:48:17 pm »
Hello,

I'm writing a plugin to control the Yamaha RX-A1000 receiver over IP. Unlike the Onkyo and Denon receivers which already have plugins and basically open an IO port and read / write serial commands, the Yamaha receivers use an HTTP POST request containing a body of XML data. And here in lies the problem - unlike the standard IO methods where the Action sends the request and Incoming handles the response, the request and response occur in the Action.

I've been trying to wrap my brain around the best method for handling this case for the last couple weeks. I've reviewed many of the published plugins for possible solutions, but I've come up empty. Do any coders have any suggestions how to / where best handle the responses?

Here's an example to illustrate using the commands to power ON the Main Zone.

Request body:
Code: [Select]
<YAMAHA_AV cmd="PUT">
<Main_Zone>
<Power_Control>
<Power>On</Power>
</Power_Control>
</Main_Zone>
</YAMAHA_AV>

Response body:
Code: [Select]
<YAMAHA_AV rsp="PUT">
<Main_Zone>
<Power_Control>
<Power></Power>
</Power_Control>
</Main_Zone>
</YAMAHA_AV>

The response just acknowledges that request was valid and properly formatted (an HTTP error code would be returned if it wasn't valid). To actually find out the state of the power, the following request is required:

Code: [Select]
<YAMAHA_AV cmd="GET">
<Main_Zone>
<Power_Control>
<Power>GetParam</Power>
</Power_Control>
</Main_Zone>
</YAMAHA_AV>

And the response:
Code: [Select]
<YAMAHA_AV rsp="GET">
<Main_Zone>
<Power_Control>
<Power>On</Power>
</Power_Control>
</Main_Zone>
</YAMAHA_AV>

So, two HTTP POST requests are required to turn the power on and verify it is completed. Now, unlike some serial protocols that may take time to respond, the response from the receiver is pretty much instantaneous, so while not ideal according to the wiki, the responses could be completely handled in the Action code with little performance impact.

Finally, the last item I'm considering is checking the status before sending the command. For example, if the ON action is issued when the receiver is already ON, an onscreen display shows on the TV. A simple check as to the status first, would prevent unnecessary commands from being sent (and the onscreen notifications). Is it best to just add another HTTP request, or should there be a service written that maintains the system status and is queried instead?

Here's the power ON Code:
Code: [Select]
        <action>
            <serviceId>urn:micasaverde-com:serviceId:DiscretePower1</serviceId>
            <name>On</name>
            <run>
                -- Power on
action = {"PUT", "Main_Zone", "Power_Control", "Power", "On"}
                sendCommand(action)
            </run>
        </action>

And the functions:
Code: [Select]
local ipAddress
local YAMAHACTRLURL = "/YamahaRemoteControl/ctrl"
local http = require("socket.http")

function log(message)
luup.log("YamahaReceiver: " ..message)
end

function createRequest(getORput, zone, area, func, cmd)
local body = '&lt;YAMAHA_AV cmd="'..getORput..'"&gt;&lt;'..zone..'&gt;&lt;'..area..'&gt;&lt;'..func..'&gt;'..cmd..'&lt;/'..func..'&gt;&lt;/'..area..'&gt;&lt;/'..zone..'&gt;&lt;/YAMAHA_AV&gt;'
return body
end
function sendCommand(action)
luup.log("Yamaha Send Command")

http.TIMEOUT = 5

local requestBody = createRequest(action[1], action[2], action[3], action[4], action[5])

log("Body:"..requestBody)

address = "http://"..ipAddress..YAMAHACTRLURL

result, status = http.request(address, requestBody)

log("Status"..status)
log("Result"..result)

if (status == 200) then
return true
end

return false
end

Given all that, any recommendations?

Thanks!



Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: How best to handle HTTP posts - not a standard IO method
« Reply #1 on: June 24, 2012, 09:25:03 pm »
There's no real problem with using HTTP directly in the ACTION, or a function called from the ACTION.  There are definitely cases you need to do that instead of using the <INCOMING> Blocks.

You do need to a be little careful about timeouts, which you've got hooks for, since Vera will restart if any ACTION handler fails to respond in a period of time (<30 seconds in UI4, <60 seconds in UI5)... but you're on track with what you have there.

The downside of the Yamaha style is that you'd need to Poll in order to get accurate status, and there'll be a long list of things that need to be polled in order to represent "current state" or have any type of accurate events off your device (such as On/Off, or Volume Up/Down) that people might scene-subscribe to.

Yamaha does expose a MCAST version of some of this data as change events, and it would be handy to see if the built-in IO/Incoming mechanism can be setup to use that MCAST IP/Port combination, and get that data out-of-band (I doubt it, but anything is possible)

Other than that, you're taking the correct track.  You can parse the XML responses with a full-blown Lua XML parser if you want, but if they're a simple pattern it may be possible to use Lua string/pattern matches instead.  Some of this logic (for fixed XML responses) in the Sonos Plugin:
    http://code.mios.com/trac/mios_sonos-wireless-music-systems/browser/trunk/L_Sonos1.lua

I'd also strongly recommend you move to using a Library file (L_*.lua) if you're working a lot with XML (etc), since it'll avoid all the escaping. The Sonos Plugin does this and may be a reasonable template to clone.

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #2 on: June 25, 2012, 02:03:13 am »
Hi guessed,

Thanks for validating I'm on the right track.

Quote
The downside of the Yamaha style is that you'd need to Poll in order to get accurate status, and there'll be a long list of things that need to be polled in order to represent "current state" or have any type of accurate events off your device (such as On/Off, or Volume Up/Down) that people might scene-subscribe to.

There is a short status query that returns a lot of this information in one large response. It would just be a matter of parsing it with the XML parser.

Quote
Yamaha does expose a MCAST version of some of this data as change events, and it would be handy to see if the built-in IO/Incoming mechanism can be setup to use that MCAST IP/Port combination, and get that data out-of-band (I doubt it, but anything is possible)

I googled to find out more about the MCAST but wasn't able to find anything. If you have somewhere to point me that would be appreciated.

In my searching, I came across some more documentation that I hadn't found before (I had done my major searching over a year ago). The method I'm using in this implementation is the Yamaha Network Command (YNC) protocol. Turns out there is a Yamaha Network Command Alias (YNCA) protocol, which is similar to the Denon and Onyko protocols, where the RS-232 and IP commands are the same and uses a dedicated port. Using the YNCA method, provided you keep the TCP/IP connection open (by a query every 30 seconds), certain receiver functions will provide the status change information (which could be processed by the INCOMING blocks).

It could be possible to send commands using the YNC and have a process that receives status changes using the YNCA protocol. Keeping the connection open in order to receive status change information is the tricky part. If you have to do a query every 30 seconds (not sure how to do this yet), it may be just as easy to do the status query with the YNC protocol as mentioned above every 30 seconds. Using two protocols could get messy...

Quote
I'd also strongly recommend you move to using a Library file (L_*.lua) if you're working a lot with XML (etc), since it'll avoid all the escaping. The Sonos Plugin does this and may be a reasonable template to clone.

I definitely plan on doing this. The escaping was driving me nuts!


Offline baxy_AU

  • Sr. Member
  • ****
  • Posts: 269
  • Karma: +5/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #3 on: June 25, 2012, 02:33:40 am »
I was just going to mention the YNCA protocol...I have been hoping for a yamaha plugin for awhile and finally resolved into looking into it myself and come across this in the weekend, but I have zero coding experience. I managed to get as far as opening a telnet connection to my receiver and issue a command and get a number of status responses but the connection times out after around 30sec as you mentioned.
Why could you not just use YNCA to issue commands as well as receive status?
« Last Edit: June 25, 2012, 02:37:11 am by baxy_AU »

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #4 on: June 25, 2012, 08:40:42 am »
Why could you not just use YNCA to issue commands as well as receive status?

I've been thinking about this, and I've still go some more to do before I continue. My initial thoughts were the YNC is more powerful, and appears to be more universal, whereas the YNCA is more model specific. As well, there is a lot more information available using YNC then YNCA. If I'm going spend the time, I might as well do it on the "fuller" protocol. Plus, I've already started it.

Just my thoughts so far - I'm open to suggestions.

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: How best to handle HTTP posts - not a standard IO method
« Reply #5 on: June 25, 2012, 09:26:59 am »
For the event notification stuff, it's on the Event notification tab of the following document:
    http://webpix.ca/files/RX-V3900_Z7_ETHERNET_IF_Spec_e_1.0.xls

Not sure if it applies to all models, but I found this while I was hunting for the API references.  It'll tell you when something has changed, and what attributes those were, and you call back to find the latest values.

If you decide to cutover to YNCA it's fairly simple to use a timer (see luup.call_timer) to do something periodically like the poll that's needed.  Word of warning for UI4 though, this combination (timer + IO + user Action calls, or any two of these) can be fatal... Causing Vera to restart if they happen "at the same time".  This was fixed in UI5, but UI4 is wholly broken in this respect.  In some cases, it can even cause data corruption.

The weather plugin does it if you wanted sample code.  There are a bunch of others also.  Vera itself will keep the connection open, and will establish new connections if the client gracefully drops the connection, when you use the standard IO framework.

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #6 on: June 26, 2012, 12:05:10 am »
Thanks for the document, guessed. It's doesn't provide all the information needed, but that didn't stop me. :)

I fired up WireShark to see what the receiver was multicasting on the network. I was receiving packets, but only SSDP packets advertising the different services available. None of my power, input selection, or volume change events were being broadcasted. Not ready to give up, I dug through these packets captured and found a url to call to get the unit description. From this description I was able figure out the command that should turn on Event Notifications (which I have yet to find in any documentation). Using this new information, I tired to turn Event Notifications on, but it wasn't taking. I finally realized it was giving me an error code because the receiver was power off (D'oh!).

After correcting the error, I was able to successfully turn on Event Notifications. So despite having incomplete documentation, I got the Event Notifications multicasting from the receiver. I never found this option in the on-screen menus, which leads me to believe this option can only be set by using the YNC protocol.

I now just have to test out receiving and processing these notifications using the INCOMING block. Once that's done, I believe I have everything I need to continue developing the plug-in using the YNC protocol.

Thanks for all your input!
« Last Edit: June 26, 2012, 12:06:58 am by dgdev »

Offline guessed

  • Community Beta
  • Master Member
  • ******
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: How best to handle HTTP posts - not a standard IO method
« Reply #7 on: June 26, 2012, 12:28:56 am »
Cool, sounds like you've had some good findings.  I'll be interested if V can attach to the MCast stream.  Some of the interactions you're describing Lund more like the UPnP event notification stream, btw.  If that's what you're doing, you'd need to look at the UPnP specs.

Gotta love Wireshark (and raw tcpdump, for that matter).  Swiss army knife of network/app diagnostics....

Offline baxy_AU

  • Sr. Member
  • ****
  • Posts: 269
  • Karma: +5/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #8 on: June 26, 2012, 04:19:23 am »
From what i read YNCA should be easier to support models without ethernet as the same protocol is used for serial connections as well..and the availible command set is more than adequate for anything i would want to control from my vera.

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #9 on: June 30, 2012, 03:26:15 pm »
From what i read YNCA should be easier to support models without ethernet as the same protocol is used for serial connections as well..and the availible command set is more than adequate for anything i would want to control from my vera.

baxy_AU, you win. Despite all the pieces I was able to dig up and put together, I've reached a major roadblock that I'm not prepared to even attempt to overcome. As has been outlined in numerous different threads, there is no easy way to generate a TCP listener on a defined port. This is a roadblock in terms of being able to receive the broadcasted state change messages.

So, it looks like I'll be implementing the YNCA protocol for both IP and Serial Communication.

To wrap this thread up, and answer what guessed was suggesting was UPnP traffic from the receiver, yes it is. The RX-A1000 broadcasts the following services:
AVTransport
ConnectionManager
RenderingControl

My understanding is theses are useful for media playback by the receiver, but have no ability to control basic functions like power, input selection, etc.

Thanks again to everyone for your input.

Offline baxy_AU

  • Sr. Member
  • ****
  • Posts: 269
  • Karma: +5/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #10 on: July 01, 2012, 06:20:09 am »
I wasn't trying to win you round...just offering a (semi) informed opinion  ;-)
Btw I see that someone else released a yamaha receiver plugin today..I'm not sure what method they implemented..it looks to be for serial only receivers?

Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #11 on: July 01, 2012, 01:27:50 pm »
I wasn't trying to win you round...just offering a (semi) informed opinion  ;-)
:)

Btw I see that someone else released a yamaha receiver plugin today..I'm not sure what method they implemented..it looks to be for serial only receivers?

I took a look at this plug-in: wow, it's extensive! I planned to release the basic functions, that is almost everything and an interface as well. Looking at the underlying code, they are using serial commands. I looked at my reference material, but couldn't couldn't match it to anything. Doesn't look like it will work with the RX-A series.

The Yamaha Network Command (YNC) protocol uses the http posts of XML data. I've only found the manual for the RX-V3900 / Z7, but this information / protocol is for the most part valid for the RX-A1000.  The RS-232 information in this manual does not match what I'm seeing in this plugin.

The Yamaha Network Command Alias (YNCA) protocol, which is RS-232 or IP compatible, I found the manual which says it is valid for RX-A700, RX-A800, RX-A1000, RX-A1000, RX-A2000, RX-A3000, and RX-V867. The YNCA does not match the RS-232 protocol for the RX-V3900 / Z7. I'm part way through the basics of getting this setup, but seeing this new plug-in presents some interesting ideas.

It seems like each series of receivers has a little different protocol. Oh Yamaha, why couldn't you just have a standard protocol?  :P

baxy_AU, from your previous posts, I assumed your RX-A is serially connected. If so, can you try this plug-in? Maybe it does work on the RX-A and I'm just missing that protocol information...


Offline dgdev

  • Sr. Newbie
  • *
  • Posts: 24
  • Karma: +0/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #12 on: July 01, 2012, 04:05:35 pm »
baxy_AU, from your previous posts, I assumed your RX-A is serially connected. If so, can you try this plug-in? Maybe it does work on the RX-A and I'm just missing that protocol information...
Opps...this would require a modification to the plugin to allow it to do serial connections. Never mind.

Offline baxy_AU

  • Sr. Member
  • ****
  • Posts: 269
  • Karma: +5/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #13 on: July 01, 2012, 05:11:05 pm »
Unfortunatly mine is ethernet connected and I'm currently away from home for work for the next 3weeks so I can't help much sorry.
« Last Edit: July 01, 2012, 05:17:20 pm by baxy_AU »

Offline baxy_AU

  • Sr. Member
  • ****
  • Posts: 269
  • Karma: +5/-0
Re: How best to handle HTTP posts - not a standard IO method
« Reply #14 on: July 01, 2012, 05:13:04 pm »
Btw the model number of my receiver is RX V671.