We have moved at community.getvera.com

Author Topic: lu_block_jobs  (Read 1158 times)

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #15 on: January 25, 2019, 07:22:59 am »
This is for a luup.io.open() issued on from the parent with the child device number?

Yes, and same for IP assigned as attribute to the child device.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: lu_block_jobs
« Reply #16 on: January 25, 2019, 09:15:23 am »
Hi,

Got some testing done and basically I get the following on the intercept function. It basically seems to work as documented on Vera. That is:
 1) call intercept
 2) call write
 3) call read

What does not make a difference is to call intercept on just receiving. In my example I read the WebSocket protocol that is a few header bytes including the payload length and then that payload. So I tried to have the incoming handler process the header, and then use luup.io.read to read the payload. That basically does not work, and adding intercept makes no difference. I have to use the incoming handler to process the payload bytes.

The luup.io.is_connected is no good indication for an open connection. After the peer closes the socket from their end, is_connected still returns true. In my case the peer starts replying with an HTTP 500 message, so I could use that to restore the connection.

Next test is openLuup.

Cheers Rene

2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #17 on: January 25, 2019, 10:30:33 am »
Hi,

Got some testing done and basically I get the following on the intercept function. It basically seems to work as documented on Vera. That is:
 1) call intercept
 2) call write
 3) call read

What does not make a difference is to call intercept on just receiving. In my example I read the WebSocket protocol that is a few header bytes including the payload length and then that payload. So I tried to have the incoming handler process the header, and then use luup.io.read to read the payload. That basically does not work, and adding intercept makes no difference. I have to use the incoming handler to process the payload bytes.

The luup.io.is_connected is no good indication for an open connection. After the peer closes the socket from their end, is_connected still returns true. In my case the peer starts replying with an HTTP 500 message, so I could use that to restore the connection.

Next test is openLuup.

Cheers Rene

Rene, my testing (also doing WebSocket) pretty much concurs. Trying identical things (using io.read() to get the payload data once I have the length) came to similar conclusions: switching to luup.io.read() for the payload yielded no benefit performance wise. I was able to get it to work using intercept() first, and noted that if I didn't call intercept() first, a deadlock often resulted. IMPORTANT: this is specifically from within the <incoming> handler. In other code, skipping the intercept() call does not result in a io.read() hang, you just run the risk of missing data.

That said, I have successfully used luup.io.read(), intercept() and <incoming> together, like this (in "raw" mode) to handle getting the connection upgraded to WebSocket:

1) Call luup.io.intercept();
2) Use luup.io.write() to send the HTTP request with headers to upgrade to Websocket;
3) Use luup.io.read() to gather response data, scanning for end of headers (consecutive cr/lf/crlf);
4) Celebrate successful conversion to WebSocket;
5) Send a WebSocket request (no-op is fine); this is necessary to reset the effect of the previous intercept();
6) Use <incoming> from that point forward.

I imagine this is similar to what you are doing.

But have you noticed the performance? Really, really poor. Appalling, actually. I removed all processing and just counted bytes and returned immediately from my incoming handler, trying to see how fast it would pass a 32KB JSON from the WebSocket endpoint, and the best I could do (on my Vera3) was just shy of 1800 bytes per second (18 seconds for the 32KB response--awful, just terrible). Receiving one byte at a time is just way too much overhead. I've already sent a note to Sorin requesting a "rawblock" protocol to return the entirety of a larger socket.receive() (e.g. 1024 bytes, connection MTU, or something even larger). But of course, that will be far off, if they agree to do it at all. For my application, as it is today it's a non-starter, as it takes 15-20 seconds to receive the average payload, but updates come every 5 seconds or less on an active endpoint--it can't keep up, not even close.

@akbooer, it would be great for openLuup if we could set a switch somewhere (openluup.longrawread = true?) to have the raw protocol return sock:recv(1024) or larger as described. At least then, performance on openLuup would shine over Vera (again).

Note: I also can't stress enough that, as a best practice, string concatenation should not be used to reassemble the packet in an incoming handler. String concats in Lua are slowwww, and get worse as the string grows. It's much faster to insert the data into a table, and then concat the table to a string when you have the entire payload.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: lu_block_jobs
« Reply #18 on: January 25, 2019, 11:08:13 am »
But have you noticed the performance? Really, really poor... Receiving one byte at a time is just way too much overhead.

This, of course, is why the <crlf> protocol is good, receiving a whole line at a time.  Unfortunately, you don't have that option for a binary stream.

Quote
@akbooer, it would be great for openLuup if we could set a switch somewhere (openluup.longrawread = true?) to have the raw protocol return sock:recv(1024) or larger as described. At least then, performance on openLuup would shine over Vera (again).

So what would terminate the read in the event of a shorter stream?

Quote
Note: I also can't stress enough that, as a best practice, string concatenation should not be used to reassemble the packet in an incoming handler. String concats in Lua are slowwww, and get worse as the string grows. It's much faster to insert the data into a table, and then concat the table to a string when you have the entire payload.

Amen to that!

My preference is to repackage some of the asynchronous I/O which is implemented in openLuup for servers, which quite happily handle incoming callbacks to jobs.  At the moment, this only works for TCP connections initiated by an incoming server socket, but there's no reason they couldn't apply to a specifically opened socket.  Currently, also, the callbacks are to a regular function, rather than code in an <incoming> block since this is much easier to set up.  The called routine receives a proxy socket as an argument to do with what it will.  Special handling ensures that the socket is correctly closed in the event of an error or a timeout and removed from the scheduler's socket watch list.

I think this would be far less frustrating rather than trying to invent a way that Vera/MiOS I/O should or might work in the future.  But you would, of course, be stuck with something that is openLuup-specific.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #19 on: January 25, 2019, 12:03:24 pm »
@akbooer, it would be great for openLuup if we could set a switch somewhere (openluup.longrawread = true?) to have the raw protocol return sock:recv(1024) or larger as described. At least then, performance on openLuup would shine over Vera (again).

So what would terminate the read in the event of a shorter stream?


A non-blocking, no-timeout socket:receive( 1024 ) will return up to 1024 bytes, shorter if less data is pending on the socket:

Code: [Select]
sock:settimeout(0)
local data, err, partial = sock:receive( 1024 )
if data == nil then
   -- if partial not empty ("") dispatch to handler
   -- if err ~= "timeout" then handle error; for timeout, go do something else and come back later?
else
   -- dispatch data to incoming handler
end

EDIT: Fixed code. Sorry, rushing around today... third return parameter to receive is partial data if the timeout occurs, which it will almost immediately. Dispatch that to the incoming handler.
« Last Edit: January 28, 2019, 09:04:32 am by rigpapa »
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: lu_block_jobs
« Reply #20 on: January 25, 2019, 12:19:42 pm »
...so the easiest thing is simply to describe a new protocol option raw_1024 ?

__________

Edit: ah... I think that is what you were suggesting earlier.
« Last Edit: January 25, 2019, 12:21:21 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #21 on: January 25, 2019, 12:23:01 pm »
...so the easiest thing is simply to describe a new protocol option raw_1024 ?

Two reasons I would not do it this way:

1) It requires a change in the device file to use the different protocol, and thus you need different device files for Vera vs openLuup? Isn't that a first?

2) Vera owns that "namespace", and if it were me, I'd want to avoid potential conflicts in the (admittedly unlikely) event they someday create a "raw_1024" with different semantics. More principle than likely conflict, I'll admit.

Edit for clarity: I am recommending VERA add a new protocol. Until that's done, and you can know what it is and its particular implementation semantics (if they ever do it), I am recommending to you just a flag in openLuup to change the behavior of "raw" so that it can do either: the Vera way of returning 1 byte, or the openLuup way of returning a larger block. Then when Vera does whatever it may do, yours still works, and you can implement theirs when you get around to it.
« Last Edit: January 25, 2019, 12:28:58 pm by rigpapa »
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: lu_block_jobs
« Reply #22 on: January 25, 2019, 12:40:30 pm »
Hi,

@Patrick, yes my logic is just like yours.

It was my experience that it is slow indeed. On a Vera Lite it takes about 20 seconds to get 75K of data. It is one I do not need often, but still. With native sockets it is 3-4 seconds. Most replies from the Harmony are well below 1KB and take a 1 sec max. I guess I can get it to work, but I then have to make some sending queue as well so a reply is handled before a next message gets send. The Harmony also sends status messages when it feels to (without doing a write first) I need to process.

The connection is now kept alive by sending a WS ping, so as long that works it is ok. However, when it stops and I try again to connect internal errors from LuaInterface.cpp start flying around. Same happens when trying a new WS handshake by intercept, write,read. Both are a sure way to trigger luup reloads.

For now I'm going to stick to working with sockets directly. This is too shaky for my taste.

For improved handling I would first ask for a functioning is_connected and a close and (re)connect possibility without taking luup down (thats an ask from Vera, not you AK). Also to be able to read specified number of bytes on the raw protocol would help, i.e. an extra parameter. Many needing raw will likely have a similar message format as websockets so you know what to look for.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: lu_block_jobs
« Reply #23 on: January 25, 2019, 01:54:39 pm »
BTW,

Just tested handling that large request with intercept and using luup.io.read. Five minutes !!!! to get that same 75KB. And yes, exact same logic as used in the incoming, just wrapped in a repeat and a luup.io.read to get each of those characters one by one.  So that's 10 time slower than incoming and utterly useless.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline HSD99

  • Sr. Member
  • ****
  • Posts: 384
  • Karma: +20/-0
Re: lu_block_jobs
« Reply #24 on: January 25, 2019, 03:11:41 pm »
Quote
My bigger-scope prediction is that the bulk of IoT devices are going to be WiFi, and Z-Wave and its ilk will slowly be relegated to a very specific subset of devices. Time to market and cost will be drivers away from protocols like Z-Wave, not to mention the ubiquity of WiFi. So, for that to be at all sane, Vera needs to get its act together on sockets. LuaSocket under Luup doesn't cut it, and luup.io isn't showing well either. This all needs fresh eyes on it.

I agree 100% with the above. I did a high-end residential wireless control system in the late 1990's (pre-Z-Wave, Zigbee, etc.) in the 900 MHz range with wallbox dimmers, switches and plug-in modules. A tiny cheap ~ 900 Mhz radio was all you could do in 1998---the idea of putting an 802.11b interface with software stack into a small device was simply not in the cards.

With HaLow (https://en.wikipedia.org/wiki/IEEE_802.11ah) coming along and cheap, low-power 802.11g/n interfaces integrated into small MCUs, Z-Wave, Zigbee, and Bluetooth et al will fade away from the HA and IoT marketplace.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: lu_block_jobs
« Reply #25 on: January 26, 2019, 06:29:13 pm »
On a similar note to the discussion of the <incoming> block, would it be possible to include a timeout tag of some sort in the device file.  Perhaps       <device><olTimeout>15</olTimeout></device> That way you wouldn't have to modify io.open to accept additional parameters. This would apply to  <incoming> data only and not any specific calls to io.open for files and what not. And if you allowed the overall luup.io timeout to be set in an exposed variable, that would take care of general need based on one's local network/machine response et al. 

I don't know enough about the UPnP spec to know if this would be disruptive to a plugin running on a vera-- in the sense of vera crashing because of an unrecognized UPnP variable.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #26 on: January 26, 2019, 07:01:05 pm »
On a similar note to the discussion of the <incoming> block, would it be possible to include a timeout tag of some sort in the device file.  Perhaps       <device><olTimeout>15</olTimeout></device> That way you wouldn't have to modify io.open to accept additional parameters. This would apply to  <incoming> data only and not any specific calls to io.open for files and what not. And if you allowed the overall luup.io timeout to be set in an exposed variable, that would take care of general need based on one's local network/machine response et al. 

I don't know enough about the UPnP spec to know if this would be disruptive to a plugin running on a vera-- in the sense of vera crashing because of an unrecognized UPnP variable.

If you do your read/<incoming> in a <job>, the 5,X return from the job lets you specify the timeout (the X return parameter). This seems to work as advertised on Vera, IIRC, although I've had no luck doing something responsive when the timeout occurs. There is a <timeout> tag, which supposedly may contain Lua to execute when the timeout occurs, but I don't think that worked as planned when I last tried, and I could have just brain-faded it. I need to go back and play with it; that issue wasn't my focus at the time. The docs are just so vague and fragmented in this area. Ref: http://wiki.micasaverde.com/index.php/Luup_Declarations#.3Ctimeout.3E
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline Buxton

  • Full Member
  • ***
  • Posts: 209
  • Karma: +12/-0
Re: lu_block_jobs
« Reply #27 on: January 26, 2019, 07:13:14 pm »
Ok, but assuming <incoming> is first handled/started in the vera with something similar to io.open (using the socket library), is there a way to set the timeout for opening the socket in the device file?  This seems to be where the plugin I'm using stalls.  Once the socket is open, data flows as predicted, so I don't need to mess with the read timeout as that is handled by the plugin itself in the way you outlined.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: lu_block_jobs
« Reply #28 on: January 26, 2019, 07:48:59 pm »
Ok, but assuming <incoming> is first handled/started in the vera with something similar to io.open (using the socket library), is there a way to set the timeout for opening the socket in the device file?  This seems to be where the plugin I'm using stalls.  Once the socket is open, data flows as predicted, so I don't need to mess with the read timeout as that is handled by the plugin itself in the way you outlined.

Yeah, no idea there. AFAIK, Vera just keeps on trying to open the socket until it succeeds. Even luup.io.open() doesn't have a timeout parameter, and it just returns and the socket is opened asynchronously (ouch). The docs basically say you need to write to it to see if the socket is open (fails if not). "...the actual opening of the socket occurs asynchronously and this function returns nothing. You will know that the socket opening failed if your subsequent call to write fails." SMH  :o
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.