Author Topic: Latest release ControlTimeout Too long  (Read 362 times)

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Latest release ControlTimeout Too long
« on: September 29, 2018, 12:02:52 pm »
@amg0

In the latest release somehow the variable ControlTimeout became too long. After installation mine became 60000 (60s) and was causing openLuup to intermittently freeze (API and API unresponsive) for about 1min. Remote connection in that case would also be not functional. I had to bring it down to 1000 to eliminate the issue.

I am not sure under what situation a value greater than 5s would be helpful.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3090
  • Karma: +206/-8
Re: Latest release ControlTimeout Too long
« Reply #1 on: September 29, 2018, 12:55:43 pm »
@amg0

In the latest release somehow the variable ControlTimeout became too long. After installation mine became 60000 (60s) and was causing openLuup to intermittently freeze (API and API unresponsive) for about 1min. Remote connection in that case would also be not functional. I had to bring it down to 1000 to eliminate the issue.

I am not sure under what situation a value greater than 5s would be helpful.
For remote access to vera it is needed , at least for me
You should be able to change that default value

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Re: Latest release ControlTimeout Too long
« Reply #2 on: September 30, 2018, 02:19:11 am »
Yes I did so. I can confirm that this value was the reason for my openLuup to be temporarily unresponsive. I have not had a problem with it since. openLuup is single threaded and I guess the long timeout caused it to wait for something? I have a couple of bridges polling the openLuup instance.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3090
  • Karma: +206/-8
Re: Latest release ControlTimeout Too long
« Reply #3 on: September 30, 2018, 03:31:25 am »
Yes I did so. I can confirm that this value was the reason for my openLuup to be temporarily unresponsive. I have not had a problem with it since. openLuup is single threaded and I guess the long timeout caused it to wait for something? I have a couple of bridges polling the openLuup instance.
I find that hard to believe, it is a client side timer waiting in JS to receive the first user_data. There must be some environmental issue. Is open Luup up and running when you start?

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Re: Latest release ControlTimeout Too long
« Reply #4 on: September 30, 2018, 04:36:51 am »
Yes openluup is up and running. I noticed the problem after updating ALTUI. I have been running my own branch of openluup for a few months now. When I check the service, it is still up and running even though both the API and ALTUI don't respond... then I try to kill the service and it would make me wait for quite a long time before it closes (less than a min). I reduced the timeout to 1000ms and have not seen the problem for the past 2 days. It was at 10000ms before the ALTUI update.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3090
  • Karma: +206/-8
Re: Latest release ControlTimeout Too long
« Reply #5 on: September 30, 2018, 05:34:34 am »
Yes openluup is up and running. I noticed the problem after updating ALTUI. I have been running my own branch of openluup for a few months now. When I check the service, it is still up and running even though both the API and ALTUI don't respond... then I try to kill the service and it would make me wait for quite a long time before it closes (less than a min). I reduced the timeout to 1000ms and have not seen the problem for the past 2 days. It was at 10000ms before the ALTUI update.
Reducing the timer just make Altui stop waiting for an answer. You check the js console logs and please share your altui config esp around controller setup.
You probably want to debug this without altui, just with the direct url to get user_data and see why it is not responding fast enough

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Re: Latest release ControlTimeout Too long
« Reply #6 on: October 01, 2018, 03:01:45 am »
I actually think it is a problem with openLuup. When it stops responding, I don't even get the ALTUI page with the "waiting for initial data", I get no response at all. Homewave (which uses get user data) also gets no response from openLuup and apparently neither can systemd shut it down. I understand what you are saying and can't explain why it does what it does. In my case, most of why remote access work with a timeout of 3 seconds.
My ALTUI config is pretty vanilla, I have a pretty big startup Lua of 170ish lines including the attribute settings.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3090
  • Karma: +206/-8
Re: Latest release ControlTimeout Too long
« Reply #7 on: October 01, 2018, 03:25:02 am »
I actually think it is a problem with openLuup. When it stops responding, I don't even get the ALTUI page with the "waiting for initial data", I get no response at all. Homewave (which uses get user data) also gets no response from openLuup and apparently neither can systemd shut it down. I understand what you are saying and can't explain why it does what it does. In my case, most of why remote access work with a timeout of 3 seconds.
My ALTUI config is pretty vanilla, I have a pretty big startup Lua of 170ish lines including the attribute settings.
it works before the timeout fails but ALTUI continues so later on the call maybe successfull and you feel it works. you should see error in console log though.
I agree, it sounds like a openluup multiple entry / single thread problem

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Re: Latest release ControlTimeout Too long
« Reply #8 on: October 06, 2018, 01:30:47 pm »
I actually think it is a problem with openLuup. When it stops responding, I don't even get the ALTUI page with the "waiting for initial data", I get no response at all. Homewave (which uses get user data) also gets no response from openLuup and apparently neither can systemd shut it down. I understand what you are saying and can't explain why it does what it does. In my case, most of why remote access work with a timeout of 3 seconds.
My ALTUI config is pretty vanilla, I have a pretty big startup Lua of 170ish lines including the attribute settings.
it works before the timeout fails but ALTUI continues so later on the call maybe successfull and you feel it works. you should see error in console log though.
I agree, it sounds like a openluup multiple entry / single thread problem

I am not seeing an error in the console and I can now confirm that it is a significant problem on my openLuup installation which runs quite a few plugins pinging other APIs. It appears that openLuup just temporary freezes while waiting for this timeout to expire. Now that I reduced this timeout to 1s, the occasional missed updates from plugins like Vrainsensor, iPhonelocator, Ecobee or plantlink stopped. They work with a Luup.call_delay on a function which seem to have been missed while openLuup was temporarily frozen and therefore got themselves out of self updating.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline amg0

  • Moderator
  • Master Member
  • *****
  • Posts: 3090
  • Karma: +206/-8
Re: Latest release ControlTimeout Too long
« Reply #9 on: October 06, 2018, 04:30:31 pm »
I actually think it is a problem with openLuup. When it stops responding, I don't even get the ALTUI page with the "waiting for initial data", I get no response at all. Homewave (which uses get user data) also gets no response from openLuup and apparently neither can systemd shut it down. I understand what you are saying and can't explain why it does what it does. In my case, most of why remote access work with a timeout of 3 seconds.
My ALTUI config is pretty vanilla, I have a pretty big startup Lua of 170ish lines including the attribute settings.
Hoping Akboer can look at it
it works before the timeout fails but ALTUI continues so later on the call maybe successfull and you feel it works. you should see error in console log though.
I agree, it sounds like a openluup multiple entry / single thread problem

I am not seeing an error in the console and I can now confirm that it is a significant problem on my openLuup installation which runs quite a few plugins pinging other APIs. It appears that openLuup just temporary freezes while waiting for this timeout to expire. Now that I reduced this timeout to 1s, the occasional missed updates from plugins like Vrainsensor, iPhonelocator, Ecobee or plantlink stopped. They work with a Luup.call_delay on a function which seem to have been missed while openLuup was temporarily frozen and therefore got themselves out of self updating.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6203
  • Karma: +276/-70
  • "Less is more"
Re: Latest release ControlTimeout Too long
« Reply #10 on: October 06, 2018, 04:56:22 pm »
They work with a Luup.call_delay on a function which seem to have been missed while openLuup was temporarily frozen and therefore got themselves out of self updating.

If a process hogs the CPU beyond the time that a luup.call_delay() function should run, then it is run as soon as possible afterwards.  They are not missed.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1419
  • Karma: +80/-30
Re: Latest release ControlTimeout Too long
« Reply #11 on: October 08, 2018, 05:10:19 am »
Hi rafale77,

I have the time out set to 6 seconds because of remote access delays. I have not seen that lock up openLuup though. Just scanned the log file around the moment of logging in remotely and not seeing any pause in the log messages suggesting a pause.

I had one other scenario with an https request. The timeout in LUA does not seem to work in all cases and that would lockup or stall openLuup. Oddly only on one particular post request out of many to the same web service, sometimes after a day, sometimes not for two weeks, I never managed to figure out why as the https request simply would hang. I now redesigned to plugin and eliminated that particular request. Have you eliminated all those possible sources for lock ups or delays?

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1410
  • Karma: +71/-24
Re: Latest release ControlTimeout Too long
« Reply #12 on: October 09, 2018, 04:59:09 am »
Hi Rene,

Thank you for this insightful post. No I have not looked into other possibilities. I only noticed that I started getting frequent temporary lock ups since the last ALTUI release. I proceeded by elimination. Since I have not touched openLuup or any other plugins in a few months, it had to be something in ALTUI. The ControlTimeout when I looked at it was extremely long. (60s) I believe to have manually set it at 6 or 7s in the past but it's been incrementally extended over the past few months by updates. I admit to be running a lot of plugins which are making various API calls, including your Harmony with 3 remotes and fairly frequency. I too have observed what you are referring to but with the latest 60s release the lockups were daily. In any case, I have eliminated the problem by reducing the timeout back down to 1.2s. I am thinking that the lockups still occur but only last 1.2s at the most and then the backlog of actions continue as AK mentions. At 60s, it seems the backlog of code to run is too long and some things get dropped or crash out of sync.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (138 zwave nodes, 8 Zigbee nodes, 205 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!