Author Topic: openLuup - running unmodified plugins on any machine  (Read 100058 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6340
  • Karma: +288/-70
  • "Less is more"
openLuup - running unmodified plugins on any machine
« on: May 22, 2015, 01:31:03 pm »
This is my 2000th post, so I wanted to make it significant to me, if to nobody else!

I'm aware that this can be a pivotal time for some developers (three years ago @Ap15e made his 2000th post his last: My final post) and it's sadly no surprise to note the issues which drove him away:
  • "My personal ROI from MCV is just too low"
  • "resources visible to me are just too limited to create a sufficiently stable and bug free and documented product"
  • "There is no offical documentation for developers, and when MCV does document items, there is no guarantee that MiOS works as documented"
  • "Installing UI5 on my Vera V2 left the device in a totally unusable state: severe slowdowns, several lockups per week requiring a hard reboot, several soft reboots per day."
  • "New state of my plugins/scripts: EOL, unsupported."
Nowadays, read 'UI7' and 'Vera3/Lite'.  We've seen others remove their apps from the forum too. The direction of developer drift now seems to be OpenHAB... but I'm loathe to abandon my investment in MiOS, and Lua, so I agree with this comment:
Quote from: Ap15e
IMHO, MiOS' architecture is quite excellent (leaving aside security aspects, mainly inherited from UPnP)

So what to do?  I'm not going to pull my apps from the public domain (EventWatcher, Netatmo, DataYours) in fact I want to continue to develop and support them.  But I want to work in a more open and stable environment.  I note the following:
  • Vera is a great Z-wave to HTTP bridge
  • The plugins developed here on the forum are great
  • The Luup API and HTTP Requests are great
  • Scenes, in particular the definition of triggers are abysmal
  • Lua is the best language in the world (OK, forgive me)
All would be solved if Luup was open source and could be run on the plethora of cheap and reliable hardware available today.  But it's not.  But we could get something like that effect if we engineered a sufficient subset of Luup to run on such a platform.  Could it be done?  What would we need?
  • UI
  • scheduler
  • web server
  • Luup compatible API
  • Device and Implementation xml file reader
  • Zwave bridge to Vera
  • runs most plugins without modification
What we wouldn't need is UPnP.

What have we (nearly) got already?
  • We have, courtesy of @amg0, the most excellent altUI: Alternate UI to UI7, and that, I think, is probably the hardest one to do in the above list.
  • Items 2 -  5, and 7, I've prototyped, in pure Lua, and posted elsewhere: DataYours on Raspberry Pi, running selected plugins unmodified, including: DataYours, EventWatcher, Netatmo, RBLuaTest, altUI.  See screenshot attached

Is it worth the effort?  Probably not.  Will I pursue this quest?  Yes, because as @Aaron noted in @Ap15's final thread:
The coders do what they do for 2 reasons... 1) they want the code for themselves & 2) they want to help the community ...

I plan to post the code on GitHub in due course: and now it is here https://github.com/akbooer/openLuup

=============================

October 2015: A summary of the project status...

openLuup is an environment which supports the running of some MiOS (Vera) plugins on generic Unix systems (or, indeed, Windows systems.) Processors such as Raspberry Pi and BeagleBone Black are ideal for running this environment, although it can also run on Apple Mac, Microsoft Windows PCs, anything, in fact, which can run Lua code (most things can - even an Arduino Yun board.) The intention is to offload processing (cpu and memory use) from a running Vera to a remote machine to increase system reliability.

Running on non-specific hardware means that there is no native support for Z-wave, although plugins to handle Z-wave USB sticks may support this. The full range of MySensors (http://www.mysensors.org/) Arduino devices are supported though the Ethernet Bridge plugin available on that site. A plugin to provide a bi-directional 'bridge' (monitoring / control) to remote MiOS (Vera) systems is provided in the openLuup installation.

openLuup is extremely fast to start (a few seconds before it starts running any created devices startup code) has very low cpu load, and has a very compact memory footprint. Whereas each plugin on a Vera system might take ~4 Mbytes, it's far less than this under openLuup, in fact, the whole system can fit into that sort of space. Since the hardware on which it runs is anyway likely to have much more physical memory than current Vera systems, memory is not really an issue.

There is no built-in user interface, but we have, courtesy of @amg0, the most excellent altUI: Alternate UI to UI7 (see the Vera forum board http://forum.micasaverde.com/index.php/board,78.0.html) An automated way of installing and updating the ALTUI environment is now built-in to openLuup. There's actually no requirement for any user interface if all that's needed is an environment to run plugins.

Devices, scenes, rooms and attributes are persisted across restarts. The startup initialisation process supports both the option of starting with a 'factory-reset' system, or any saved image, or continuing seamlessly with the previously saved environment. A separate utility is provided to transfer a complete set of uncompressed device files and icons from any Vera on your network to the openLuup target machine.

What openLuup does:
  • runs the ALTUI plugin to give a great UI experience
  • runs the MySensors Arduino plugin (ethernet connection to gateway only) which is really the main goal - to have a Vera-like machine built entirely from third-party bits (open source)
  • includes a bridge app to link to remote Veras (which can be running UI5 or UI7 and require no additional software.)
  • runs many plugins unmodified - particularly those which just create virtual devices 
 (eg. Netatmo, ...)
  • uses a tiny amount of memory and boots up very quickly (a few seconds)
  • supports scenes with timers and ALTUI-style triggers
  • has its own port 3480 HTTP server supporting multiple asynchronous client requests
  • has a fairly complete implementation of the Luup API and the HTTP requests
  • has a simple to understand log structure - written to LuaUPnP.log in the current directory - most events generate just one entry each in the log.
  • writes variables to a separate log file for ALTUI to display variable and scene changes.

What it doesn't do:
  • Some less-used HTML requests are not yet implemented, eg. lu_invoke.
  • Doesn't support the <incoming> or <timeout> action tags in service files, 
 but does support the device-level <incoming> tag (for asynchronous socket I/O.)
  • Doesn't directly support local serial I/O hardware (there are work-arounds.)
  • Doesn't run encrypted, or licensed, plugins.
  • Doesn't use lots of memory.
  • Doesn't use lots of cpu.
  • Doesn't constantly reload (like Vera often does, for no apparent reason.)
  • Doesn't do UPnP (and never will.) 


Latest release available now available on GitHub: https://github.com/akbooer/openLuup
« Last Edit: November 26, 2015, 06:31:02 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Tank

  • Sr. Member
  • ****
  • Posts: 322
  • Karma: +11/-7
Re: openLuup - running unmodified plugins on any machine
« Reply #1 on: May 22, 2015, 03:04:08 pm »
This is fantastic!   I will be watching this project closely  :)

Offline wilme2

  • Hero Member
  • *****
  • Posts: 1262
  • Karma: +32/-13
Re: openLuup - running unmodified plugins on any machine
« Reply #2 on: May 22, 2015, 03:19:51 pm »
All would be solved if Luup was open source and could be run on the plethora of cheap and reliable hardware available today.  But it's not.  But we could get something like that effect if we engineered a sufficient subset of Luup to run on such a platform.  Could it be done?  What would we need?

I wish MiOS would adopt a software model.  I just don't trust them to put enough specs into VeraPlus or VeraSecure.  But I would pay good money to run their software on the hardware of my choosing.

Failing that, I will watch this project with interest...
Long Live UI5!  (OK finally made the switch to UI7 in January 2018, and am pretty happy....)

Offline nitehawk

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +1/-1
Re: openLuup - running unmodified plugins on any machine
« Reply #3 on: May 26, 2015, 02:11:16 pm »

Can't wait to try this out!

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6340
  • Karma: +288/-70
  • "Less is more"
Re: openLuup - running unmodified plugins on any machine
« Reply #4 on: May 26, 2015, 02:16:30 pm »
Can't wait to try this out!

Might be looking for suitable guinea pigs soon!  Have you a particular use case (favourite apps, ...) you'd want to try first (starting simple) ?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Brientim

  • Sr. Hero Member
  • ******
  • Posts: 2497
  • Karma: +78/-7
Re: openLuup - running unmodified plugins on any machine
« Reply #5 on: May 26, 2015, 03:15:55 pm »
Fantastic 2000 post and significant roadmap of future achievements.

Offline amg0

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3164
  • Karma: +209/-8
Re: openLuup - running unmodified plugins on any machine
« Reply #6 on: June 06, 2015, 04:41:31 pm »
happy to support where I can.

Offline racarter

  • Hero Member
  • *****
  • Posts: 518
  • Karma: +12/-4
Re: openLuup - running unmodified plugins on any machine
« Reply #7 on: June 07, 2015, 01:03:26 am »
+1 here  :)
Indigo, Vera Edge (for Datamine only), Everspring and TKBHOME socket switches, SM103 door detectors, Homepro ZRP210 appliance modules, Fibaro relay switches, EZMotion, Heatmiser thermostats, Netatmo Weather, Camera and Healthy Home, Hikvision DVR/CCTV, Sonos, Texecom, Diax motorised deadlock.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6340
  • Karma: +288/-70
  • "Less is more"
Re: openLuup - running unmodified plugins on any machine
« Reply #8 on: June 07, 2015, 05:52:47 pm »
Just an update on progress... I'm afraid it's been a bit slow since I've not had as much time to work continuously on this as I wanted, and anyway, I'm not as quick as you @amg0.

One of the frustrations in reverse engineering some of the functionality has been that it's obvious that there are some better ways to do things, but you have to stick to the same API and the same format.  One example: the crucial internal variable "DataVersion" which keeps track of when anything changes, but has hugely inconsistent naming in HTTP request outputs:
  • in /data_request?id=status, it is actually called "DataVersion".  There is a separate variable called "UserData_DataVersion"
  • in /data_request?id=sdata, it is called "dataversion"
  • and in /data_request?id=user_data, the variable called "DataVersion" has the value of "UserData_DataVersion" and the value of "DataVersion" is not shown, but is a critical query parameter in calls to that request.

Sigh...

Anyway, some progress is being made.  From the outset, I wanted as few external dependencies as possible - everything is in Lua.  This means that I've had to craft my own HTTP server for the port 3480 access, although I rely on the native web server of the target platform to deliver the port 80 requests.  I did toy with the idea of using CGI or FastCGI, but in the end that seemed to make things harder rather than easier.  I also have a job scheduler and the complete (well, nearly) Luup API.  I've had enormous fun writing the timer code - particularly the sunrise/sunset calculations (I see that @futzle went through this ages ago when writing the heliotrope plugin.)

What it doesn't have, at the moment, is:
  • Persistence.  Everything is created in the startup code.  This is not as bizarre as it sounds because: (a) variables will be cloned by bridging to a real Vera or imported from non-Zwave devices; and (b) it's not supposed to restart as often as Vera (in fact, one version I have has been running without restart for weeks.)  I see no real difficulty in using an exact clone of a user_data file for implementing this if necessary.
  • Scenes.  This is a more serious shortcoming right now.  I actually have no intention of recreating Vera's exact scene functionality, because it is so flawed.  Whilst scene actions and timers are fine, the trigger definitions, which are so tied up with UPnP, are seriously lacking.  I envisage a HomeSeer-like interface which simply scripts the right Lua code.  Hence this solution can run on a native Vera too.  Think PLEG, but much simpler. (I have no expectations that PLEG will ever run on openLuup - not sure of the licensing implications anyway.)
  • Z-wave bridge.  This should be straight-forward and require no code at all on the real Veras.  I know how to do it (really just the same as any remote interface) but just haven't got around to it.
  • Arduino bridge.  The Arduino plugin should run, since all the Luup infrastructure is there, but I haven't yet got any Arduino sensors myself, so would need some help there.
  • UPnP.  No intention of ever implementing this, but all the UPnP-like HTTP requests are there.  Device references are always device numbers, not UDN strings. This means that S_*.xml service files are completely redundant.

I just have a bit more work to do on the device loader before making a release.

I'd welcome any comments on the architecture, such as I have described it, or simple test cases to start out with (in particular, then, favourite non-Zwave plugins.)
« Last Edit: June 08, 2015, 03:12:30 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline nitehawk

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +1/-1
Re: openLuup - running unmodified plugins on any machine
« Reply #9 on: June 12, 2015, 05:46:43 am »
One of the first plugins I want to try is : NorthQ Watt Calculator

I'm using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.

I also have Hue, Netatmo, Foscam and KodiRemote running, Hue and netatmo are already working on openluup.

It would be nice to run the Vera free of (non zwave) plugins and free up some of the very needed ram.
I have an raspberry 2 available for testing.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6340
  • Karma: +288/-70
  • "Less is more"
Re: openLuup - running unmodified plugins on any machine
« Reply #10 on: June 12, 2015, 05:56:44 am »
One of the first plugins I want to try is : NorthQ Watt Calculator

I'm using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.
Interesting.  I have a stand-alone implementation of this (runs on Vera) which does the calculation for all installed NorthQ devices and puts the Watts result back into their respective devices.  Not a plugin, but runs as startup code, taking almost no memory.

Quote
I also have Hue, Netatmo, Foscam and KodiRemote running, Hue and netatmo are already working on openluup.
Oh! ...so you've done some hacking already, with Hue running OK?  I don't have a Hue - what did you need to change to make this work?

Quote
It would be nice to run the Vera free of (non zwave) plugins and free up some of the very needed ram.
I have an raspberry 2 available for testing.
One of the things that's missing in the published code so far is implementation of device actions.  I've been working on this, and am just about there.  Rather depends on how involved you want to be as to when I share this (ie. how much beta testing you really want to do!)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline nitehawk

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +1/-1
Re: openLuup - running unmodified plugins on any machine
« Reply #11 on: June 12, 2015, 06:37:25 am »

I haven't have openluup running at the moment. That was my bad english, I thought i read somewhere that you had those running already.


Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6340
  • Karma: +288/-70
  • "Less is more"
Re: openLuup - running unmodified plugins on any machine
« Reply #12 on: June 12, 2015, 01:20:58 pm »
One of the first plugins I want to try is : NorthQ Watt Calculator

I'm using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.

Further to this, I've posted my NorthQ code here: http://forum.micasaverde.com/index.php/topic,16082.msg237849.html#msg237849.  You can remove your multiple NorthQ Watt Calculator plugins entirely.  This should save some memory on Vera whilst I keep working on openLuup!
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline nitehawk

  • Sr. Newbie
  • *
  • Posts: 43
  • Karma: +1/-1
Re: openLuup - running unmodified plugins on any machine
« Reply #13 on: June 15, 2015, 03:17:16 pm »
Thank you for the code.

I don't have multiple northq energy meters, just one, i was using the calculator to calculate watts for my green wave powernodes. http://forum.micasaverde.com/index.php/topic,18747.msg142098.html#msg142098

Can i use your code to see if i can get it to work with these powernodes?

« Last Edit: June 15, 2015, 03:30:29 pm by nitehawk »

Offline Ramias

  • Sr. Member
  • ****
  • Posts: 275
  • Karma: +7/-5
Re: openLuup - running unmodified plugins on any machine
« Reply #14 on: June 16, 2015, 12:50:52 am »
I am not skilled enough to contribute much at all but I wholeheartedly offer my moral support.

I have openHab downloaded (but not running) at the moment.  But I just have too much invested in Vera/LUUP and I know how to do what I need to do (various LUUP scripts, nested decisions etc.

But my Vera is just too unstable; perhaps due to resources.  The ability to keep Vera for HW gateway and then code separately is very appealing.