The Vera Community forums have moved!

Advanced => Plugins & Plugin Development => Programming => openLuup => Topic started by: vwout on August 16, 2018, 03:22:00 pm

Title: openLuup on Docker (Hub)
Post by: vwout on August 16, 2018, 03:22:00 pm
To simplify plugin development I created a Docker image for openLuup (https://hub.docker.com/r/vwout/openluup/) and made it available on Docker Hub.

The Docker image is based on Debian (9, slim) and installs the master branch release of openLuup.
The image uses the automated installation script of openLuup and therefore also contains AltUI from the start.

It can be used directly from Docker Hub using
Code: [Select]
docker pull vwout/openluup
The image exposes port 3480, the (default) port that runs the openLuup console and AltUI.

To allow settings and installed plugins to survive removal of the container, the image uses a number of volumes:
The directory /etc/cmh-lu is supported to be mounted as a bind for easy plugin development.

More details on how to use the image are available on Github (https://github.com/vwout/docker-openluup).
The Github repository also contains a docker-compose file that allows quick creation of your own image based on either the master or development branch. The docker-compose setup also defines named volumes that are automatically mounted to an openLuup container.

The sources for the image are available on GitHub.
Title: Re: openLuup on Docker (Hub)
Post by: akbooer on August 18, 2018, 06:04:53 pm
Kudos to @vwout for creating this Docker image.

This has been requested several times in the past, so I hope that we get some willing folk to try it out.
Title: Re: openLuup on Docker (Hub)
Post by: RHCPNG on August 22, 2018, 04:13:17 am
This is really great, vwout! Thanks! I will certainly try it in the future.
Title: Re: openLuup on Docker (Hub)
Post by: vwout on August 26, 2018, 04:52:22 pm
The openLuup Docker image (https://hub.docker.com/r/vwout/openluup/) was updated to Master Branch: openLuup 2018 Release 8.10 (http://forum.micasaverde.com/index.php/topic,34526.msg413042.html#msg413042).

The scripts for generating the image received some fixes and the image now is about 16MB smaller because of a small optimization during the build.
For details, check the Github repository at vwout/docker-openluup (https://github.com/vwout/docker-openluup).
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on September 22, 2018, 07:51:00 pm
@vwout --

Fantastic!  I'm about to start using openluup and recently changed by basement server to Centos with the intention of doing most everything using docker.  It was previously esxi but I wanted to trim down a bit on the vm overheads, fool around with docker etc since it's a small server.

Which brings me to my point -- I'm wondering if maybe openluup would be viable on a smaller *nix install in docker versus Debian?  Put another way - what is the minimum set of *nix capabilities that openLuup needs.  I'm guessing the Debian base is quite "beefy" (~125MB) compared to (say) Alpine at about 5MB.

Thoughts anyone ?
Title: Re: openLuup on Docker (Hub)
Post by: akbooer on September 23, 2018, 06:57:28 am
Which brings me to my point -- I'm wondering if maybe openluup would be viable on a smaller *nix install in docker versus Debian?  Put another way - what is the minimum set of *nix capabilities that openLuup needs.

I know nothing about Docker, which is why I'm grateful to @vwout for this thread.

However, in terms of required OS resources, openLuup needs nothing but its interpreter and a few standard Lua libraries.  I've even had it running on an old Arduino Yun board (not on the Arduino, obviously, but the other single chip system running a stripped-down OpenWrt install.)  There may be a few basic shell utilities which some plugins use with os.execute()  - naughty them!
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on September 23, 2018, 05:13:18 pm
However, in terms of required OS resources, openLuup needs nothing but its interpreter and a few standard Lua libraries.

I too am grateful -  I hope nothing I said seemed to be negative.  When I saw this docker offering, I immediately installed it !!

The thing with docker is that the containers require an operating system (so in effect you have an operating system inside the docker container that is deployed on the host operating system).  Sounds like overhead (which it is) but it also has many advantages because of the isolation.

The size does not matter too much, given sufficient resource on the host, so my interest was more a nicety than anything too serious. Hence my questions about the requirements of openluup.  From what you have said it may be that a smaller footprint is possible.  I know my plugin uses several os.execute calls, all basic with the exception of openssl :-)

If I get some time over the next couple of weeks I'll do some experiments.   First though is cleaning up GCal3 and github :-)

Title: Re: openLuup on Docker (Hub)
Post by: vwout on September 26, 2018, 04:46:36 pm
Thanks for the feedback and heads up.

I deliberately choose Debian as a base. On one hand it is easier to setup, since you can e.g. just apt-get lua. On the other hand, a Debian image has more out of the box features compared to something like alpine, which makes the image easier to use by others that are more used to running Openluup on regular Linux.

A bit on Docker in general: To reduce the overhead, I choose the slim base image of Debian Stretch. The slim image is (only) 55MB. Keep in mind that Docker images use layers. If you use a number of services (images), it actually is likely that you have more images that are based on Debian. In that case, the base image is only present once on your system, not in copies for every image. And regarding the overhead, please note that you don't have a full OS, with its services and stuff, running in the container. Unless you spin it, no other processes besides the one you start, is running in a container.

The complete openluup image is 72.7MB in its current state.
But I do get your point.

Making a small image with lua does not seem to be that easy by judging several images that are around, all based on alpine:
- pirogoeth/alpine-lua (https://hub.docker.com/r/pirogoeth/alpine-lua/): 1K pulls, last updated 2 years ago: 176MB
- superpaintman/lua (https://hub.docker.com/r/superpaintman/lua/): 10K+ pulls, last updated 2 years ago: 612MB

Though there is light at the end of the tunnel: akorn/luarocks (https://hub.docker.com/r/akorn/luarocks/): 12MB
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 01, 2018, 09:32:35 pm
Making a small image with lua does not seem to be that easy by judging several images that are around, all based on alpine:

Makes sense.

Since I like to use docker-compose, I thought I'd post the docker-compose.yml file I use as a starting point for others who my be inclined.
I include a mount point to use for transferring files into and out of the running container (via /mnt/uploads in the container).

Code: [Select]

version: '2'
services:
  openluup:
    container_name: openluup
    image: vwout/openluup
    restart:  unless-stopped
    environment:
      - TZ=America/Denver
    network_mode: host
    volumes:
      - <a convenient folder on your host>:/mnt/uploads

Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 02, 2018, 05:35:48 pm
Thanks for sharing the compose file.
There is one in my github repository (https://github.com/vwout/docker-openluup/blob/master/docker-compose.yml) that hosts the sources for my docker image as well, but that focusses more on building the various variants than on using the ready-made image from docker hub.

Even though I have some doubts on the effective added value, your request for a smaller image challenged me. Since I investigated it already a bit, I knew it was possible. So I created an additional variant of the openLuup image based on Alpine. And indeed, the image is a lot smaller. Total size including everything is 18MB now 8)
The Dockerfile is more complex, but as a user that should not bother you too much.

The source is on github and the image vwout/openluup:alpine (https://hub.docker.com/r/vwout/openluup/tags/) is on Docker Hub. Be sure to use the tag alpine.
I must admit that I did not spend a lot of time on testing this variant, so please provide feedback.
Title: Re: openLuup on Docker (Hub)
Post by: akbooer on October 02, 2018, 06:04:11 pm
Oh, nice work!

The 18Mb number fits much more comfortably with the whole openLuup concept of being parsimonious with resources... makes you realise just how profligate MiOS/Vera is, and shows that HA really shouldn't need heavy-duty hardware, on the whole.
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 02, 2018, 07:23:24 pm

So I created an additional variant of the openLuup image based on Alpine. And indeed, the image is a lot smaller. Total size including everything is 18MB now 8)


 :D :D Wooo Hooo !!!!   :D :D

Brilliant - I will begin testing it tonight.  If I spot anything I will report back.
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 05, 2018, 02:48:52 pm
I must admit that I did not spend a lot of time on testing this variant, so please provide feedback.

Looking good !!
I have only found two things so far and both are relatively minor and easy to fix.

1.  The folder /var/log/cmh is missing.  As a result, log files are not being created.
2.  Timezones are not implemented.  Here is what I did:
     apt -U tzdata
     and then create a mount /etc/localtime :/etc/localtime
This way it will automagically pick up the time from the host (assuming the host uses tzdata).

Note: it is possible to make this more general by passing in an environment variable e.g. TZ and then running some commands like:
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone  (not sure the second command affects anything btw)

In any case - for now I have gone with the first approach.  So now my docker-compose.yml looks like this (remove the TZ variable and added the second mount):

Code: [Select]
version: '2'
services:
  openluup:
    container_name: openluup
    image: vwout/openluup
    restart:  unless-stopped
    network_mode: host
    volumes:
      - <a convenient folder on your host>:/mnt/uploads
      - /etc/localtime:/etc/localtime

Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 05, 2018, 04:26:24 pm
Good point about the timezones.

The logs mounts are equal to the Debian image. The openLuup logs are in /etc/cmh-ludl/logs/, or does it use /var/log/cmh (where Vera stores its logs) as well?
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 05, 2018, 04:40:07 pm
or does it use /var/log/cmh (where Vera stores its logs) as well?

So it looks like (understand I'm new to this environment):
1.  ALTUI looks to, by default, log to /var/log/cmh  BUT,  if running on openluup, then that location is on openluup
2.  openluup logs by default to /etc/cmh-ludl/logs -- so that's where the logs for plugins installed on openluup go
3.  Vera of course logs to /var/log/cmh

There may be some startup mapping for openluup that can be tweaked (not got there yet) .....

If I go to ALTUI --> Misc --> osCommand --> Tail Logs (with your default install) it does not return anything.  Adding that directory and restarting then provides log information. 
The default command is:  tail -n 50 /var/log/cmh/LuaUPnP.log
By the looks of it, it's high level stuff, but "something" is definitely placing log entries there.

Executing tail -n 50 /etc/cmh-ludl/logs/LuaUPnP.log  gives openluup information as well as locally installed plugins.

Not yet sure how to get to the vera logs from here  :o

These may end up being some tweaks / mapping for the image as it gets figured out .....
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 06, 2018, 02:23:16 pm
So - we are on the right track, this from the official openluup documentation

Quote
openLuup also maintains another log, which is a subset of the main one, and contains only device variable changes, scene invocations, and workflow messages. Those entries are written to a log file, if the /var/cmh/ directory exists, in /var/cmh/LuaUPnP.log,

One other thing that would likely be helpful:  set up /etc/cmh-ludl as a docker volume so that updates to the image do not over write that directory (there may be others ...)  or instructions on how to copy from that directory to a folder on the host and then mount the host directory to the image (I can do that tomorrow).  I kinda prefer the later since it will survive an accidental docker reset.

Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 06, 2018, 03:08:41 pm
Not yet sure how to get to the vera logs from here  :o

Well, there is no Vera in scope of the openLuup container. openLuup could be linked to your Vera, but the Vera logs won't be in the container on /var/log/cmh, but on your Vera. To access them from altUI running on openLuup that is on another box, you need to mount the folders from your vera. sshfs could be an option there.


So - we are on the right track, this from the official openluup documentation

Quote
openLuup also maintains another log, which is a subset of the main one, and contains only device variable changes, scene invocations, and workflow messages. Those entries are written to a log file, if the /var/cmh/ directory exists, in /var/cmh/LuaUPnP.log,
I can add the volume /var/cmh to have the Vera-like openLuup logs. Since you already have the full log, this may not add much other than the fact the removing information makes the remain stand out more, which could be usefull.

One other thing that would likely be helpful:  set up /etc/cmh-ludl as a docker volume so that updates to the image do not over write that directory (there may be others ...)  or instructions on how to copy from that directory to a folder on the host and then mount the host directory to the image (I can do that tomorrow).  I kinda prefer the later since it will survive an accidental docker reset.

This is something I discussed with akbooer in the prelude to publishing the image, see https://github.com/akbooer/openLuup/pull/13 (https://github.com/akbooer/openLuup/pull/13). Detaching openLuup from the image could reduce the image to basically a lua image with some plugins. The directory /var/cmh-ludl is a volume already. It means that docker creates the volume when the image is started first time, Docker will copy the contents of the folder to the newly created volume in that case. Using a pre-created (e.g. named) volume won't work, since it will remain empty.
Title: Re: openLuup on Docker (Hub)
Post by: akbooer on October 06, 2018, 06:27:46 pm
FWIW, I generally add a sym link from /var/log/cmh to /etc/cmh-ludl/logs, and then the AltUI tail logs command works as is.
Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 07, 2018, 08:50:55 am
The images at vwout/openluup on Docker Hub (https://hub.docker.com/r/vwout/openluup/) are updated.

I applied the following changes:
- Providing /var/log/cmh Vera-style log folder (symlinked to openLuup logs)
- Debian based image: New tag (next to latest): 'slim'
- Alpine based image: Enable timezone configuration via TZ environment variable
- Update to compose-file
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 08, 2018, 08:03:58 pm
Looking very good !

I was getting errors from the Alpine version, basically complaining about syntax errors.  I did not check the debian version.

I believe the error is in  the file  openLuup_reload_for_docker   Specifically the line

Code: [Select]
function openLuupShutdown {

should be

Code: [Select]
openLuupShutdown(){

at least when I make that change, all seems well  :o
Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 09, 2018, 04:11:16 pm
When/how did you get any errors? I can't reproduce them.

I did notice that the shutdown script does not work how it should ... it no longer prints to log to stdout for tracing by `docker logs`
Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 09, 2018, 04:45:18 pm
The errors occurred when I tried to start a new container or restart with different runtime parameters.  Cannot remember exactly.  I was grinding through and not paying as much attention as I should.  I mostly use a docker-compose file to create the container.  It's possible that I fat fingered something procedurally and 'fixed' something that did not need fixing.

I can go back and try again with the original syntax later tonight / tomorrow and see if I can reproduce.   The error message was something about unexpected } -- and mentioned the file openLuup_reload_for_docker so that's what led me to suspect the syntax on the function definition.

One other thing - and I need to dig a little more:  the volume declarations in the Dockerfile .....     My understanding (could easily be flawed) is that they end up being created at container creation time with a random named (for the volume).  But what happens if you later delete the container and create a new one - do not these volumes get recreated?  If that's the case the date (from the previous container ) would not carry over to the new one.   So I'm thinking it may be better to create named volumes the docker run command, or mount bind to folders on the host.  Thoughts?
Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 09, 2018, 05:39:06 pm
Nevermind, I linted the script(s) and they had errors anyhow. No curly braces though.

Your understanding of volumes is more or less correct. Volumes are created automatically by Docker when the container starts - at least for the volumes specified in the Dockerfile, unless a mount or bind is specified for the volume. This is a so called unnamed volume. The volumes however are not deleted when the container is removed.
So your data will be preserved. Just hard to find.

Thing is that the volumes that are created by Docker at startup of a container, are unnamed, so hard to recognize. You can, as I do in the composefile (or using `docker volume create`) manually create a named volume to mount with a container. The difference is that Docker only copies the data from the image to the volume in case of creation of an unnamed volume. When you bind or mount a volume, Docker does not do this.
At least, that was the case. Apparently, recently Docker changed. It also copies data to a newly created empty volume.

And that is what the composefile is using. It creates empty volumes for ludl (/etc/cmh-ludl/), logs and backups. At startup, the contents of the openLuup image is copied into the ludl volume. As long as this volume is used with a container, the data should be preserved. Updating openLuup in that case should happen via the auto-update feature of openLuup itself, not by installing a new image. A new image is only required to get environment bufixes and security updates.

Title: Re: openLuup on Docker (Hub)
Post by: Stuart on October 09, 2018, 05:45:16 pm
Nevermind, I linted the script(s) and they had errors anyhow. No curly braces though.

The script openLuup_reload_for_docker had curly braces in the function declaration.  Are we talking at cross purposes ?

Quote
You can, as I do in the composefile (or using `docker volume create`) manually create a named volume to mount with a container.

I was not using your compose file -- so I best go take another look !

Again - thanks for taking this on and doing such a great job !!!!
Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 10, 2018, 03:01:23 pm
Are we talking at cross purposes ?
I meant to say that the linter did not complain about curly braces, but that there were a few other issues. One of which was the use of the keyword 'function', which apparently is allowed in bash, but not in sh.
Title: Re: openLuup on Docker (Hub)
Post by: parkerc on October 20, 2018, 03:10:18 am
Hi @vwout

Looking at your username it?s clear which one is yours,  but it was interesting to see other Docker options for openluup listed too.

slightlyaskew/openluup
DOCKER - openLuup Container

vwout/openluup
DOCKER - Automated Build Ready to use openLuup environment with AltUI based on Debian with mountpoints for custom plugins.

airedale/openluup
DOCKER
Title: Re: openLuup on Docker (Hub)
Post by: parkerc on October 20, 2018, 03:15:33 am
I'm going to use Container Station on my QNAP NAS to install the Docker image, I believe the approach will be similar to Synology's Container Station too ?

Are there any specific commands I should look to enter when creating the Openluup container?
Title: Re: openLuup on Docker (Hub)
Post by: vwout on October 21, 2018, 05:40:12 pm
Looking at your username it?s clear which one is yours,  but it was interesting to see other Docker options for openluup listed too.
Yep :) I decided to roll my own image since the others are big, undocumented and not updated recently.

Are there any specific commands I should look to enter when creating the Openluup container?

The image is usable without any particular configuration, though you do want to map at last the exposed port 3480 to be able to access AltUI.
I don't see this being configured in the screenshots. Maybe it is automatic on Qnap (I don't own one), but you might want to check this in the advanced settings. The container does not have any other configuration that you must set. And I think you can reduce the amount of memory. OpenLuup does not need 8GB :)

There should not be that much difference between using it on any Docker host. The only thing is that to use persistent data stores for openLuup, logs and backups, you probably have to use the command line, just like on Synology. Qnap also does not seem to offer the use of named mounts via their UI.
Title: Migrate openluup to docker
Post by: RHCPNG on December 29, 2018, 10:14:12 am
Hi guys,

I'm running openluup in vbox now and want to migrate to docker, because of some strange behavior in openluup/AltUI. What is the easiest way to restore my setup in the docker container?

I don't want to copy all files, because then I will probably have the same issues? The issue seems related to AltUI.

I know I need the user_data.json of course, but that is probably not enough.

Any suggestions?
Title: Re: openLuup on Docker (Hub)
Post by: vwout on January 03, 2019, 02:56:18 pm
Most likely reason for strange behavior is a rogue plugin. Do you have plugins installed?

Migrating configuration is the easy part, your dependency on plugins is the complicating part here since there is no way to automatically install plugins.
When you have the user_data.json, you can provide this to docker when you start the container for the first time, see the documentation on https://github.com/vwout/docker-openluup, or in the docker-compose file at https://github.com/vwout/docker-openluup/blob/master/docker-compose.yml.
Openluup will import the data provided by the user_data.json as specified in the environment variable USER_DATA_JSON into its configuration. Any functionality depending on plugins won't work until the plugins are installed.

I don't know whether configuration related to not available plugins will survive, or when the configuration is kept when plugins are installed. This will require some experimentation.
If you know which plugins you need and which files are related to these plugins, you can try exporting these from vbox and storing them in a directory. Bind this directory as /etc/cmh-lu/ and openluup will load the plugins right from the start.

The good news is that you can try as many times as you want. As long as you keep your vbox configuration as is, you can recreate the openluup container as many times as you want.
I would recommend using a volume for at last the openluup environment, that is /etc/cmh-ludl/.
Title: Re: openLuup on Docker (Hub)
Post by: RHCPNG on January 04, 2019, 05:20:41 am
Most likely reason for strange behavior is a rogue plugin. Do you have plugins installed?

Migrating configuration is the easy part, your dependency on plugins is the complicating part here since there is no way to automatically install plugins.
When you have the user_data.json, you can provide this to docker when you start the container for the first time, see the documentation on https://github.com/vwout/docker-openluup, or in the docker-compose file at https://github.com/vwout/docker-openluup/blob/master/docker-compose.yml.
Openluup will import the data provided by the user_data.json as specified in the environment variable USER_DATA_JSON into its configuration. Any functionality depending on plugins won't work until the plugins are installed.

I don't know whether configuration related to not available plugins will survive, or when the configuration is kept when plugins are installed. This will require some experimentation.
If you know which plugins you need and which files are related to these plugins, you can try exporting these from vbox and storing them in a directory. Bind this directory as /etc/cmh-lu/ and openluup will load the plugins right from the start.

The good news is that you can try as many times as you want. As long as you keep your vbox configuration as is, you can recreate the openluup container as many times as you want.
I would recommend using a volume for at last the openluup environment, that is /etc/cmh-ludl/.

Yes I have multiple plugins installed. I understand that will be the tricky part. Maybe I will try to migrate in the near future, but there is no hurry anymore, because I have mij vbox running quite well again.

Thanks for the docker! I will definitely be using it in the near future. Probably when someone else has migrated successfully with plugins ;-)