We have moved at community.getvera.com

Author Topic: openLuup: Version Log  (Read 8355 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #30 on: March 23, 2018, 10:58:00 am »
2018 Release 3.23
  • House Mode delay when switching from Home mode to any other
  • device loader modifications to facilitate debugging with ZBS
  • intra-LAN SMTP email server
  • I_openLuupCamera1.xml file to support cameras with motion sensors
  • images@openLuup.local email address to store camera email attached images
  • SendToTrash and EmptyTrash actions added to openLuup plugin
  • new Server pages in console with HTTP and SMTP server statistics

The camera implementation, SMTP server, and SendToTrash/EmptyTrash actions are further described in this thread: openLuup: Cameras
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #31 on: April 07, 2018, 06:35:23 am »
Development Branch: 2018 Release 4.7
  • SMTP server: fix image handling
  • fix device status periodic toggling
  • sandbox system libraries for each device
  • console Scheduler > Sandboxes page to report library modification by plugins
  • documentation updates

The sandboxing of system libraries per device was due to a conflict noted here:
http://forum.micasaverde.com/index.php/topic,77266.msg374686.html#msg374686

The problem arises when more than one plugin decides to modify a library module (eg. string) by adding extra functions with the same name, but possibly different functionality.  For example

Code: [Select]
function string:trim()
  return self:match "^%s*(.-)%s*$"
end

Whilst it's relatively straight-forward to ensure that different plugins using this syntax are isolated from each other...

Code: [Select]
local paddedString = "    a string with extra blanks at the ends    "
local trimmed = string.trim (paddedString)

...it's actually somewhat harder to handle this syntax, which also needs to work...

Code: [Select]
local trimmed = paddedString: trim()

However, the latest release does handle this. 

There is an additional console > Scheduler > Sandboxes page (the scheduler handles both scheduling and context switching) to show which plugins are the offenders:
Code: [Select]
Sandboxed system functions 

string.sandbox:
  device #3
    altui_split = function: 0x2045e50
    trim = function: 0x2145990
    starts = function: 0x2042030
    template = function: 0x2145c88
  device #345
    split = function: 0x238c8f0 

table.sandbox:
    empty 

Here, device #3 is AltUI and #345 is ALTHue.    You can see the code fix which @amg0 made to avoid a name conflict with the split function, before this was fixed in openLuup.  No such code work-arounds should now be necessary.  Both string and table libraries are now sandboxed.

I still believe, however, that it is bad practice to modify system libraries.

« Last Edit: April 08, 2018, 08:16:08 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #32 on: April 12, 2018, 09:55:55 am »
Development Branch: 2018 Release 4.12
  • POP3 email server and console page
  • mail and events mailboxes
  • add archive_video request (&format=1 for snapshots only)
  • refactored HTTP and SMTP servers
  • improved io.server closed socket handling

The big change here is a full implementation POP3 server, along with some permanent mailboxes:

Together with the SMTP server, the POP3 server allows a fully internal LAN system to send, store, and retrieve email messages.  It can conveniently be accessed by almost any email client.  I send trigger messages from my camera to both images@openLuup.local  which just stores the attached images in the images/ folder, and also events@openLuup.local which lets me review any trigger times on an old iPod Touch. 

Of course, if you want this to work when you are away from your home LAN, then you'd need to forward these via VeraAlerts or some such additional plugin.

Your email client can be used to manage the deletion of files from the mailboxes, or, alternatively, a timed scene using the openLuup SendToTrash and EmptyTrash actions will do the job automatically.

Other applications are up to your own imagination (please share!)


« Last Edit: April 12, 2018, 04:21:48 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #33 on: April 19, 2018, 06:35:29 am »
Development Branch: 2018 Release 4.19
  • UDP register_handler callbacks
  • UDP console Server page
  • VeraBridge 'onDashboard' attribute, to allow Vera favourites (thanks @rafale77)
  • add Trash to console Files menu
  • link to images from console Images page
  • fix LastTrip name in I_openLuupCamera1.xml motion sensor (was LastTripped, wrongly)

The major enhancement here is a callback for incoming UDP datagrams from specified ports.

The luup.register_handler() function has been extended to be able to specify the UDP protocol and a port to listen on.  The handler is called with a similar parameter list to regular callback functions, but with slightly different semantics:

  • port - the incoming port as a string, eg. "2222"
  • data - a table  with {datagram = received_datagram_string, ip = sender_ip}
  • protocol - "udp", in this case

So, for example, in Lua Startup, or within a plugin you could write:
Code: [Select]

function myUDPcallback (port, data, protocol)
    print "Incoming UDP!"
    print (port)
    print ((json.encode(data)))
    print (protocol)
end

luup.register_handler ("myUDPcallback", "udp:2222")

The callback handler is called for each incoming datagram.
« Last Edit: April 19, 2018, 06:49:10 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #34 on: May 16, 2018, 08:14:27 am »
Testing Branch: 2018 Release 5.15

I don't use the testing branch very often, but this is a slightly unusual set of circumstances.

To address some fundamental problems of device loading, I have replaced the XML decoder with a DOM-style implementation.  This has meant some significant modifications to the loader module, which reads all the device / service / implementation files.  Since it's such a basic change, I thought it wise to test thoroughly.  I have done my best to do so, updating the unit tests (now over 200 separate ones are run) and running the loader itself on over 350 UPnP files.  In the process I have uncovered some interesting errors and omissions, not only in third-party files, but also in standard Vera/MiOS ones.  I've let a few other plugin developers know about theirs, but hold out no hope for Vera changing anything.

You may blame @a-lurker for instigating this change:
A couple of observations:

This is OK:
Code: [Select]
<actionList><action></action><action></action></actionList>
but causes an error:
Code: [Select]
<actionList><action></action></actionList>
There is some problem with empty tags or tags that contain no 'child' tags being searched for when there is a single 'parent' tag. Just white space, or missing child tag or completely empty is allowed between tags but in this case needs to be ignored.

As an additional carrot to try this version out, I've taken the opportunity to fix/add a couple of other things:
  • RunScene - the openLuup plugin now supports this action (within the openLuup service.  Thanks to @rafale77 for the suggestion.)  It means that this is now available as a menu-selectable scene action.  Use with care and try not to create recursive calls to the same scene, or an infinite loop!
  • SetHouseMode - the VeraBridge plugin now supports this action (within the VeraBridge service.)  It means that changing house mode is now accessible as a menu-selectable scene action
  • category and subcategory - should now be correctly retained across reloads (thanks @rafale77)
  • scheduled scene time - should now update correctly for scenes which are cancelled by their Lua code (thanks @rafale77)

Although this is in the testing branch, I'm using it on one of my everyday systems.  It seems fine, but I hope I haven't broken too many things for others, you can always revert to the development branch. 

Just enter 'testing' in the Update box and click the update button to try this out.

I'm going to be away from my development system for the next 10 days, and not checking the forum, as I'm still recuperating, so you're on your own until then.

____________________

PS: Something more about the new XML parser:

Code: [Select]
Features include:
  - ignores processing instructions, comments, and CDATA
  - expands self-closing tags
  - reads and saves attributes
  - element relationship links:
    - parentNode, firstChild, lastChild,
    - previousSibling, nextSibling
  - some navigation routines:
    - nextNode(), including optional filter function
    - getElementsByTagName()
  - basic XPath searching:
    - xpath(),
    - xpathIterator()
  - .documentElement field of the model accesses the root XML document element

In a break from the WWW3 standard, text is NOT stored in a child text element,
but in the .nodeValue attribute of the element itself.  It seemed much easier this way.

It's quite easy to navigate this structure and extract information you might need from any XML file.
« Last Edit: May 16, 2018, 06:31:54 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #35 on: June 26, 2018, 11:47:03 am »
Development Branch: 2018 Release 6.26

This development release is a significant enhancement to openLuup functionality.  It includes a 'Data Historian' which stores previous device variable values in both in-memory cache and (optionally) on-disk archives.  Taking lessons learned from DataYours and EventWatcher, also from AltUI's Data Storage Providers, it offers an almost configuration-free approach to storing (and retrieving) variable values.  In addition, and in contrast to those other options and the dataMine plugin, it provides a natural way of reading historic variable values through the usual Luup interface.  Also included is a Grafana Data Source API so that data may be easily visualised if you already have a Grafana installation.  A big plus here is that you don't have to have set up storage for any specific variable, they are (almost) all available.

This development has been long in the making, and indeed the ideas for it come from well before openLuup was even thought of...

The State of Vera: Home Control Vs. Home Automation (August, 2013)

What this is not is a proper Data Historian API layer, which would allow you to do all the things which Ian Mercer blogged: count transitions, calculate max / min / averages, resample, ... this will be my next but one project.
That's why I'm implementing 'VariableWithHistory' on all my key sensors http://blog.abodit.com/2013/02/variablewithhistory-making-persistence-invisible-making-history-visible/

I do encourage you to read the (short) Ian Mercer blog page linked to by the above URL.

Some more on the Data Historian:
  • only numeric variable values are stored
  • By default, the in-memory cache stores all variable changes since system startup, with a small number of exceptions, retaining the most recent 1000 values (by default)
  • Variables not cached (by default) include timestamp-like ones, including LastUpdate, Polls, low-level Zwave, and some dates
  • The on-disk archive is enabled by the single LuaStartup line luup.attr_set ("openLuup.Historian.Directory","history/"), where 'history/' is the path (including trailing '/') relative to cmh-ludl/ where you want the data to reside.
  • By default, only a small subset of cached variables are archived on disk, including security sensors Tripped, temperature, humidity, and generic sensors Current values, energy metering KWH and Watts, battery levels, openLuup system memory and cpu stats, ...
  • The on-disk archives are implemented as Graphite Whisper files (as in DataYours)
  • different data types have different sample rates and retention policies.  For example, temperature data is sampled every 20 minutes, initially, but reduced in a number of stages to once per day after a year, and battery levels are just sampled once per day.
  • Default total on-disk archive duration is 10 years.  Typical file sizes, one per variable (containing multiple resolution data) are about 300Kbyte.
  • System impact is minimal, with high performance.  The additional system load is less than writing to the logs.
  • Database maintenance effort is zero.  The disk archive is of fixed size per variable, and total file space is roughly proportional to the number of archived variables.
  • the openLuup Console has two pages to view historian activities: openLuup > Historian (for the in-memory cache), and Files > History DB (for the on-disk archives)
  • the URL for the Grafana Data Source interface is simply your openLuup:3480 port
  • programmatically, data is retrieved using the luup.variable_get() call

I've been running versions of this for a while now.  On my 'production' system which is linked to 3 Veras, Netatmo, Philips Hue (thank you @amg0!), and some MySensors Arduinos, there are over 4000 device variables.  About 300 of those are cached in memory, and their on-disk archives take about 90 Mbyte.  There are, on average, 12 updates per minute, each one taking about 1.5 mS on an RPi.  (I don't have any security sensors triggering frequently on this system.)

For plugin developers, reading data from the cache/archive is trivial, using an additional {start,end} time parameter to variable_get():

Code: [Select]
local v0,t0 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33)
print ("normal current value", v0,t0)

local v2,t2 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33, {os.time()-3600, os.time()})
print ("over the last hour", pretty {value = v2, times=t2})

gives

Code: [Select]
normal current value 31.3 1530026929
over the last hour {
times = {1530023886,1530023926.5466,1530024526.9677,1530025127.1189,1530026327.7046,1530026929.0275,1530027486},
value = {31.1,31,31.2,31.3,31.2,31.3,31.3}
}

If the data over the requested time range is all in cache, then the time resolution will be better than 1mS.  Once on disk, it depends on the retention policy of the particular archive (sensible defaults, but all configurable if necessary) never finer than 1 second, but usually 5-10 minutes.  If there is no data within the time range, then arrays are empty, but within a non-empty array there are no nil points, the times simply denoting (approximately) when the variable changed.

There are further developments to do before turning this into a master release:
  • mirroring on-disk archives to remote Graphite or InfluxDB servers
  • total replacement for DataYours

Also, I plan to implement the shockingly missing Digest authorization for luup.inet.wget().

___________

Edit: this release also includes the luup.openLuup flag to denote that this is openLuup, not Vera.
« Last Edit: June 27, 2018, 03:02:19 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #36 on: August 22, 2018, 07:34:44 am »
Master Branch: 2018 Release 8.10

I've just merged the lastest development branch with the master to roll-up all the various changes and bug fixes which have accumulated.

This is not quite the master release I wanted to make, but it does fix a couple of significant problems, and, of course, include the latest things like Data Historian.  However, it doesn't include digest authorization in luup.inet.wget() or a couple of other things I wanted to add.  It seemed necessary, though, with recent interest in openLuup, and in particular fixed a long-standing install error. 

I've been a bit short of quality development time recently.

Thanks to all who have helped to make improvements over the last few months (and before!)
« Last Edit: August 22, 2018, 07:39:31 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #37 on: November 15, 2018, 06:10:05 am »
Development Branch: 2018 Release 11.15

Fixes synchronisation issue between system Mode attribute and openLuup HouseMode variable if luup.attr_set() was used to change mode.  Thanks to @DesT for pointing this one out.

Other minor code improvements.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #38 on: November 27, 2018, 10:54:00 am »
Development Branch: 2018 Release 11.26

  • enables historian mirroring to external Graphite and InfluxDB databases via UDP
  • implements missing "actions" request

In addition to the internal Graphite database used by the Data Historian, all archived variables can be mirrored to an external Graphite and/or InfluxDB database.  In order to make this efficient in terms of CPU and network resources, the transaction-free UDP protocol is used, avoiding any possibility of system deadlock.

InfluxDB instructions for configuring a UDP port are here:

    https://docs.influxdata.com/influxdb/v1.7/supported_protocols/udp/

DataYours may also be used as a destination for the Graphite historian mirror.

To enable sending to Graphite or InfluxDB, the following openLuup system attributes (for example) with IP and port number should be set in the Lua Startup:
Code: [Select]
luup.attr_set ("openLuup.Historian.Graphite_UDP", "127.0.0.1:2003")
luup.attr_set ("openLuup.Historian.InfluxDB_UDP", "172.16.42.129:8089")

Either of those target databases may then be queried for archived variable data.  Grafana queries work well, with auto-population of variable name menus.

As ever, please alert me of any problems.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #39 on: January 17, 2019, 10:35:53 am »
Master Branch: 2019 Release 01.17

New baseline release for 2019.  Rolls up previous 5 months of development branch changes.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #40 on: January 22, 2019, 03:07:50 pm »
Development Branch: 2019 Release 01.22
  • Schedule VeraBridge remote actions using <job> rather than <run> invocation.
This addresses a technical issue with remote Veras not being able to keep up with openLuup systems running on faster processors.  It mitigates the lack of a properly managed command request queue on Vera and apparently decreases the chance of a Vera restart.

Most users should notice little difference, but, as ever, if you find a problem, don't hesitate to raise an issue.



3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #41 on: January 29, 2019, 05:28:57 pm »
Development Branch: 2019 Release 1.29

  • console uses HTTP tables
  • console CSS in virtual file system
  • VeraBridge can link to openLuup systems

The look and feel of the console pages can now be changed through modifications of the openLuup_console.css file in the virtual file system file.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: openLuup: Version Log
« Reply #42 on: February 12, 2019, 03:32:18 pm »
Master Branch: 2019 Release 2.12

  • improved Console HTML pages
  • VeraBridge links to openLuup as well a Vera systems
  • fix Graphite API CGI Unix epoch time handling (used by Grafana)
  • fix SSL tslv1.2 error in openLuup_install.lua script
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.