The Vera Community forums have moved!

Advanced => Programming => Plugins & Plugin Development => Topic started by: 4integration on November 07, 2013, 04:43:14 pm

Title: MQTT Client implementation
Post by: 4integration on November 07, 2013, 04:43:14 pm
I am trying to load a lua module but fails...all files uploaded

In "I_SensorMqtt1.xml" I have the following code but it fails at

u = require("utility")

Do you have any hints what's wrong?

Code: [Select]
<?xml version="1.0"?>
<implementation>
    <functions>
        local p
    local u
        local m
       
        function startup(lul_device)
            luup.log("SensorMqtt plugin: loading library L_SensorMqtt ...")
            if (package.path:find ("/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua", 1, true) == nil) then
                package.path = package.path .. ";/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua"
            end
           
            package.loaded.L_SensorMqtt = nil
            p = require("L_SensorMqtt")
            if (package.loaded.L_SensorMqtt == nil)
            then
                luup.log("SensorMqtt plugin: plugin is not installed correctly. Library L_SensorMqtt cannot be loaded.", 1)
                luup.task("Plugin not correctly installed", 2, "SensorMqtt plugin", -1)
                return false
            end
            luup.log("SensorMqtt plugin: library L_SensorMqtt loaded")

            luup.log("SensorMqtt plugin: loading library utility ...")
            package.loaded.utility = nil
            u = require("utility")
            if (package.loaded.utility == nil)
            then
                luup.log("SensorMqtt plugin: plugin is not installed correctly. Library utility cannot be loaded.", 1)
                luup.task("utility not correctly installed", 2, "SensorMqtt plugin", -1)
                return false
            end
            luup.log("SensorMqtt plugin: library utility loaded")

            package.loaded.mqtt_library = nil
            m = require("mqtt_library")
            if (package.loaded.mqtt_library == nil)
            then
                luup.log("SensorMqtt plugin: plugin is not installed correctly. Library mqtt_library cannot be loaded.", 1)
                luup.task("mqtt_library not correctly installed", 2, "SensorMqtt plugin", -1)
                return false
            end
            luup.log("SensorMqtt plugin: library mqtt_library loaded")
           
            return p.startup(lul_device)
        end
    </functions>
    <startup>startup</startup>
</implementation>
Title: Re: Loading lua modules
Post by: watou on November 07, 2013, 04:50:01 pm
I am trying to load a lua module but fails...all files uploaded

Loading would fail if there were a syntax error in utility.lua . 

Also, if your module is encrypted (file name extension .luaenc) then it has to explicitly name itself in the first line like this:

Code: [Select]
module("utility",package.seeall)
If you tried to instead use:

Code: [Select]
module("...",package.seeall)
Then MIOS's custom module loader for encrypted modules will fail (probably a minor bug in their loader, at least as of 1.5.408).

watou
Title: Re: Loading lua modules
Post by: 4integration on November 08, 2013, 06:09:28 pm
Thanks watou, tried that but no success. Moved the other loading into the lua code and that works but now I have another issue.

The MQTT Lua modules is here: http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua

Do you see anything strange? (I know it's a bit out of scope for MCV but since you are skilled Lua guys/girls)

Error:
Code: [Select]
01      11/09/13 0:00:36.101    luup_log:62: SensorMqtt plugin: Enter sendToMqtt <0x2d773680>
01      11/09/13 0:00:36.101    luup_log:62: SensorMqtt plugin: Create MQTT Client <0x2d773680>
01      11/09/13 0:00:36.102    luup_log:62: SensorMqtt plugin: Connect to 192.168.1.21 and port 1883 <0x2d773680>
01      11/09/13 0:00:36.102    LuaInterface::CallFunction_Timer-5 function sendToMqtt failed [string "module("L_SensorMqtt", package.seeall)..."]:47: attempt to index field 'client' (a nil value) <0x2d773680>

The code:
Code: [Select]
module("L_SensorMqtt", package.seeall)

-- Service ID strings used by this device.
SERVICE_ID = "urn:upnp-sensor-mqtt-se:serviceId:SensorMqtt1"

local mqttServerIp = nil
local mqttServerPort = 0

local MQTT = require("mqtt_library")
local SENSOR_MQTT_LOG_NAME = "SensorMqtt plugin: "

local function log(text, level)
luup.log("SensorMqtt plugin: " .. text, (level or 50))
end

function startup(lul_device)
_G["sendToMqtt"] = sendToMqtt
-- Help prevent race condition ???
-- luup.io.intercept()

SENSOR_MQTT_DEVICE = lul_device
log("Initialising SensorMqtt", 1)

--Reading variables
mqttServerIp = luup.variable_get(SERVICE_ID, "mqttServerIp", SENSOR_MQTT_DEVICE)
if(mqttServerIp == nil) then
mqttServerIp = "0.0.0.0"
luup.variable_set(SERVICE_ID, "mqttServerIp", mqttServerIp, SENSOR_MQTT_DEVICE)
end

mqttServerPort = luup.variable_get(SERVICE_ID, "mqttServerPort", SENSOR_MQTT_DEVICE)
if(mqttServerPort == nil) then
mqttServerPort = "0"
luup.variable_set(SERVICE_ID, "mqttServerPort", mqttServerPort, SENSOR_MQTT_DEVICE)
end
log("Start - Preparing worker thread", 1)
-- Prepare the worker "thread"
luup.call_delay("sendToMqtt", 20, "")
log("Done - Preparing worker thread", 1)
end

function sendToMqtt(luup_data)
luup.log(SENSOR_MQTT_LOG_NAME .. "Enter sendToMqtt", 1)
luup.log(SENSOR_MQTT_LOG_NAME .. "Create MQTT Client", 1)
luup.log(SENSOR_MQTT_LOG_NAME .. "Connect to " .. mqttServerIp .. " and port " .. mqttServerPort, 1)
mqttServerPort = tonumber(mqttServerPort)
local mqtt_client = MQTT.client.create(mqttServerIp, mqttServerPort)

luup.log(SENSOR_MQTT_LOG_NAME .. "Connect MQTT Client", 1)
mqtt_client:connect("Vera")

luup.log(SENSOR_MQTT_LOG_NAME .. "Publish MQTT message", 1)
mqtt_client:publish("Vera/Test", "*** Vera test start ***")

--mqtt_client:subscribe({ args.topic_s })

luup.log(SENSOR_MQTT_LOG_NAME .. "Destroy MQTT Client", 1)
mqtt_client:destroy()

luup.log(SENSOR_MQTT_LOG_NAME .. "Done with MQTT", 1)

luup.call_delay("sendToMqtt", 10, "")
luup.log(SENSOR_MQTT_LOG_NAME .. "Leaving sendToMqtt", 1)
end

Title: Re: Loading lua modules
Post by: watou on November 08, 2013, 06:22:42 pm
I think your MQTT variable needs global scope so it can be seen when luup.call_delay calls your sendToMqtt function.  Since it's not visible in scope MQTT.client has a nil value, since it's not the MQTT you think it is.

I think that's it, anyway.

watou


Title: Re: Loading lua modules
Post by: 4integration on November 09, 2013, 08:07:57 am
Thanks for you idea, tested with both removing "local" as well as using _G ...
Code: [Select]
MQTT = require("mqtt_library")
_G["MQTT"] = MQTT

To test more simple code I removed "luup.call_delay(...)" and called MQTT in the startup method, but got the same result :(
Title: Re: Loading lua modules
Post by: watou on November 09, 2013, 08:19:21 am
Thanks for you idea, tested with both removing "local" as well as using _G ...
Code: [Select]
MQTT = require("mqtt_library")
_G["MQTT"] = MQTT

To test more simple code I removed "luup.call_delay(...)" and called MQTT in the startup method, but got the same result :(

I just noticed that the mqtt_library.lua at the URL you provided (http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua/mqtt_library.lua (http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua/mqtt_library.lua)) does not have a module statement at its top.  I don't know why that is or what the ramifications are of that.

If you add this:

Code: [Select]
module("mqtt_library", package.seeall)
to the top of that mqtt_lbibrary.lua, does it help?  If this is also a dead-end, I'm probably out of advice (even bad advice :) ). 

But I'm very interested to see the results of your labor with MQTT.  I'm trying to get into some Arduino tinkering, and I think this is a good middleware to start with.

Regards,
watou
Title: Re: Loading lua modules
Post by: 4integration on November 10, 2013, 05:08:51 am
Hey,

I got it working! :D

But I have to admit I am on deep water and no fully satisfied :)

I read up a little on Lua modules (http://lua-users.org/wiki/ModulesTutorial) and checked the Vera-style and MQTT-style. It seems that Vera implements the the old style (and deprecated) way for Lua 5.0 and early 5.1 and MQTT uses the new for Lua 5.1 and 5.2. Then suspected them somewhat being incompatible.

MQTT and Utility uses "objects" (or "classes"), not sure about the correct terminology since I new at Lua).
So last night I hacked mqtt_library and utility and removed these objects and made some other small changes required since I introduced some conflicts that way.

I have attached the hacked MQTT source so you can diff in full but to mention some changes

Added:
Code: [Select]
module("mqtt_library", package.seeall)
Removed:
Code: [Select]
local MQTT = {}and all references to MQTT in mqtt_library e.g.:

Code: [Select]
MQTT.Utility = require("utility")
...
function MQTT.client.create(
to
Code: [Select]
Utility = require("utility")
...
function client.create(

The variable conflict solved from:
Code: [Select]
MQTT.message = {}
MQTT.message.TYPE_RESERVED    = 0x00
...etc
to
Code: [Select]
messageType = {}
messageType.TYPE_RESERVED    = 0x00
...etc

For Utility I had to remove the "object" and it functions and make all local functions global instead of local.

Even if I got it working I don't like this solution since now I have my own fork of MQTT Lua libraries.

So some questions to you guys:
1.  Is it a fact that modules implemented with different Lua versions are incompatible?
2. Can this be solved without touching the MQTT library?
3. Is it possible to make use of the new module style in Vera? I made a quick test to use the new module style in Vera but seems to fail even if it's 5.1 (maybe it's the "early 5.1" as mentioned in the link above

Title: Re: Loading lua modules
Post by: Thiemen on November 10, 2013, 05:32:57 am
I hope some other people can assist you, since i think this can be an interesting add on to vera, especially with the "internet of things" coming.
For now i will follow this tread...

Maybe you should also rename the topic of this tread, so other people recognise it is about MQTT?
Title: Re: MQTT Client implementation
Post by: averell on May 19, 2014, 06:33:56 am
I managed to get this MQTT integration working, based on the code you posted here. My idea is to try to build a plugin that allows me to control devices on the vera by sending messages on MQTT, and the other way atround, to offer a generic way of creating devices in vera that communicate via MQTT to the raspberry, some arduino's etc. that do the real work. This way I can use the vera sometimes as a master (I love my homewave app!), but sometimes as a slave when I want to control zwave devices from my other solutions.
Anybody else out there already playing with this? My next step is going to be to handle incoming messages.

Averell
Title: Re: MQTT Client implementation
Post by: hek on May 19, 2014, 10:30:20 am
Looking forward to see your results.
Title: Re: MQTT Client implementation
Post by: aPL on May 20, 2014, 08:19:19 am
As a side-note I'm building a logging module in node for mqtt messages with storage in memory, memcache, redis or mysql.
Title: Re: MQTT Client implementation
Post by: averell on May 21, 2014, 08:59:17 am
So the good thing is that I can indeed react to incoming messages. The bad thing is, it's not working based on a push, but I need to pull MQTT with the mqtt_client:handler("") function from the MQTT library. I'm used to working with node.js where the callback function get's invoked whenever a messages comes in, here I need to tell the MQTT client to go check, only then it calls the callback function.
For building devices that just need to be updated with some information like temperature etc. it is quite ok, I can call the client:handler every few seconds or so, and update the device status. For turning on devices, ideally I would see my very react a bit faster, but I wonder about the overhead of calling this function to often.
I will play a bit with this to see how fast I can make this work without causing too much problems, every second or so might work out ok, or I might go for a solution where I limit the direct MQTT connectivity to updates to devices (incoming) and switching external stuff on (outgoing), and directly swithing on devices on the vera via http, letting my raspbberry translate mqtt commands to HTTP. Unsatisfactory though it may be...

Av.
Title: Re: MQTT Client implementation
Post by: xAPPO on May 23, 2014, 06:37:41 am
Watching this topic with great interest. MQTT support would be a really useful addition to Vera.
Title: Re: MQTT Client implementation
Post by: 4integration on May 24, 2014, 05:06:44 am
I have a MQTT plugin running that monitor variables and fire off MQTT messages. The configuration is pretty "raw" though but doesn't change so often
Title: Re: MQTT Client implementation
Post by: averell on May 25, 2014, 04:37:15 am
Very interested to see what you did 4integration, as part of what I want to do, I also want to publish out all variables. I didn't get far due to travel, but I got child devices to send messages, and have temperature sensors publish messages over MQTT that set the device current temperature on the vera. Now thinking about the architecture, naming, making it hardcoded or flexible, etc. I will probably end up with a table with some hardcoded stuff that suits my needs. Some of the knowledge I will keep outside of the vera anyway, the whole idea is to keep some of the rules etc. away from the vera.
Title: Re: MQTT Client implementation
Post by: xAPPO on May 25, 2014, 07:06:31 am
Hi Averell,   

Do you think the poll (cf event) limitation is something that can be worked around as the code matures or is it always going to be required ?  Polling is so wasteful so it would be really nice to eliminate if possible.

Will you (and 4Integration) be releasing some usable Vera LUA code for this , albeit with the hardcoded bits that suit your setup ? Or should I just dive in from the code snippets above ?

K
Title: Re: MQTT Client implementation
Post by: akbooer on May 25, 2014, 07:20:01 am
Now thinking about the architecture, naming, making it hardcoded or flexible, etc.

I don't know if this helps, but the scheme I used for naming in the DataYours app was:

vera.device.service.variable (for example: Vera-12345678.043.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel)

This gives a unique path to every variable on multiple Veras (the Vera number comes from the unique machine number found in the variable luup.pk_accesspoint) and contains all the information that you need to get or set the variable.  For use on a file system, I've had to substitute the colon ':' in serviceIds with a caret '^' to ensure compatibility with all file systems that it's likely to come across.
Title: Re: MQTT Client implementation
Post by: 4integration on May 25, 2014, 02:45:41 pm
@averell, @XAPPO
I am not very proud of the code though....beginner at Lua and have not had available time to get it nice.
Upload the files and create the device with "D_SensorMqtt1.xml", then define IP/Port for your MQTT server on the device (I use ActiveMQ on my server@home).
To monitor variables add a list of devices and variable at "monitorConfig" with the format
<devideId>:<variable>|<devideId>:<variable>|<devideId>:<variable>|..etc

variable is from
Code: [Select]
local DATATYPE_TEMPERATURE = 1
local DATATYPE_HUMIDITY = 2
local DATATYPE_ENERGY_WATTS = 3
local DATATYPE_ENERGY_KWH = 4
local DATATYPE_ENERGY_UWATTS = 5
So I monitor energy-watts with config "12:3|14:3|102:3|102:4|89:3"
Title: Re: MQTT Client implementation
Post by: 4integration on May 25, 2014, 02:49:21 pm
Noted I didn't post the dependencies....MQTT libs and JSON I use in the code.
I have those files in "usr\lib\lua"
Title: Re: MQTT Client implementation
Post by: averell on May 25, 2014, 05:27:52 pm
4integration,

Thanks, I will reuse some of the stuff you did, I'm also a lua newbie, not aiming for the cleanest code, just for working code.
 @xappo: I do plan to publish what I make in a short time, for now I'm still figuring out too much, and all still is too messy with hardcoded stuff, lot's of logs etc. once I know the layout, and clean up the code, I will put it all here. I will try to get the event stuff worked out, the code examples suggest it is not a lua limitation as such, but I still have to learn a bit more about how lua on the vera works to know how I may fix it. I agree that polling is messy :)

Av
Title: Re: MQTT Client implementation
Post by: averell on May 28, 2014, 01:12:28 pm
@4integration:
I was thinking about also using json to publish messages to MQTT, but I can't reproduce the way you use the JSON module, on calling json:encode, I get:
 LuaInterface::CallFunction_Startup-1 device 106 function startup failed [string "..."]:62: attempt to call method 'encode' (a nil value)
Are you sure the JSON module you put in your zipfile is the one that is running on your vera?

av
Title: Re: MQTT Client implementation
Post by: 4integration on May 29, 2014, 06:03:06 am
@averell
I downloaded the files again and also tried to remove JSON.lua from /usr/lib/lua and then I get
Code: [Select]
LuaInterface::CallFunction_Variable func: watchSensorVariable Device_Variable 102 urn:micasaverde-com:serviceId:EnergyMetering1:Watts failed [string "module("L_SensorMqtt1", package.seeall)..."]:77: attempt to call method 'encode' (a nil value)Putting it back again it works, so it's the JSON I use

Have you put it into that directory?
Do you load the lib?
Code: [Select]
package.loaded.JSON = nil
json = require("JSON")

Title: Re: MQTT Client implementation
Post by: averell on May 30, 2014, 09:38:00 am
@ 4integration:
I didn't put JSON.lua in that directory, so that might just be it, I just uploaded it like the other files. I didn't get any arrors on the require part, so it seemed to load ok. I will try it again, I just wrote my own small concatenation to create a JSON string, snd was now working on the other end, displaying the events using angular.js.
I will give it another try, thanks for your reply  :)

My framework is now shaping up, I implemented 3 (actually 4) things now. The first is sending out events for all zwave devices that are directly controlled by the Vera. That way my central component on node.js will know those statusses. Works quite will, except for Duwi dimmers, they don't report back at all if I manually change the lights.
The second is updating the display values of devices such as temperature and power devices. They get updated every few seconds or so. The third one is sending out direct command for some devices, in my case a few switches and shutters that I control from my arduino devices. Here I didn't use the events only, but implemented the actions per child device. A fourth function I implemented only for making sure both controllers (Vera and my node.js based solution) stay in sync, which is an inform request over MQTT, which leads to the vera status of all devices to be sent over MQTT.
Directly switching on devices on vera I still do via HTTP, as long as the LUA implementation of MQTT client is still not asynchronous (which I feel might be the case forever).

As I'm working on both node.js, angular.js, learning LUA and learning about the vera, and also building the arduinos and my raspberry pi, it might be a while before it all works the way I want, but I'll try to copy a still very rudimentary version of the plugin asap.

Av
Title: Re: MQTT Client implementation
Post by: Matlock on October 08, 2014, 02:56:10 am
@averell - any chance you'd be able to share your code for the Vera side of things?  I'm also building a central "coordinator" for my home control stuff with MQTT for message passing.  I just picked up a Veralite to add to my other stuff, and it sounds like you've pretty much implemented exactly what I'd like to be able to do - just have the Vera publish all its events via MQTT, so I can consolidate them with my central control.  Triggering controls on the Vera is fine with the web api...

Matt
Title: Re: MQTT Client implementation
Post by: bside on January 04, 2015, 07:04:26 am
I'm with Matlock on this one, any chance of getting your stuff shared, I want to be able to publish mqtt events when things happen in Vera as well...
Title: Re: MQTT Client implementation
Post by: bluman on January 10, 2015, 06:08:10 am
Hows the progress on the MQTT plugin going? Im keen to use MQTT with Vera also.
Title: Re: MQTT Client implementation
Post by: Lockzi on March 19, 2015, 06:04:38 pm
Me too. If possible please share.

I even registered to write back here. :)

Another way of doing it for me as I already have OpenHAB running since about 6 months is to simply use the MIOS binding. Through that I could get OpenHAB to MQTT the status changes.

Topic subscribed.
Title: Re: MQTT Client implementation
Post by: davidsmoot on May 15, 2015, 03:12:54 pm
Interested too.  Replying to subscribe...
Title: Re: MQTT Client implementation
Post by: chongsiang on October 25, 2015, 10:40:29 am
just wonder does any one have any user guide for setup mqtt in vera 3
Title: Re: MQTT Client implementation
Post by: Snaxmuppet on December 17, 2015, 04:23:44 am
Just setting up a new HA system that integrates with Open Energy Monitor and some Arduinos and so I would benefit a lot from having MQTT available on my new Vera Edge.

Where are we with a working MQTT interface?
Title: Re: MQTT Client implementation
Post by: SchattenMann on January 19, 2016, 08:36:01 am
This may help http://forum.micasaverde.com/index.php/topic,35848.0.html