Author Topic: UDP datagrams  (Read 656 times)

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
UDP datagrams
« on: April 18, 2018, 01:17:04 am »
I have several cameras that transmit data every 30 seconds as a binary UDP datagram of 132 bytes.  The data packet contains several fields, one of which describes if motion has been detected by the camera's motion detector.

I read the literature for datayours but wasn't sure if the plugin would be suited to capture this data, scan it for the proper motion codes, and then act on the scan (basically http gets). In your humble opinion ;D,  is this the right tool to use (on openLuup) or should I go the Apache/PHP listener route using a separate server.  It wasn't clear to me if the plugin specifically targeted Veras, .... or if it could be used for any datagram on ones network.  And to add, i really dislike PHP......

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #1 on: April 18, 2018, 02:45:44 am »
The UDP datagram used by DataYours has a very specific format and semantics (Whisper plaintext format, an industry standard.)

However, the code to receive UDP datagrams is quite generic, and, strangely enough, I am about to implement a general-purpose UDP watch callback in openLuup.  This means that you could simply (a one-liner) register your own callback routine to be called when a UDP datagram is received on a specific socket address.

What you'd put in the callback routine would be up to you, but it should just be a couple of lines of code to, say, trigger a device.  Sounds like fun.  Let's do this!

What camera is it, so that I can download the docs?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #2 on: April 18, 2018, 05:20:14 pm »
Awesome!  I have several camera brands--mostly Acti, but the camera I'm working on is an American Dynamics security camera. You won't find any info on the particular model (ADCIPE3712OCN) as I had to scour the net, cobbling together info for a basic understanding of the camera's functionality.  I ended up capturing exposed javascript from the camera's web server to get at its motion detection feature.  My NVR doesn't recognize the camera's packets, but the NVR does have an API that accepts Http calls to start and stop recording.  So that's essentially where I'm going with this.  Perhaps some HA calls as well.

Attached is a pdf (that was in the camera) related to the motion detection, and a map that I created of the data.  The pdf description of the data is slightly off, while the map is an accurate description of the data structure.  The underlying camera data is binary, requiring a bin2hex conversion to get it to look like the data map.  If you want a wireshark capture, let me know and I'll pm you with the data as it of course has some network sensitive information buried in the packets. 

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #3 on: April 19, 2018, 06:17:52 am »
openLuup development commit 2018.04.19 now supports UDP datagram callbacks...

...so all that is now required is a few lines to translate the datagram format you have discovered into useful Lua data.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #4 on: April 19, 2018, 10:58:49 am »
...so all that is now required is a few lines to translate the datagram format you have discovered into useful Lua data.

So the format is a bit messy, but here's a decode function which will do the trick:

Code: [Select]
local function decode (data)
  local function integer (bytes)
    local i = 0
    for c in bytes: gmatch "." do i = i * 0x100 + c: byte() end
    return i
  end

  local fields1 = [[^(..)(..)(..)(..)(....)(....)(....)(..)(.)(.)(.)(.)(.)(.+)$]]
  local fields2 = [[^(......)(................)(.........)(.........)(.+)$]]

  local names1 = {
    "magic", "serial", "devtype", "msgtype",
    "event", "sub1", "sub2",
    "year", "month", "day", "hour", "min", "sec"}

  local names2 = {"mac", "title", "user", "password", "message"}

  local x = {data: match (fields1)}
  local y; y, x[#x] = {x[#x]: match (fields2)}

  local mac = {}
  for z in y[1]: gmatch "." do     -- format mac address
    mac[#mac+1] = ("%02x"): format (integer(z))
  end

  local info = {}
  for i,v in ipairs (x) do info[names1[i]] = integer(v) end       -- convert numeric data
  for i,v in ipairs (y) do info[names2[i]] = v: match "%C+" end   -- remove nulls from strings

  info.mac = table.concat (mac,':')
  return info
end

When applied to the example in your documentation with this code:
Code: [Select]
local info = decode(datagram)
print (pretty(info))

you get this structure:
Code: [Select]
{
  day = 20,
  devtype = 5747,
  event = 32,
  hour = 0,
  mac = "00:30:46:01:49:c2",
  magic = 21930,
  message = "KeepAlive",
  min = 22,
  month = 12,
  msgtype = 1,
  password = "SoleSole",
  sec = 30,
  serial = 37319,
  sub1 = 0,
  sub2 = 168887314,
  title = "WestSouth",
  user = "admin",
  year = 2016
}

I may not have interpreted your format table quite correctly, but you get the idea.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #5 on: April 19, 2018, 06:10:16 pm »
AK,  you have some mad skills.  Thank you so much.  The data output looks exactly as I translated it.  Your LUA approach will be so much more useful than had I written a listener/decoder in PHP.  I now can EASILY integrate my HA watches with my camera data, not to mention having one less RasPi to maintain.

I'll take a crack at integrating the function into something workable over the weekend. 

Thanks again.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #6 on: April 20, 2018, 02:03:01 am »
AK,  you have some mad skills.

Oh, sorry if that was a bit exotic.

Here's a more conventional, but still stylish, way to do it.  Perhaps easier to modify if you need to apply it elsewhere:

Code: [Select]
local function decode2 (data)
  local function str (x) return x: match "%C+" end
  local function hex (x) return  ("%02x"): format (x: byte()) end
  local function num (x) return tonumber(x: gsub ('.', hex),16) end
  local function hac (x) return hex(x) .. ':' end
  local function mac (x) return (x: gsub ('.', hac)): sub (1, -2) end

  local fields = {
    {"magic",2}, {"serial",2}, {"devtype",2}, {"msgtype",2}, {"event",4},
    {"sub1",4}, {"sub2",4}, {"year",2}, {"month",1}, {"day",1}, {"hour",1}, {"min",1}, {"sec",1},
    {"mac",6,mac}, {"title",16,str}, {"user",9,str}, {"password",9,str}, {"message",64,str}}
   
  local info, n = {}, 1
  for _,field in ipairs (fields) do
    local name, width, form = unpack(field)
    info[name] = (form or num) (data: sub(n,n+width-1))
    n = n + width
  end
 
  return info
end
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #7 on: April 21, 2018, 02:25:21 am »
Very cool.  Each camera (there are 4) is time-synced via local NTP, and each will send a keep alive datagram every 30 seconds (as well as intermittent start and stop motion datagrams).  Do I need to point to unique ports (per camera) or can the listener handle multiple datagrams coming in at the same time. 

I'm guessing the bulk of the UDP code is in io.lua.  Do I just follow your function params.

Thx again.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #8 on: April 21, 2018, 08:42:24 am »
Do I need to point to unique ports (per camera) or can the listener handle multiple datagrams coming in at the same time. 

You just need one.   The port you listen on has a queue of requests, and your callback routine processes them sequentially.  For UDP, these requests are all atomic and not transaction based (essentially, just one-way) but for others, like HTTP and SMTP, etc., there can be many requests running in parallel and the processing is interleaved.
 
Quote
I'm guessing the bulk of the UDP code is in io.lua.  Do I just follow your function params.

Not wise to probe too much the internals of openLuup.  The 'external' API is almost exclusively through standard Luup HTTP requests or luup.xxx() Lua calls.  In this case, it's luup.register_handler()  The documentation, such as it is, for the UDP callback is here: Development Branch: 2018 Release 4.19

Unless you're planning on writing a full-blown plugin (which I don't recommend at the outset) then you should put your Lua code into Lua Startup, using the AltUI > Misc > Lua Startup editor.  All you need is your callback routine and a single one-time call to register it as a UDP handler...

Code: [Select]
function myUDPcallback (port, data)
  -- port will be whatever you registered the callback to listen for
  -- you'll need to define your decode routine somewhere here too

  local info = decode(data.datagram)
  if info then
    -- you can detect which camera from a number of the info variables, eg mac or title, or sender 's data.ip
    if info.title == "WestSouth" then
      -- whatever you need here, including HTTP commands using luup.inet.wget()
    end
  end 
end

luup.register_handler ("myUDPcallback", "udp:2345")    -- listen for UDP on port 2345


Note that, as with all Luup callback functions, the routine needs to be defined as a global.  You'll need to reload explicitly every time you edit the startup code.  Ignore the AltUI message saying that you have to wait for a reload... it doesn't happen (because I always found that frustrating.)

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #9 on: April 21, 2018, 06:19:26 pm »
Ah, my bad; I didn't read the release notes.  Looks good. I can imagine many different uses for an extended register_handler.  Great work.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #10 on: April 27, 2018, 03:51:32 am »
I have the callback working now and it runs very smoothly.  With one exception and I'm not sure if this is a luup issue, or if I need additional http headers for my NVR call.  Hoping you can shed some light on this.   Here is the basic code for the get call that fails. 

Code: [Select]
local NVRDeviceID = 7
local myStatus = luup.inet.wget('Http://xxxx:yyyyyyyy@10.xx.xx.xx:xxxx/Media/ManualRecord?deviceid=NVRDeviceID&action=start')
and this code does not work either:
Code: [Select]
local URL = 'Http://xxxx:yyyyyyyy@10.xx.xx.xx:xxxx/Media/ManualRecord?deviceid=NVRDeviceID&action=start'
local myStatus = luup.inet.wget(URL)

However, this code works with a hard coded deviceid:

Code: [Select]
local myStatus = luup.inet.wget('Http://xxxx:yyyyyyyy@10.xx.xx.xx:xxxx/Media/ManualRecord?deviceid=7&action=start')
I'm not sure if I can, or how to force a value for the NVRDeviceID variable in the call itself.  Which would explain many of the curl calls I see using variables, in various plugins....

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #11 on: April 27, 2018, 12:34:42 pm »
I have the callback working now and it runs very smoothly.  With one exception and I'm not sure if this is a luup issue, or if I need additional http headers for my NVR call.  Hoping you can shed some light on this.   Here is the basic code for the get call that fails. 

Ah, you need to brush up a bit on your Lua skills.

What you need is:
Code: [Select]
local NVRDeviceID = 7
local myStatus = luup.inet.wget('Http://xxxx:yyyyyyyy@10.xx.xx.xx:xxxx/Media/ManualRecord?deviceid=' .. NVRDeviceID .. '&action=start')

In fact, the luup.inet.wget() request can build the special authorization syntax for you, per the very sparse documentation here and so you can also use variables for username, password, and request timeout.
Code: [Select]
local NVRDeviceID = 7
local timeout = 5         -- in seconds
local user = "xxxx"
local pass = "yyyyyyyy"

local myStatus = luup.inet.wget('Http://10.xx.xx.xx:xxxx/Media/ManualRecord?deviceid=' .. NVRDeviceID .. '&action=start', timeout, user, pass)

Quote
I'm not sure if I can, or how to force a value for the NVRDeviceID variable in the call itself.  Which would explain many of the curl calls I see using variables, in various plugins....

As you see, you can.  It is really bad practice, and leads to poor performance, to invoke shell commands with os.execute() unless you really, really have to.  It's almost always unnecessary, and for many operations (eg. file rename, delete, directory traversal, ...) there are os-independent alternatives.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #12 on: April 27, 2018, 05:04:42 pm »
Thank you so much. 

I had tried using the concatenation operator, but I didn't have my quotes bracketing the variable properly.  That did the trick.  I also looked everywhere for some insight, but couldn't find anything that worked, and I knew from using a similar technique in SQL injection, that it was doable.  So I guess it is time to bear down and learn Lua.  It should pay off though as I can see so many possibilities to apply HA around my home.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6043
  • Karma: +264/-69
  • "Less is more"
Re: UDP datagrams
« Reply #13 on: April 28, 2018, 01:59:11 am »
So I guess it is time to bear down and learn Lua.

What you need is this: Programming in Lua

I have the third edition hardcopy which actually relates to Lua 5.2, but is clearly flagged where this differs from 5.1, which is what you need for Vera and openLuup.  The first edition is freely available from that site online and is largely relevant.  There is a fourth edition for Lua 5.3, but I haven't read it.

Once you have you head around that, a very useful and complete online resource is Lua 5.1 Reference Manual

It's a very small language syntactically, but extremely powerful, quite subtle, compact and fast, and runs just about anywhere.  (I like it!)

As a testament to that, openLuup is pure Lua, about 10,000 lines of code, and complete systems run in only about 5 - 10 Mbyte of memory (compare that to Java!)

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Buxton

  • Full Member
  • ***
  • Posts: 120
  • Karma: +8/-0
Re: UDP datagrams
« Reply #14 on: April 28, 2018, 06:58:01 pm »
Will do.  Thanks