We have moved at community.getvera.com

Author Topic: openLuup: Cameras  (Read 3900 times)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
openLuup: Cameras
« on: March 14, 2018, 11:30:54 am »
The latest release of openLuup (v18.3.14) includes a built-in SMTP server to handle email messages, and a special implementation file for cameras which use this to trigger an associated motion detector device.

SMTP (Simple Mail Transfer Protocol) server:

This is a minimal implementation of an RFC 5321 compliant server (essentially, an original RFC 821 implementation) without authentication or Transport Layer Security.  It only handles messages within the LAN sent to a specific TCP port (2525 by default) and does not relay them further, except to pass them to internal handlers, each of which may be registered to receive emails from a specific email address.

I_openLuupCamera1.xml implementation file:

A camera device created with this implementation file will create an associated child Motion Sensor device which is triggered when the camera's own motion detection algorithm sends an email. 

Configuration:

Out of the box, openLuup will start the SMTP server on port 2525.  This can be changed in Lua Startup code with the following line:

Code: [Select]
luup.attr_set ("openLuup.SMTP.Port", 1234)    -- use port 1234 instead

The camera's device implementation file may be set on the openLuup device's Attributes page, followed by a Luup reload.  The only other significant parameters are the usual: ip attribute, and the URL and DirectStreamingURL device variables.

Camera configuration is obviously device-specific.  For my Foscam camera (thanks to @Spanners) the important parameters are:
  • Enable - ticked
  • SMTP Server - the IP address of openLuup on your LAN eg. 172.16.42.156
  • SMTP Port - 2525, or whatever other port number you configured in openLuup startup
  • Need Authentication - No
  • SMTP Username / Password - not used
  • Sender Email - must include the form xxx@yyy, for example Foscam@Study.local
  • First Receiver - openLuup@openLuup.local

My camera (FI9831P) also sends three snapshots as email attachments.  These are currently ignored, but could easily be written to a folder accessible from openLuup.

The Motion Sensor device will remain triggered for 30 seconds (or longer if the camera is re-triggered within that time.)  In keeping with the latest security sensor service file, in addition to the Tripped variable, there is also an ArmedTripped variable which is only set/reset when the device is armed.  This makes AltUI device watch triggers easy to write when wanting only to respond when the device is actually armed.
« Last Edit: March 18, 2018, 02:57:56 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #1 on: March 23, 2018, 10:52:27 am »
The latest master release of openLuup (v18.3.23) includes some enhancements to the I_openLuupCamera1.xml implementation file, and some additional openLuup plugin actions to support file management.

The ArchiveVideo action may be used to save single camera snapshots to openLuup's images/ folder.  At present, the Format and Duration parameters are unused (since this is currently only implemented for single frames.)

A new openLuup email address images@openLuup.local may be used as a destination address for camera trigger emails, instead of openLuup@openLuup.local.  Messages sent to this address will be examined for attachments, and any images written to the image/ folder in the openLuup home directory.

This has been tested with a Foscam camera, which attaches three images per trigger event.  The filenames are specified by the camera in the multipart ContentType header of the email message, which includes a clause like name="Snap_20180323-115529-0.jpg".  Other cameras may use a different approach, but I'm more than happy to try and accommodate any different formats.

This feature brings with it the possibility of generating quite a large number of files, in due course.  As a result I've found it helpful to include some sort of file retention policy implementation.  There are two new openLuup plugin actions:


serviceId = "openLuup", action = "SendToTrash"

This action moves selected files to openLuup's trash/ folder, and has four parameters:
  • Folder - the path of a folder below the openLuup home directory.  For the application described here, this should be images
  • MaxDays - the maximum age in days of files which should be retained
  • MaxFiles - the maximum number of files that should be retained
  • FileTypes - a string listing the types the types of files to which this applies, for example jpg gif bmp.  It may also be set to * in which case, any file is applicable.  Note that this does not act on subfolders or their contents.
One or both of MaxDays and MaxFiles may be specified.  If either of the conditions is true, then the file is moved to trash.

Moving files to the trash folder is reversible (manually!) but to delete them permanently, you may use the following action.


serviceId = "openLuup", action = "EmptyTrash"

This action has one parameter:
  • AreYouSure - needs to have the value "yes" (case-insensitive) to delete all the files in openLuup's trash/ folder.

The purpose of the AreYouSure parameter is to avoid accidental presses of the EmptyTrash button on the device actions page having any effect.


The above actions are easily included into scheduled scenes which might, for example, run the SendToTrash action early in the morning every day, and the EmptyTrash every weekend.


__________

Edit: corrected implementation file name!
« Last Edit: March 24, 2018, 03:51:04 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: openLuup: Cameras
« Reply #2 on: March 24, 2018, 02:53:02 pm »
I'm missing something obvious: where is the I_openLuupCamera.xml file?  I've loaded the latest master release, but I don't see that file?  I don't see it in github either.

I can see my Amcrest try to connect to the SMTP server.  Amcrest thinks it failed, but that's probably because I'm missing the implementation file.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #3 on: March 24, 2018, 03:09:19 pm »
It's not a physical file.

Just specify it in the create device process and openLuup will find it.

To convince yourself that it's there, try the following URL in your browser (with the appropriate IP address):

Code: [Select]
http://openLuupIP:3480/I_openLuupCamera1.xml

Without that file for the device, no motion sensor will be created.

__________

Edit: it probably didn't help that I left out the trailing '1' in my second post below!

« Last Edit: March 24, 2018, 03:48:36 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #4 on: March 24, 2018, 05:24:19 pm »
You can test the SMTP server easily from any unix-like machine on the network (here, I'm doing it on the same machine)...

Code: [Select]
% nc 127.0.0.1 2525
220 (openLuup.smtp v18.3.24) [0.0.0.0] Service ready
helo
250 OK
mail from:akb@here
250 OK
rcpt to:test@openLuup.local
250 OK
data
354 Start mail input; end with <CRLF>.<CRLF>
this is a test message.
.
250 OK
quit
221 (openLuup.smtp v18.3.24) [0.0.0.0] Service closing transmission channel
%

The all lower-case lines are user input.  You have to mail TO: a recognised email address - the console SMTP server page will list those.   The FROM: address can be anything of the form xxx@yyy.
 
Entries in the log file should be something like this:
Code: [Select]
2018-03-24 21:52:25.218   openLuup.smtp:: new SMTP client connection from 127.0.0.1: tcp{client}: 0x1999978
2018-03-24 21:53:07.328   openLuup.smtp:: EMAIL delivered to handler for: test@openLuup.local
2018-03-24 21:53:10.968   openLuup.smtp:: SMTP QUIT received tcp{client}: 0x1999978

« Last Edit: March 24, 2018, 05:55:13 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: openLuup: Cameras
« Reply #5 on: March 24, 2018, 05:33:00 pm »
Needs some sort of device file too, right?  Do you use D_DigitalSecurityCamera1.xml?  Or another custom version?

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #6 on: March 24, 2018, 05:50:02 pm »
The standard version is fine.  Sorry about not being specific on that.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: openLuup: Cameras
« Reply #7 on: March 25, 2018, 06:01:01 pm »
My camera seems to connect to the smtp port, but apparently it doesn't like the interaction.  The camera says the test email fails, but openLuup doesn't seem unhappy.  All I see in the log is this:

Code: [Select]
2018-03-25 14:55:09.108   openLuup.smtp:: new SMTP client connection from 192.168.1.202: tcp{client}: 0x1d6f360
2018-03-25 14:55:09.119   openLuup.smtp:: SMTP QUIT received tcp{client}: 0x1d6f360
2018-03-25 14:55:20.790   openLuup.smtp:: new SMTP client connection from 192.168.1.202: tcp{client}: 0x2261360
2018-03-25 14:55:20.800   openLuup.smtp:: SMTP QUIT received tcp{client}: 0x2261360

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #8 on: March 25, 2018, 06:09:52 pm »
Need to switch on full debugging for the module to see the handshake.

The latest version (not yet released to the development branch) would then write each step to the log.  There's also a test@openLuup.local destination which would dump the whole of the delivered message.  I'll post that tomorrow. 

Right now, though, what does the console / Servers / SMTP page show?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #9 on: March 25, 2018, 06:19:10 pm »
Another thought...

...I've just remembered that the test from my camera also failed, because it tried to send email to its own address via the server.  Since the camera's own email is not one of the defined openLuup mailboxes, this fails.

Have you tried with an actual video trigger event (which seems to use the supplied destination address correctly)?

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: openLuup: Cameras
« Reply #10 on: March 25, 2018, 06:36:59 pm »
console log below.  Tried a real trigger, but the log remains the same.  I wonder if my camera is trying to login even if not needed so the handshake fails?  I don't have any option to not login - it says "anonymous login" or the user/password boxes.  Of course there is no information on exactly what the camera means by "anonymous login".

Code: [Select]
SMTP eMail Server, Sun Mar 25 15:30:50 2018 

 Received connections: 
     IP address                       #connects     date     time
 
     192.168.1.202                         17   2018-03-25 15:28:56 

 Registered email sender IPs: 
     IP address                       #messages for device
 
     192.168.1.202                          0   [22] Deck Camera 

 Registered destination mailboxes: 
     eMail address                    #messages for device
 
     test@openLuup.local                    0   [0] system 
     images@openLuup.local                  0   [2] openLuup 
     postmaster@openLuup.local              0   [0] system 
     openLuup@openLuup.local                0   [2] openLuup

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #11 on: March 26, 2018, 03:01:41 am »
console log below.  Tried a real trigger, but the log remains the same.  I wonder if my camera is trying to login even if not needed so the handshake fails?  I don't have any option to not login - it says "anonymous login" or the user/password boxes.  Of course there is no information on exactly what the camera means by "anonymous login".

You may be right, perhaps I will have to add authorization to the server.  This should not be so if the camera SMTP client is RFC compliant.

So we need a little more debug info.  Please update to the development branch latest (18.3.26) and add the following to your Lua Startup, then reload.

Code: [Select]
do -- SMTP debug
    local smtp = require "openLuup.smtp"
    smtp.ABOUT.DEBUG = true
end

Repeat the test (both the test button and an actual camera trigger) and let's see what ends up in the log.

It may be that the client gives up the moment that it discovers the server doesn't support the authorization extension, or we may get a little further into the transaction and discover that a destination email address in not accepted.  Or something else ...?

Sorry for the trouble, but only having one Foscam camera, I can't test this myself.
« Last Edit: March 26, 2018, 06:43:35 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #12 on: March 26, 2018, 03:22:56 pm »
I have added a basic SMTP authentication method to the latest development branch (18.3.26b)

I've been playing with the authentication on my Foscam and it turns out that it works with this method.

Changes include:
  • SMTP basic AUTH LOGIN authentication
  • In DEBUG mode (described in previous post) the STMP module will accept email to any mailbox address
  • Any Username and Password will be accepted
  • Does not use Transport Layer Security (TLS / SSL)

With these changes, the TEST mode on the camera's email setup page works and authentication works for actual triggers.

Of course, your mileage may vary, but the enhanced debugging should point the way if further changes are needed.   The most likely thing is that I have to implement further protocols, the PLAIN one being the most likely.  But please give it a go and see if it works.  If not, then the logs should be useful.  Thanks.
« Last Edit: March 26, 2018, 05:28:54 pm by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline jswim788

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +58/-2
Re: openLuup: Cameras
« Reply #13 on: March 26, 2018, 11:43:08 pm »
Much better so far.  The test email works fine.  Unfortunately I am no longer physically near this camera so I can't make it trigger, but I left it pointing out a window at a high traffic area so it should get a trigger tomorrow and we'll see how it does.  I'm guessing it will work!

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup: Cameras
« Reply #14 on: March 27, 2018, 03:01:49 am »
Oh, that's good news!

Did you use anonymous login, or just pick a username and password?

If it continues to go well, then don't forget to remove the debug statement in your Lua Startup.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.