We have moved at community.getvera.com

Author Topic: Capturing Vera logs to my own FTP server instead of Vera  (Read 1584 times)

Offline niharmehta

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +17/-0
Capturing Vera logs to my own FTP server instead of Vera
« on: April 03, 2016, 01:29:33 pm »
I have been a little frustrated with the lack of flexibility in how Vera logging.  However, for some reason I could never easily capture a log entry of a problem I am trying to troubleshoot  even with USB logging since for some reason only 1 archive of the log is kept.  ( If someone knows of a simple way to keep 10 archives of logs on USB,  please share).

 In searching forums, I could not find if others have shared this previously.  Taking advantage of the complete lack of security with vera around this, I discovered an extremely simple way of capturing all of logs on my server vs sending to Vera. This basically tricks Vera in to thinking your FTP server is their own.

1) Log in to your Vera and copy the FTP login credentials with the logs section of /etc/cmh/servers.conf
Code: [Select]
root@MiOS_PRIVATE:/etc/cmh# more servers.conf | grep Log
## Log
Server_Log=vera-us-oem-log11.mios.com
Server_Log_Alt=vera-us-oem-log12.mios.com
Use_Server_Log_Alt=0
Server_Log_User=logs
Server_Log_Pass=XXXX


Using whatever FTP server you want to use,  create a login with user "logs" and password with whatever you found in the above file.
In the FTP root directory, create a subdirectory named the serial# of your Vera found in the Unit Settings section or from the home portal . The name should just be the serial#.    Make sure this is writable by the "logs" user.


Back on the Vera,  edit the /etc/hosts file  and add entries for the log server names  and use your server IP address as the destination. I also added the old names for good measure :
Code: [Select]
root@MiOS_PRIVATE:/etc# cat /etc/hosts
127.0.0.1 localhost
192.168.14.94 local.mios.com local.my.mios.com
192.168.1.5 logs1.mios.com
192.168.1.5 logs2.mios.com
192.168.1.5 vera-us-oem-log11.mios.com
192.168.1.5 vera-us-oem-log12.mios.com
root@MiOS_PRIVATE:/etc#

Next, if not already selected within the logs section of the Vera GUI,  select "Archive old logs on MiOS (recommended)"

To test,  force a log Rotate which should push logs to your server:
Code: [Select]
root@MiOS_PRIVATE:~# /usr/bin/Rotate_Logs.sh --force
=== Start rotating logs PID=8911, force rotate=1, free space=127232 KB, min=5000 KB, cmh logs size=40 KB, max=19154 KB, now=1459704372, last rotation=1459704355, diff=0 min 17 sec, max=12 hours ===
=== Successfully uploaded logs for serial=PRIVATE, version=1.7.1754, server=vera-us-oem-log11.mios.com, date=2016-04-03_10-26-12 ===

That is all.  You will start seeing logs populate every few minutes within your FTP server.  Since there is no true server authentication taking place, your Vera will happily send your logs to your own server. 

Note: I doubt this will survive a firmware upgrade (I have not tested yet).  After a firmware upgrade you may need to update your  /etc/hosts file again to point to your own server.  If you ever want to restore back and archive to Vera, just comment out the entries in the /etc/hosts file.

Hope this helps




2x VeraLite; 2xTrane Tstats; 45 x Switches/Dimmers/Appliance Modules; 4x Everspring Water Sensors; DSC Integration; 2 x Zwave Door Locks; 1x Ted5K; 1x Rainforest Eagle; Onkyo AVR; 6x Squeezebox;

Offline Catman

  • Sr. Member
  • ****
  • Posts: 338
  • Karma: +10/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #1 on: February 28, 2019, 03:08:10 am »
I took a slightly different tack.
Modify /etc/cmh/servers.conf

# Log
Server_Log=ftp.ipaddress.whatever
#Server_Log_Alt=oem-log4.mios.com
Use_Server_Log_Alt=0
Server_Log_User=xxxxxx
Server_Log_Pass=xxxxxx


*BUT* this file might be re-written at Lua-restart. Or at least partially as I've just gone to write this and teh ip address has been altered back to the FQDN of the vera server.  User / Pass were unaltered.

Also I had to modify Rotate_Logs.sh to account for the login location of my ftp server:

log "Sending file with CMD: curl -s -S -f --connect-timeout 60 --max-time 300 -u \"$ftpUser:$ftpPass\" -T \"${logFile}.gz\" \"ftp://$ftpServer/array1/Catman/VeraLogs/${pkap}/${pkap}_$(echo $logFile | awk -F '/' '{print $NF}').gz\""
                        sendOk=0
                        try=0
                        while [ $((++try)) -le 3 ]; do
                                log "Try $try/3"
                                curl -s -S -f --connect-timeout 60 --max-time 300 -u "$ftpUser:$ftpPass" -T "${logFile}.gz" "ftp://$ftpServer/array1/Catman/VeraLogs/${pkap}/${pkap}_$(echo $logFile | awk -F '/' '{print $NF}').gz" 1>>$log_file 2>&1

I'll check after the next restart. The resolv.conf option may be better!

Cheers

C

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #2 on: February 28, 2019, 04:40:43 am »
Hi,

After disabling the option Archive old logs on MiOS and since then it keeps ten log files in gz format. Not just the LuaUPnP.log but a whole bunch.

So if you just need an occasional check you can think of toggeling that option.

Cheers Rene
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline Catman

  • Sr. Member
  • ****
  • Posts: 338
  • Karma: +10/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #3 on: February 28, 2019, 05:28:08 am »
Hi,

After disabling the option Archive old logs on MiOS and since then it keeps ten log files in gz format. Not just the LuaUPnP.log but a whole bunch.

So if you just need an occasional check you can think of toggeling that option.

Cheers Rene

Indeed.  Less of a problem now I have SSD but not a massive fan of the rotation anyway.  Working out which one you need to gunzip etc etc.  10 logs is not that far on the normal rotation schedule either.
Under FTP I've changed (I hope) to 24 hour rotation and I'm getting datestamped logs up on my SAN which also makes it quick and easier to do analysis as the versions of grep etc on my other hosts are rather better than on the Plus itself ;)

C

Offline Catman

  • Sr. Member
  • ****
  • Posts: 338
  • Karma: +10/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #4 on: February 28, 2019, 05:30:52 am »
Oh I might make some changes to the naming format as well, and possibly the upload directory. Having every log prefaced with the unit name is clearly a win for Mios support. Not so much for a single user.
Datestamp postfix less useful as well but...

C

Offline niharmehta

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +17/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #5 on: March 01, 2019, 09:14:06 pm »
Several good ways ways to solve the problem. For me, it was easier to just tinker with the hosts file and only manage that file when FW updates invariably change the contents. On my server, I just had the FTP file  root for the user to be the directory I wanted the logs deposited. 

Good call on the increasing the rotation length since you have an SSD already. Reduces the number of files filling up the FTP server. Also, I have had a few instances where a script lock file sticks, and rotations fails until I manually  delete it.
2x VeraLite; 2xTrane Tstats; 45 x Switches/Dimmers/Appliance Modules; 4x Everspring Water Sensors; DSC Integration; 2 x Zwave Door Locks; 1x Ted5K; 1x Rainforest Eagle; Onkyo AVR; 6x Squeezebox;

Offline Florin

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #6 on: March 02, 2019, 11:01:49 am »
I've used scp with keys. Created a script in /tmp/log/cmh that copies the files and then executes rotate_logs. Then changed the cron to execute my script instead of rotate_logs. I also changed the time from every minute to every 5 minutes and everything works perfect.

Offline rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1121
  • Karma: +187/-3
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #7 on: March 02, 2019, 11:06:19 am »
My version (Florin posted while I was writing--our approaches are similar):

Goal: copy log files during normal log rotation to local *nix server via SCP, to unprivileged (with login/shell-disabled) account on server.

Prerequisites: Linux server to receive files, account created with restrictions mentioned above, and you have your SSH server set up for public key authentication (many have this on by default).

1) SSH into Vera and make a new key for this purpose:
    dropbearkey -t rsa -f /etc/dropbear/rigpapa_rsa_key
    This will create a file in /etc/dropbear/ with the private key, and print the public key on screen.

2) Copy-pasta the public key into ~/.ssh/authorized_keys on the Linux server, creating the subdirectory and file if necessary (~ refers to the home directory of the account created to receive the files).

3) Test (from Vera): scp /var/log/cmh/LuaUPnP.log user@host:~/
    This should work without the need to enter a password for host.

4) Create the following script file in /etc/cmh-ludl/. I call mine /etc/cmh-ludl/local-upload-logs.sh (and the balance of these instructions assumes that name):
Code: [Select]
#!/bin/sh

# Change the rigpapa_rsa_key to whatever filename you used in step 1.
scp -i /etc/dropbear/rigpapa_rsa_key /var/log/cmh/LuaUPnP.log user@host:~/LuaUPnP-`date +%F.%T`.log
Note the use of back-ticks around the date command--that's not a single quote, it's ASCII 96. I recommend copy-paste from here to get the right character.

5) Remove the symbolic link for the system's Rotate_Logs.sh and replace it with the real file so we can edit it (by default, it is linked to a file on a read-only filesystem):
    cd /usr/bin
    cp Rotate_Logs.sh Rotate_Logs.sh-orig
    rm Rotate_Logs.sh
    mv Rotate_Logs.sh-orig Rotate_Logs.sh

6) Now modify the Rotate_Logs.sh file to run the script created in step 4. Open the file in your favorite available text editor (I personally have no trouble with vi as I have decodes of muscle-memory connected to it):
    a) Find the line that reads: log "Start log rotation"
    b) Add the following line immediately after:
        [ -x "/etc/cmh-ludl/local-upload-log.sh" ] && /etc/cmh-ludl/local-upload-log.sh

7) Make the rotate script executable: chmod a+rx /etc/cmh-ludl/local-upload-log.sh

8.) Test: Rotate_Logs.sh 1 or Rotate_Logs.sh --force
    You should find a fresh copy of the log file on your Linux server in the target account's home directory.

I put the actual copy command in a separate script because I often like to run it manually just to push the current log up to the server without actually doing a full rotation (so I can just type /etc/cmh-ludl/local-upload-logs.sh as a command and done).
« Last Edit: March 02, 2019, 09:27:25 pm by rigpapa »
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3, Lite. Hassio, Slapdash.

Offline akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +291/-70
  • "Less is more"
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #8 on: March 02, 2019, 11:13:13 am »
I was thinking of adding an FTP server to openLuup and redirecting Vera's output to there.   Could then be browsed in the same way as openLuup's own logs.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #9 on: March 02, 2019, 11:19:05 am »
My 2 cents: is there any option to change the ftp url to something local? So I can alter my internal DNS server to my ftp server? And optionally to change the ftp server address to my internally in dns registered ftp server...?
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline Florin

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #10 on: March 02, 2019, 02:26:07 pm »
My version (Florin posted while I was writing--our approaches are similar):


Thanks for detailing the solution  :D
The difference in my version is that I don't touch the rotate_logs.sh script. I just call it at the end of my script (local-upload-logs.sh in your version) instead of modifying it to include my code.

And the crontab line
#*/1 * * * * /usr/bin/Rotate_Logs.sh #Rotate_Logs
was replaced with
*/5 * * * * /tmp/log/cmh/copy-log.sh #copy&rotate logs

I saw some old threads with a similar subject. Some users complained that by changing the rotate logs script, the logs either stopped rotating or they were touched every minute but not emptied. I figured I can skip some problems by letting that script untouched.

Copying works like a charms and already I have a ton of logs to parse.

Offline Catman

  • Sr. Member
  • ****
  • Posts: 338
  • Karma: +10/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #11 on: March 02, 2019, 03:46:52 pm »
My 2 cents: is there any option to change the ftp url to something local? So I can alter my internal DNS server to my ftp server? And optionally to change the ftp server address to my internally in dns registered ftp server...?

Do you mean the Vera log server URL? Then yes, it's trivial

Edit /etc/hosts on your Vera

local.ip.address oem-log3.mios.com
local.ip.address oem-log5.mios.com

You'll need to check in /etc/cmh/servers.conf (Log section) that that's the FQDN of the servers you're configured to use.

Then you either set the user / pass on your local ftp server to the same as that specified in the Log section of servers.conf, or edit servers.conf to meet your user / pass on the ftp server.

So sounds actually a bit more than trivial, but it is really :)

Does that help?

C

Offline Sender

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +552/-467
  • Sr. manager of my Vera
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #12 on: March 03, 2019, 02:55:20 pm »
Hi, I meant that if akboer would make some alterations on vera I would be able to change the IP or DNS names of the vera logservers more easy (via altui) instead of chaging config files in vera...
Good firmware is more important than good karma. (and this costed me lots of Karma)

Offline Catman

  • Sr. Member
  • ****
  • Posts: 338
  • Karma: +10/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #13 on: March 04, 2019, 04:14:02 am »
Hi, I meant that if akboer would make some alterations on vera I would be able to change the IP or DNS names of the vera logservers more easy (via altui) instead of chaging config files in vera...

Ahh I see!

C

Offline niharmehta

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +17/-0
Re: Capturing Vera logs to my own FTP server instead of Vera
« Reply #14 on: March 05, 2019, 08:18:07 pm »
Nice job on the SCP option.  If only Vera used SCP to upload logs like you have done with the trivial changes, instead of the unencrypted FTP they have (and still use) for years to their servers.  I have posted in the past a detailed breakdown of the type of data that is sent in the logs, and it is quite sensitive information (including security system pin codes)
2x VeraLite; 2xTrane Tstats; 45 x Switches/Dimmers/Appliance Modules; 4x Everspring Water Sensors; DSC Integration; 2 x Zwave Door Locks; 1x Ted5K; 1x Rainforest Eagle; Onkyo AVR; 6x Squeezebox;