We have moved at community.getvera.com

Author Topic: openLuup : uploading a file  (Read 3054 times)

Offline vosmont

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 687
  • Karma: +60/-8
openLuup : uploading a file
« on: February 09, 2016, 03:47:39 am »
Hello,

I would like to be able to upload a file via openLuup.
Legacy Vera use the script "upload_upnp_file.sh" in /cgi, but this script does not exist in openLuup (besides it would be a problem for windows configurations).

Another solution is to accept "POST" request for handlers (this doesn't work on Vera, and even sometimes crash the box)

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup : uploading a file
« Reply #1 on: February 09, 2016, 07:04:15 am »
You're talking about upload to /etc/cmh-ludl/ line the files menu in the Vera UI allows?  (obviously, without compression!)

...or to an arbitrary location?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup : uploading a file
« Reply #2 on: February 09, 2016, 07:16:15 am »
Actually, the files upload seems to call
Code: [Select]
http://VeraIP/cgi-bin/cmh/upload_upnp_file.sh

with a POST request.

At the moment, openLuup intercepts, but does not execute, cgi calls to port 3480.  It doesn't currently accept POST requests, but it could.

Alternatively, I could choose a complete different syntax (and perhaps we could ask @amg0 to give us an AltUI menu for this.)
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline vosmont

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 687
  • Karma: +60/-8
Re: openLuup : uploading a file
« Reply #3 on: February 09, 2016, 08:04:31 am »
The goal is to be able to upload files on the Vera in "/etc/cmh-ludl/",  from javascript client.
Then, a plugin get the uploaded file (eventually decompress it) and uses its content.

Offline vosmont

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 687
  • Karma: +60/-8
Re: openLuup : uploading a file
« Reply #4 on: February 11, 2016, 09:48:23 am »
Hello akbooer,

I have implemented this feature (quick and dirty).
It's inspired from https://github.com/keplerproject/wsapi




Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup : uploading a file
« Reply #5 on: February 11, 2016, 10:19:48 am »
Ah, I was looking at the Kepler project.  I wanted to implement multi-part POST requests but had no idea where to start.  The version of server.lua that you used is not quite the latest, but it will be easy to merge.

It's a real pain having to write a server - it must have been done a million times , mostly badly, as I have done - but it's really very closely integrated with the scheduler and I can't see a viable off-the-shelf candidate.  However, you seem to have filled a gap!  Many thanks - I need to go into testing mode.

How are you invoking it?  Do you need any integration with CGI files?
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline vosmont

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 687
  • Karma: +60/-8
Re: openLuup : uploading a file
« Reply #6 on: February 11, 2016, 11:07:16 am »
The file is directly upload from the javascript client (in ALTUI) :

an extract of the code that does that : (in J_ALTUI_RulesEngine1.js - plugin RulesEngine)
Code: [Select]
function _getCgiUrl() {
var protocol = document.location.protocol;
var host = document.location.hostname;
var httpPort = document.location.port;
var pathName = window.location.pathname;
var cgiUrl = protocol + "//" + host;
if (httpPort !== "") {
cgiUrl = cgiUrl + ":" + httpPort;
}

if ( pathName.indexOf( "/port_3480" ) !== -1 ) {
// Relay mode
pathName = pathName.replace( "/port_3480", "" );
}
cgiUrl = cgiUrl + pathName.replace( "/data_request", "" ) + "/cgi-bin/cmh";
return cgiUrl;
}

function _saveRulesFileAsync( fileName, xmlRules ) {
var d = $.Deferred();

var xmlRoot = $.parseXML( '<xml xmlns="http://www.w3.org/1999/xhtml"></xml>' );
var $xml = $( xmlRoot ).children(0);
_encodeCarriageReturns( xmlRules );
xmlRules.each( function( idx, xmlRule ) {
$xml.append( xmlRule );
} );
// Clean the XML file (domToPrettyText adds some text between nodes)
_cleanXML( xmlRoot );
var content = Blockly.Xml.domToPrettyText( xmlRoot );
var blob = new Blob( [ content ], { type: "text/xml"} );

var fd = new FormData();
fd.append( "upnp_file_1", blob );
fd.append( "upnp_file_1_name", fileName );

$.ajax( {
method: "POST",
url: _getCgiUrl() + "/upload_upnp_file.sh",
data: fd,
crossDomain: true,
headers: {'X-Requested-With': 'XMLHttpRequest'},
xhrFields: {
withCredentials: true
},
contentType: false,
processData: false
} )
.done( function( html ) {
if ( html.indexOf( "OK|" + fileName ) !== -1 ) {
d.resolve();
} else {
PageMessage.message( "Save \"" + fileName + "\" : " + html, "danger");
d.reject();
}
} )
.fail( function( jqxhr, textStatus, errorThrown ) {
PageMessage.message( "Save \"" + fileName + "\" : " + textStatus + " - " + (errorThrown || "unknown"), "danger");
_exportXml( xmlRoot );
PageMessage.message( "Modified XML file has been exported. You have to upload it on the Vera by hand.", "warning");
d.reject();
} );

return d.promise();
}

I don't think there's a need of exposing this feature in the UI (ALTUI) : the filesystem of openLuup is already easily reachable.

Besides, this feature of uploading, needs surely to be reenforced concerning security.
It allows to replace any files (yours and system's files)... for example with a file named "../passwd", I think there might be problems.

The script "upload_upnp_file.sh" seems to check some injections.

Offline reneboer

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1574
  • Karma: +110/-31
Re: openLuup : uploading a file
« Reply #7 on: February 13, 2016, 06:12:12 am »
Hi,

I made vosmonts server.lua mods and tried this in Firefox and see this error in the console
Code: [Select]
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://192.168.178.21/cgi-bin/cmh/upload_upnp_file.sh. (Reason: CORS header 'Access-Control-Allow-Origin' missing).
After added the different headers to the lighttpd.conf
Code: [Select]
Access-Control-Allow-Origin: http://192.168.178.21:3480
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, X-Requested-With
Access-Control-Allow-Methods: OPTIONS, GET, POST, PUT, DELETE
That cleared the CORS error but it is still not working and I am at the end of my Google supported skills  ;D

Also I did modify the upload_upnp_file.sh script as the default will try to compress and on openLuup it should only be moved to the cmh-ludl folder.

Update Ok, did even search a bit further. Found that the /usr/bin/haserl is not on my Pi so I pulled from sourceforce.net and installed it. But still no go.

This is the message I get:
Code: [Select]
‎13‎-‎2‎-‎2016‎ ‎12‎:‎41‎:‎26 Save "C_RulesEngine_Rules.xml" : #!/usr/bin/haserl --upload-limit=1000 Content-type: text/html <? #Copyright (C) 2009 MiOS, Ltd., a Hong Kong Corporation # www.micasaverde.com # 1 - 702 - 4879770 / 866 - 966 - casa #This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License. #This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; #without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. log_file="/tmp/log.upload_upnp_file" log(){ echo -e "$(date +%F_%T)|$$|$*" >> $log_file } log "====== START ======" env >> $log_file # Check for code injection. for var in $(env | grep '^FORM_' | awk -F '=' '{ print $1 }'); do if eval "echo \$$var | grep -q '\\\$(\|\`'"; then log "ERROR: Found invalid characters in arguments: $(env | grep '^FORM_')" echo "ERROR: Invalid characters in arguments" exit 1 fi done for i in $(seq 1 10); do #the file stored in /tmp eval file=\$FORM_upnp_file_${i} #the name of the files that was uploaded eval full_file_name=\$FORM_upnp_file_${i}_name log "No $i : file=$file full_file_name=$full_file_name" if [[ -n "$full_file_name" ]]; then [[ $(stat -c %s "$file") -gt 0 ]] || { log "ERROR : Uploaded file has a ZERO size"; continue; } #strip dirs an get only the filename file_name=$(basename $full_file_name 2>>$log_file) #file_dest="/etc/cmh-ludl/${file_name}.lzo" log "Compress : $file in $file_dest" mv "$file" /etc/cmh-ludl/"$file_name" #if /usr/bin/pluto-lzo c "$file" "$file_dest" 1>>$log_file 2>&1; then # log "SUCCESS compressing" log "SUCCESS moving" echo "OK|$file_name" #or 'failed' #else # log "ERROR : FAILED to compress" # echo "ERROR|$file_name" #fi fi done if [[ "$FORM_restart_lua" == "1" ]]; then log "Reloading LuaUPnP ..." /usr/bin/Reload.sh 1>/dev/null 2>&1 & fi log "====== END ======" ?>  Basically a sort of dump of the upload_pnp.sh file.

Cheers Rene
« Last Edit: February 13, 2016, 06:59:32 am by reneboer »
2xVeraLite, VeraEdge, openLuup, ALTUI, 20 switches, 10 dimmers, 20 sensors, 10 scene controllers, 1 Harmony Hub, many plug-ins. Not enough time.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup : uploading a file
« Reply #8 on: February 13, 2016, 06:29:14 am »
The server changes that @vosmont made do not use the cgi script at all, but simply anything sent with that request will be directly written to a file by the server.  As pointed out, this probably needs adjusting somewhat in order to avoid mistakes, or other, overwriting bits of the system.

My only advice at the moment (I haven't started to play with it myself) is to uncomment all the _log() lines and see what diagnostics you get.

Now that others are contributing code I need to add a debug directive to make this easier.
« Last Edit: February 13, 2016, 07:03:03 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline akbooer

  • Moderator
  • Master Member
  • *****
  • Posts: 6387
  • Karma: +292/-70
  • "Less is more"
Re: openLuup : uploading a file
« Reply #9 on: February 19, 2016, 10:32:38 am »
@vosmont

I followed up on your hint and took a closer look at WSAPI (Lua Web Server API.)   As a result, I have now implemented a WSAPI connector for openLuup to allow it to run WSAPI (Lua) applications as CGIs.

This means that arbitrary web server functionality can be added to openLuup simply by writing a short, stand-alone, Lua module.  The first time a CGI URL is accessed, the module is compiled and loaded.  Thereafter, it runs just as quickly as any built-in server code.  The implementation adheres exactly to the (brief) documentation here: http://keplerproject.github.io/wsapi/manual.html and the GitHub development branch https://github.com/akbooer/openLuup/tree/development/openLuup has a new commit which updates the server.lua file and adds a wsapi.lua module.

The connector supports CGI code in both the cgi/ and cgi-bin/ directories in /etc/cmh-ludl/.  So, for example, taking exactly the code from the simple example in the above documentation:

Code: [Select]
#!/usr/bin/env wsapi.cgi

module(..., package.seeall)

function run(wsapi_env)
  local headers = { ["Content-type"] = "text/html" }

  local function hello_text()
    coroutine.yield("<html><body>")
    coroutine.yield("<p>Hello Wsapi!</p>")
    coroutine.yield("<p>PATH_INFO: " .. wsapi_env.PATH_INFO .. "</p>")
    coroutine.yield("<p>SCRIPT_NAME: " .. wsapi_env.SCRIPT_NAME .. "</p>")
    coroutine.yield("</body></html>")
  end

  return 200, headers, coroutine.wrap(hello_text)
end

...and putting it into /etc/cmh-ludl/cgi/hello.lua, allows you to invoke it with
Code: [Select]
http://openLuupIP:3480/cgi/hello.lua

...and get the returned page:
Code: [Select]
Hello Wsapi!

PATH_INFO: /

SCRIPT_NAME: /cgi/hello.lua

So what I've done is to slightly refactor your suggested server modifications and turn them into a WSAPI application.  The code is attached below, so if you put this into /etc/cmh-ludl/cgi-bin/cmh/upload_upnp_file.sh then your JavaScript should be able to access it as before.

The point is, it's outside of the openLuup codebase and any other CGI action can be implemented in the same way.  I'm quite excited about this because I'm also refactoring (slowly) some of the openLuup server internals to use the same API.  What this may mean, in the long run, is that I can use real servers like Apache and lighttpd and ditch my own.  Whether this will make system configuration easier or harder, I'm not quite sure, and I may not even decide to go in this direction, but it's an experiment worth making.

I'd be extremely grateful if you could test the attached file in your environment (I've not been able to test it fully.)  You should recognise over 95% of the code as being entirely yours.
« Last Edit: February 19, 2016, 11:58:03 am by akbooer »
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline vosmont

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 687
  • Karma: +60/-8
Re: openLuup : uploading a file
« Reply #10 on: February 23, 2016, 05:04:37 am »
Thanks akbooer !!!

It seems to work very well  :)
The upload of files is OK (Windows PC)