The Vera Community forums have moved!

Advanced => Plugins & Plugin Development => Programming => Sonos Plugin => Topic started by: lolodomo on December 28, 2012, 12:41:18 pm

Title: Say action - different problems and solutions
Post by: lolodomo on December 28, 2012, 12:41:18 pm
Regarding the problem of the starting delay, I just tried with an uploaded MP3 music file having size 3 552 154 bytes and the playback starts immediately. So our problem is really the downloaded file size that is too small.

Next step: I will try to add enough 0 to make the file big enough...
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 28, 2012, 01:33:28 pm
Adding 0 to have a file size of 35000 bytes solves the problem. 8)

New problem now: the playback starts so quickly that the volume is adjusted during the playback...
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 08:34:01 am
I have now fixed the problem relative to the starting delay. The new code is in the trunk.
Problem with sound adjustment during playback was resolved too by replacing the StartAutoplay call.

I have added a delete of the MP3 file before its download from Google. It allows an additional control whether the download is ok or not. If I added this control, it is because I discover a case where the download fails.

Here is a call with a French text working: La température dans la salle à manger est de 19.7 degrés alors que la température à l'extérieur est
Unfortunately, this one fails: La température dans la salle à manger est de 19.7 degrés alors que la température à l'extérieur est de
The result file is empty. Is there a timeout to adjust ? Is it no more working when the sentence is too long ?
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 08:47:35 am
I have the same problem with this english text for example: I have now fixed the problem relative to the starting delay. The new code is in the trunk. Problem with sound adjustment during playback was resolved too by replacing the StartAutoplay call.
This one is working: I have now fixed the problem relative to the starting delay. The new code is in the trunk.

@guessed: any idea what could be wrong in our wget command ? The command seems to fail as soon as the text is to much long. Is there a limit in the size of a HTTP request ?

Note that these sentences entered directly in Google traduction are working normally.
Title: Re: Say action - starting delay problem and best way to stop it
Post by: guessed on December 30, 2012, 08:51:57 am
@lolodomo,
Great stuff.  Did you try padding the file using an is command (like dd) instead of in Lua?  I suspect it'll be faster.


For the failing stuff, we could be hitting a command line length limit.  I've not seen one before, but what happens if you paste the command directly on the cmd line?

If this is the case, we can probably shorten parts of it, like the browser User-Agent string, or put the POST data into a file and have WGET use that... Again only if the cmd line length is the issue
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 08:56:54 am
@lolodomo,
Great stuff.  Did you try padding the file using an is command (like dd) instead of in Lua?  I suspect it'll be faster.

No but the way I do it takes around 200 ms.

Quote
For the failing stuff, we could be hitting a command line length limit.  I've not seen one before, but what happens if you paste the command directly on the cmd line?

If this is the case, we can probably shorten parts of it, like the browser User-Agent string, or put the POST data into a file and have WGET use that... Again only if the cmd line length is the issue

I check...
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 09:03:04 am
When I type the following:
Code: [Select]
rm /www/Say.390.mp3 ; wget --output-document /www/Say.390.mp3 --quiet \
 --header "Accept-Charset: utf-8;q=0.7,*;q=0.3" \
 --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
 --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11" \
 "http://translate.google.com/translate_tts?tl=en&q=I%20have%20now%20fixed%20the%20problem%20relative%20to%20the%20starting%20delay%2e%20The%20new%20code%20is%20in%20the%20trunk%2e%20Problem%20with%20sound%20adjustment%20during%20playback%20was%20resolved%20too%20by%20replacing%20the%20StartAutoplay%20call%2e"
I get this error:
Code: [Select]
wget: server returned error: HTTP/1.1 404 Not Found

Note that I have to modify my rm command in case the file does not exist to avoid an error. Certainly doing a "rm -f".
Title: Re: Say action - starting delay problem and best way to stop it
Post by: guessed on December 30, 2012, 09:08:12 am
No but the way I do it takes around 200 ms.
Wow that's expensive.  I suspect that a dd would get you closer to a single IO operation overhead, which will be a lot smaller.... Assuming the openwrt version of dd  has append mode.
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 09:24:05 am
No but the way I do it takes around 200 ms.
Wow that's expensive.  I suspect that a dd would get you closer to a single IO operation overhead, which will be a lot smaller.... Assuming the openwrt version of dd  has append mode.

I checked again: it is between 20 ms for a long text approaching the current limit and 40 ms for a very short text like "test".
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 09:34:19 am
wget on the Vera seems to not accept parameter "--input-file". >:(

I tried this command:

Code: [Select]
rm /www/Say.390.mp3 ; wget --output-document /www/Say.390.mp3 --quiet \
 --header "Accept-Charset: utf-8;q=0.7,*;q=0.3" \
 --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
 --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11" \
 --input-file /www/Say.url

and got the following return:

Code: [Select]
wget: unrecognized option `--input-file'
BusyBox v1.17.3 (2012-01-09 12:40:42 PST) multi-call binary.

Usage: wget [-c|--continue] [-s|--spider] [-q|--quiet] [-O|--output-document FILE]
        [--header 'header: value'] [-Y|--proxy on/off] [-P DIR]
        [--no-check-certificate] [-U|--user-agent AGENT] URL

Retrieve files via HTTP or FTP
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 09:46:29 am
Well, thanks to a post on the Homeseer board:
    http://board.homeseer.com/showthread.php?t=155033

I did manage to find out how to stop the "Say" command repeating it's content.  It just needs a well-formed DIDL-Lite entry, like in this example:

Code: [Select]
luup.call_action('urn:upnp-org:serviceId:AVTransport',
  'SetAVTransportURI',
 {CurrentURI='http://192.168.6.66:80/Say.666.mp3',
  CurrentURIMetaData='<DIDL-Lite xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:upnp="urn:schemas-upnp-org:metadata-1-0/upnp/" xmlns:r="urn:schemas-rinconnetworks-com:metadata-1-0/" xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/"><item id="-1" parentID="-1" restricted="true"><res protocolInfo="http-get:*:audio/mpeg:*">http://192.168.6.66:80/Say.666.mp3</res><r:streamContent></r:streamContent><upnp:albumArtURI>http://www.mios.com/wp-content/themes/miostheme/images/google-logo-square.png</upnp:albumArtURI><dc:title>Vera SayIT</dc:title><upnp:class>object.item.audioItem.musicTrack</upnp:class><dc:creator>Vera Sonos Plugin</dc:creator><upnp:album>MiOS</upnp:album><r:albumArtist>Vera Sonos Plugin</r:albumArtist></item></DIDL-Lite>'},
  666)
luup.call_action('urn:upnp-org:serviceId:AVTransport', 'Play', {}, 666)

It still has the timeout-oriented delays mentioned earlier, but it steps closer to a cleaner setup.

Ok, we could add these metadata for a propoer ending.
In this case, do we change the way we trigger the restore of the playback (call_delay based on the file size) ?
Title: Re: Say action - starting delay problem and best way to stop it
Post by: guessed on December 30, 2012, 10:02:55 am
Looks like Busybox has a simplified wget built in.

curl might work.  Worst case we could write the whole thing in Lua using the HTTP part of LuaSocket.... Kinda the way that our UPnP lib works.  I originally wanted it done in the OS so it could be backgrounded, but weve moved away from that anyhow.

On the lua based padding, were making up to 600 syscalls due to the small buffer size for append.  Increasing that buffer from 60 could reduce those overheads a lot.


wget on the Vera seems to not accept parameter "--input-file". >:(

I tried this command:

Code: [Select]
rm /www/Say.390.mp3 ; wget --output-document /www/Say.390.mp3 --quiet \
 --header "Accept-Charset: utf-8;q=0.7,*;q=0.3" \
 --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
 --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11" \
 --input-file /www/Say.url

and got the following return:

Code: [Select]
wget: unrecognized option `--input-file'
BusyBox v1.17.3 (2012-01-09 12:40:42 PST) multi-call binary.

Usage: wget [-c|--continue] [-s|--spider] [-q|--quiet] [-O|--output-document FILE]
        [--header 'header: value'] [-Y|--proxy on/off] [-P DIR]
        [--no-check-certificate] [-U|--user-agent AGENT] URL

Retrieve files via HTTP or FTP
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 10:25:00 am
I tried to use the old way without the wget command for long text but it does not work better, I got an error in Sonos.
It means the problem is not new.
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 10:32:18 am
On the lua based padding, were making up to 600 syscalls due to the small buffer size for append.  Increasing that buffer from 60 could reduce those overheads a lot.

Yes, I am using a buffer of 64 bytes but of course we can increase it.
1 Ko ?
What's the best code to initiate a string of 1024 bytes ?
Code: [Select]
local buffer = ""
for i=1, 1024 do
  buffer = buffer .. "0"
end
Title: Re: Say action - starting delay problem and best way to stop it
Post by: guessed on December 30, 2012, 10:54:47 am
I'd use string.rep() on itself and see what happens.  No idea if thats the most efficient though.  1-4k chunks just to cut down on syscalls.

We can keep that buffer around if we find its expensive to create, but I doubt it.

On the lua based padding, were making up to 600 syscalls due to the small buffer size for append.  Increasing that buffer from 60 could reduce those overheads a lot.

Yes, I am using a buffer of 64 bytes but of course we can increase it.
1 Ko ?
What's the best code to initiate a string of 1024 bytes ?
Code: [Select]
local buffer = ""
for i=1, 1024 do
  buffer = buffer .. "0"
end
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 30, 2012, 11:03:22 am
I made different tests and the size of the buffer does not make a big difference.
With "test" as text, 35 ms is the best I can get for example with a 128 bytes buffer.
Increasing the buffer size does not reduce the time.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on December 30, 2012, 11:06:28 am
To summarize the remaining problems:

1 - Say not working as soon as the text is of a certain length
2 - avoiding the repeat and restore the old playback context at the right time

Point 2 is partially ok but could be hopefully improved.
Title: Re: Say action - starting delay problem and best way to stop it
Post by: guessed on December 30, 2012, 11:12:23 am
I would still make the write chunks 1-4k to reduce the system CPU load, and syscalls always strive up SYS CPU levels.  Our end-to-end time Weill be bounded by URL calls etc,but we should minimize our CPU impact, during the time we are running, since these boxes are small and doing a bunch of other stuff that's impacted by our resource usage.


I made different tests and the size of the buffer does not make a big difference.
With "test" as text, 35 ms is the best I can get for example with a 128 bytes buffer.
Increasing the buffer size does not reduce the time.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on December 31, 2012, 05:49:55 am
Unfortunately I discovered that my solution produces a strange rendering of the end of the file.
I imagine it is due to the fact that the Sonos buffers a certain number of MP3 packets before playing them.
I think the solution would be to add real MP3 packets to the file, packets corresponding to silence.
Title: Re: Say action - starting delay problem and best way to stop it
Post by: lolodomo on December 31, 2012, 06:33:37 am
@guessed: I tried to add your metadata but it does not avoid repeat of the playback.
Title: Re: Say action - different problems and solutions
Post by: teonebello on December 31, 2012, 08:57:47 am
Hi All,

as you are speaking about the say function, I would ask you a support.

I create a "good morning message" using the sonos say function.

As the text is long I had to split in the call.
I tried to separate using the deelay function.

...
luup.call_action(LS_SID, "SetURIToPlay", {URIToPlay = "x-rincon-mp3radio://translate.google.com/translate_tts?tl=it&q=buongiorno+famiglia+XXX.+al+momento+ci+sono+"..tostring( lul_temp ).."+gradi+e+vento+a+"..tostring( lul_wind ).."+kilometriorari"}, DEVICE_NO)
luup.call_action(MN_SID, "Play", {}, DEVICE_NO)
luup.sleep(6500)
luup.call_action(LS_SID, "SetURIToPlay", {URIToPlay = "x-rincon-mp3radio://translate.google.com/translate_tts?tl=it&q=le+previsioni+per+oggi+sono+"..tostring( lul_weather ).."+con+una+massima+di+"..tostring( lul_maxtemp ).."+gradi"}, DEVICE_NO)
luup.call_action(MN_SID, "Play", {}, DEVICE_NO)
luup.sleep(6500)
luup.call_action(LS_SID, "SetURIToPlay", {URIToPlay = "x-rincon-mp3radio://translate.google.com/translate_tts?tl=it&q=Il+sistema+di+allarme+eh+attualmente+"..tostring( lul_smart_text ).."+..+Buona+giornata"}, DEVICE_NO)
luup.call_action(MN_SID, "Play", {}, DEVICE_NO)
luup.sleep(7000)
luup.call_action(MN_SID, "Stop", {}, DEVICE_NO)

Is the delay function indicate? It seems that the scene is busy until the deelay are finished.
Is there still the limit of length in the say function? That can solve my issue.

Thank you for your support
Matteo
Title: Re: Say action - different problems and solutions
Post by: lolodomo on December 31, 2012, 09:22:12 am
You encountered the problem with the URL too long. I have not checked but the way you handle it is probably a way to get it work.
But why not using the Say function ?


To come back to the Say function, we could try to cut automatically the text and finally call several times Google. We will have to find the best cut points in the text (like end of sentences) as there will be some delay between each fragment playback (time for Sonos to switch to next "web-radio" file). Other alternative would be to let the user define the cut points in the text, for example with a special character ? Then we will add other cut points only if there are too much characters.
Title: Re: Say action - different problems and solutions
Post by: parkerc on January 04, 2013, 06:12:31 am
Hi lolodomo

Just thinking (outloud as i do) ;)

As it seems Google TTS has a 100 character limit per request, an idea might be to have a character count next to the SAY field (or a fixed limit on that field) so people know or can see if they are going to reach the limit on any request. (There looks to be no restriction at the moment)

I'm testing your skills now but may consider having an '+' option so you can add extra SAY commands, then if all I requested at almost the same time, it might be possible to play them in order with little to no delay.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 05:32:19 am
Hi lolodomo

Just thinking (outloud as i do) ;)

As it seems Google TTS has a 100 character limit per request

When typing my text directly in http://translate.google.fr/?hl=fr I have not the limit I get when using the API. Isn't the web site using the API ?

Quote
an idea might be to have a character count next to the SAY field (or a fixed limit on that field) so people know or can see if they are going to reach the limit on any request. (There looks to be no restriction at the moment)

You mean the field in the UI (player tab) ? As it is Javascript, it is probably possible. But that would not help the users using the other ways (scene) to use Say. So the idea is more to find a global solution, not something limited to a particular usage.

Quote
I'm testing your skills now but may consider having an '+' option so you can add extra SAY commands, then if all I requested at almost the same time, it might be possible to play them in order with little to no delay.

Sorry I don't understand your request.
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 05, 2013, 06:12:18 am
I would suggest the last comment was to enable a "+" which would provide another input for addition text.

However, if the API is limited 100 characters, it might be a better option to substring at the last complete word before the 100 character limit and send it in multile blocks.
Title: Re: Say action - different problems and solutions
Post by: parkerc on January 05, 2013, 07:13:14 am
Thanks Brientim,

Yes that was the idea, to have the ability to do multiple requests, that could then be played in order (maybe added in sequence to the queue).

@lolodomo

As for the 100 character limit - check this out http://viralpatel.net/blogs/the-unofficial-google-text-to-speech-api/. I interpret this to mean that requests not made directly into the google tts page have a limit set on them..
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 05, 2013, 07:18:32 am
Something like this chain of thought.
http://stackoverflow.com/questions/10189860/controlling-input-for-google-tts-api
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 07:54:31 am
The limit at 100 characters is exactly what I have with my initial example.
The good news is that the limit is absolutely not dependent on the way we call Google.
So we have now just to cut a more than 100 characters in several blocks.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 08:09:02 am
Something like this chain of thought.
http://stackoverflow.com/questions/10189860/controlling-input-for-google-tts-api

My question is then: can we concate all downloaded files to make one unique file ?
Some tests have to be done to check if concatening two files having the same sampling rate is ok for playing by Sonos.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 08:20:38 am
Something like this chain of thought.
http://stackoverflow.com/questions/10189860/controlling-input-for-google-tts-api

My question is then: can we concate all downloaded files to make one unique file ?
Some tests have to be done to check if concatening two files having the same sampling rate is ok for playing by Sonos.

The answer is yes, I just made a positive test 8)
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 05, 2013, 08:24:26 am
That was very quick.   :o
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 08:24:59 am
So my idea would be the following for cutting the buffer:
1 - first find # characters in the string and cut after these characters (that means the user can decide where to cut)
2 - if not enough (and/or # characters not found), cut after a .
3 - if not enough, cut at the end of the last word
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 05, 2013, 08:55:18 am
I suppose there are few options:
what you have stated where there is a distinct character to find and used to cut. I see there only issue here is the user would need to know the active character position/block length to enable them to enter the character with the blocks. E.g if first block ends @80, the next block finishes @180.

Provide multiple inputs max length 99 characters and then concatenate with a space. 

I'd work on the premise it maybe best to do this automatically without user intervention.

What should be the maximum entry allowed, 200, 300? Whatever it is allocated, the input box would need to be resized to enable a user to see full text especially if option 1 is implemented.
Title: Re: Say action - different problems and solutions
Post by: parkerc on January 05, 2013, 09:01:33 am
Agreed.

Limiting a SAY to 100 would be good, and if more are needed then the uses can choose to add '+' another SAY input box..

(How would this work in a scene)?
Title: Re: Say action - different problems and solutions
Post by: parkerc on January 05, 2013, 09:07:56 am
@Lolodomo

I'm probably pushing my luck now, but for a future release maybe move 'Speach' to its own tab and think about a dynamic 'Say' TTS builder .? (With some of these as presets - http://forum.micasaverde.com/index.php/topic,12408.0.html)

The best speeches for me are the ones that Vera makes up itself ;)
Title: Say action - different problems and solutions
Post by: Brientim on January 05, 2013, 09:13:13 am
The penny drops and hits me.  I had not thought about scenes. As I do not currently use Vera for any AV equipment I not on the habit of doing this on Vera.

How would it deal with multiple entries in a scene. One thing at a time.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 05, 2013, 09:33:32 am
If calling several times Say (in a scene), there will be a restore previous context between each call.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 06, 2013, 04:48:47 am
I'd work on the premise it maybe best to do this automatically without user intervention.

What should be the maximum entry allowed, 200, 300? Whatever it is allocated, the input box would need to be resized to enable a user to see full text especially if option 1 is implemented.

Ok, I will do it automatically but the rendering could be worst than something done cleverly by the user.
I will define no limit.
Regarding the UI, for the scene advanced tab, it is controled by MCV, I cannot change it.
For the player tab (Sonos), I miss space. As suggested, the best could be to add a new tab dedicated to TTS. But these TTS fields are here only for testing purpose. So I am not sure it is interesting to loose time on this.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 06, 2013, 05:00:50 am
@Lolodomo

I'm probably pushing my luck now, but for a future release maybe move 'Speach' to its own tab and think about a dynamic 'Say' TTS builder .? (With some of these as presets - http://forum.micasaverde.com/index.php/topic,12408.0.html)

The best speeches for me are the ones that Vera makes up itself ;)

This is of course an interesting usage of the Say command and I will myself define some Say commands in my scenes. But in my opinion, it cannot be included in the plugin because everybody has its own needs, starting by the language to be used. The best would be that everybody add their examples to your topic.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 06, 2013, 05:34:25 am
Unfortunately I discovered that my solution produces a strange rendering of the end of the file.
I imagine it is due to the fact that the Sonos buffers a certain number of MP3 packets before playing them.
I think the solution would be to add real MP3 packets to the file, packets corresponding to silence.

Anyone has skills to let me know what could be a valid MP3 silent packet ?
Title: Re: Say action - different problems and solutions
Post by: hek on January 06, 2013, 03:54:11 pm
I think the following actually works...

Code: [Select]
cat input.mp3 silence.mp3 > output.mp3
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 25, 2013, 07:51:14 pm
I have now fixed the Say action when text is longer than 100 characters.
Title: Re: Say action - different problems and solutions
Post by: teonebello on January 26, 2013, 02:13:32 am
I have now fixed the Say action when text is longer than 100 characters.

How it works?

Thanks
Matteo
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 26, 2013, 04:03:10 am
I have now fixed the Say action when text is longer than 100 characters.

How it works?

Thanks
Matteo

Text is cut in fragments of 100 characters max. To do that a space is searched.
Then each fragment is translated by Google to produce a mp3 file.
Finally all files are concatenated in a unique mp3 file that is played by the Sonos.
Title: Re: Say action - different problems and solutions
Post by: teonebello on January 26, 2013, 05:24:40 am
I have now fixed the Say action when text is longer than 100 characters.

How it works?

Thanks
Matteo

Text is cut in fragments of 100 characters max. To do that a space is searched.
Then each fragment is translated by Google to produce a mp3 file.
Finally all files are concatenated in a unique mp3 file that is played by the Sonos.

That is simply great! Thanks

Which version of the Plug in should i install to test it?
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 26, 2013, 06:16:51 am
Files from the trunk.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 26, 2013, 06:27:36 am
I think the following actually works...

Code: [Select]
cat input.mp3 silence.mp3 > output.mp3

This solution is working and solve the ending problem.

For your information, Sonos is able to manage bitrate change in a file. My silence file can be at 128 or 32 kbps, it is working (in fact I made my first tests with a short music file).

So I have produced (with Audacity) a silence file, 10 seconds at 32 kbps. It produces a file of size around 40 kb. I uploaded the file in /www/silence.mp3 and my current version of the plugin uses it.

My question to finish the work and commit my changes is: if I join the silence file to the plugin files, where will it be installed by default, considering the future automatic installation from the app store ?
Title: Re: Say action - different problems and solutions
Post by: hek on January 26, 2013, 11:28:46 am
Great work @lolodomo.
Title: Re: Say action - different problems and solutions
Post by: guessed on January 26, 2013, 05:17:42 pm
My question to finish the work and commit my changes is: if I join the silence file to the plugin files, where will it be installed by default, considering the future automatic installation from the app store ?
I would use a file of the same "naming" model as the plugin files, and keep it in the cmh-ludl directory.  I think ALL of the files should live there, from an installation standpoint, and we need to write a [os.execute(...)] script to move them into place when the Plugin starts.

In the current Apps.mios.com model, you have to enter (and re-enter) the new location for anything that doesn't live in cmh-ludl.  Each time we'd have to patch, we'd have to re-enter any custom location information.

This bug has been in the Apps store for ~1yr now, and there's no fix targeted for it.  It's one of the things that made be put the brakes on going further to get the plugin into the store... the maintenance was going to be too high once the custom images had been added.  :(
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 26, 2013, 06:40:44 pm
Good idea.
That way it will even work using the manual upload and the user will not have to take care where to put the file.
The only thing we have to consider is that the files will be lzo compressed ?
I have to remind what is the command to uncompress...
We do the copy at each restart ?
We do a copy, keeping the files at their original place too or we move them ?
Title: Re: Say action - different problems and solutions
Post by: guessed on January 26, 2013, 07:56:33 pm
Not all files get compressed, so it may not be needed.  I haven't played with working out which ones do/don't "naturally" get compressed by apps.mios.com.

If we need to perform any decompression step, then we can run
Code: [Select]
    pluto-lzo d <srcFile> <dstFile>
We should use a mv command instead of a copy.  This can be executed [unconditionally] at each restart and since we're effectively moving the files into place, it'll be benign/low-cost on subsequent calls.

I would think something like:
Code: [Select]
    os.execute('mv Sonos*.png /www/<wherever>')
If we need to go pluto-lzo d anything, then perhaps we can put a "Sonos.sh" script in that does the work, and we jut os.execute() that. 

That would compartmentalize any apps.mios.com installation work-arounds into something readily moved if they ever fix that system.
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 26, 2013, 08:07:29 pm
When you upload files to apps.mois.com, it has a check box to select to compress or not.
Title: Re: Say action - different problems and solutions
Post by: guessed on January 26, 2013, 08:15:51 pm
When you upload files to apps.mois.com, it has a check box to select to compress or not.
Yes, but I'm looking to make it so that we NEVER have to tweak/re-enter settings into apps.mios.com.  So, if it defaults to compress (and "re-defaults" (looses") the setting upon patching) then we'll run with it compressed and have our scripts handle the rest.

If it remembers this setting, on a per-file basis, then we could use it.  Basically what we're planning here is to work-around the deficiencies of apps.mios.com, since they're [currently] preventing us from putting the plugin into that system due to it's buggyness ;)
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 26, 2013, 09:10:48 pm
As far as It does not remember it on a file by files setting as it does not for location but they need to look at including a library rather then manually entering per file on each version.

Yes, I totally agree with the logic being applied... And, as there are known issues with both the apps.mios.com and its deployment of files files, managing this within the script would be an ideal way to negate any user concerns. It would for the first time where icons are included, manual operations would be consistent with uploading other files and not require additional management overheads by the end user.

I wish they would get tapatalk working again, it is so much easier than working with a mobile browser.
Title: Re: Say action - different problems and solutions
Post by: guessed on January 26, 2013, 09:32:23 pm
I wish they would get tapatalk working again, it is so much easier than working with a mobile browser.
On the plus side, at least we're not seeing anymore "Sent from my WT19i using Tapatalk 2" messages...


Send from my Macbook Pro using a Keyboard, typed by hand  8)
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 27, 2013, 12:29:52 am
Yes, I made sure I never posted with a signature from Tapatalk...
Hasbro of UI? Curious comment... Transformer, not listening, great toy, gender biased, alway growing???
Title: Re: Say action - different problems and solutions
Post by: guessed on January 27, 2013, 12:48:17 am
Hasbro of UI? Curious comment...
A toy UI, with a matching colour scheme, and readily breaks/needs to be replaced.  Something not suited for heavy use.  I change the expression every few weeks.
Title: Re: Say action - different problems and solutions
Post by: Brientim on January 27, 2013, 12:56:45 am
I thought it might be something along those lines of being a toy, but, you explained it very well. And, I can now picture it.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 27, 2013, 04:20:17 am
When files are uploaded from the ui, they are lzo compressed. Verified yesterday with a MP3 file and if I correctly remember it is the same thing with a PNG file.
The shell script is a good idea but here again we will have to decompress it first.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 27, 2013, 05:01:32 am
With the shell script, we could have another problem: run permission missing. As the script will be simple, we may keep it in the lua code of the plugin, it will avoid one preliminary step (uncompress of the shell script).

By default, root uses ash shell. I know sh, csh, ksh but not ash ;) Not a problem by the way.
When runnign os.execute from lua, the command is run using what shell ? ash ? I know how to check it but if someone has the answer, that would avoid me to loose few minutes ;D
Title: Re: Say action - different problems and solutions
Post by: guessed on January 27, 2013, 05:53:24 pm
Haven't tried it from Lua, but the following captures the essence, and can likely be done from os.execute directly (avoiding the Shell script decompress)

Code: [Select]
target="/www/cmh/skins/default/icons/"
files="Sonos.png Sonos_0.png Sonos_25.png Sonos_50.png Sonos_75.png Sonos_100.png"

for i in ${files}
do
  if test -e ${i}.lzo;
  then
    echo "${i}.lzo exists, decompressing..."
    if pluto-lzo d ${i}.lzo ${i};
    then
      echo "Removing ${i}.lzo"
      mv ${i} ${target}
      rm "${i}.lzo"
    else
      echo "Error decompressing ${i}.lzo"
    fi
  else
    echo "no ${i}.lzo exists"
  fi
done

But there's a larger issue.  I was running some tests and pluto-lzo, used in Vera's file upload, is trashing the file.  It does end up with a LZO suffix, but the file is only 16 bytes long.

I've always side-loaded these various files via scripts, so never noticed that before.  If apps.mios.com is using the same technique, then we can't have these marked as compressed.

As a result of this, I didn't test the above any further.  Of course, we can mod it to not apply compression (etc)
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 28, 2013, 05:05:28 am
We can do uncompress+move in one call using pluto-lzo.
Title: Re: Say action - different problems and solutions
Post by: guessed on January 28, 2013, 10:11:06 am
We can do uncompress+move in one call using pluto-lzo.
Yes, I know.  I whipped out an outline of a script to get syntax, and the basic concept down.  I then started running it, and realized that pluto-lzo is corrupting binary files so never got to the "optimization" phase. 

There's a bunch of stuff extra I'd do there, but it was like "painting deck-chairs on the Titanic" at that point...
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 28, 2013, 03:23:22 pm
If pluto-lzo is not working well, we have to use the other alternative, that is specifying in app store where to install the (uncompressed) files.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 30, 2013, 03:54:07 pm
I have now commited my changes.
TTS is finally working well now (quick starting, proper ending, long text managed, possible grouping of several Sonos units, ...) 8)

Please note that you must manually upload the new file Sonos_silence.mp3 to /www/Sonos_silence.mp3.
Title: Re: Say action - different problems and solutions
Post by: guessed on January 30, 2013, 04:24:17 pm
@lolodomo,
For files that don't need to be in /www/ we should just leave them in their regular install directory.  Then (one day) they'll get cleaned up correctly when a plugin is de-installed.

So things like tmp files (etc) should be done somewhere that will inherently get cleaned up (like /tmp), perm (install) files can be in /etc/cmh-ludl, and then only the "result" files that a user needs to be downloaded will end up on WWW

...that should keep it cleaner for folks, just in case we accidentally blow out the FS and they need to "reboot" to clean out stuff.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on January 30, 2013, 04:35:33 pm
As you previously said pluto-lzo is not working well, I have forgiven the idea of a proper installation.

What is certain is that the MP3 file can be everywhere on the system, we have only to adjust the CONCAT_EXECUTE variable in the plugin to reflect its location.

The png files required to be in a specific directory.

I think we have to find a solution that would work either from the app store installation or from the UI (manual upload). In the second case, we konw where are put the files and we know that they are compressed.
Title: Re: Say action - different problems and solutions
Post by: guessed on February 24, 2013, 05:34:59 pm
Ok, so I tweaked the code a little.

It now creates all the intermediate/temporary files in /tmp/.  Only the resultant output file [temporarily] ends up in /www/

If Vera reboots, or if someone manually restarts, at least the temporary files will be cleared up (leaving only the /www/Say.ID.mp3 file if we didn't get to remove it ourselves)

I also changed it so that the MP3 file can be in the standard upload directory (/etc/cmh-ludl/).  With the change as made, it also supports it in the older location (/www/), but the logic here is to minimize the work-required for "Say" to function if we move forward with an apps.mios.com deployment.

ie. I wanted to upload it via apps.mios.com, and not run into the bug they have about "location" being lost on each patch.

These changes are in place, and the Wiki has been changed to reflect where the manual-upload of the MP3 needs to go.
Title: Re: Say action - different problems and solutions
Post by: lolodomo on February 25, 2013, 04:29:17 pm
I also changed it so that the MP3 file can be in the standard upload directory (/etc/cmh-ludl/).  With the change as made, it also supports it in the older location (/www/), but the logic here is to minimize the work-required for "Say" to function if we move forward with an apps.mios.com deployment.

ie. I wanted to upload it via apps.mios.com, and not run into the bug they have about "location" being lost on each patch.

The problem is that in case of a manual installation, the user still needs to use WinSCP rather than the Vera dashboard upload feature due to the automatic compression.

I think we should run a script that will try to uncompress PNG or MP3 files inside the directory /etc/cmh-ludl/. If the decompression fails, the user will be able to try again.
For the icons, we could even automatically create a symbolic link to the files in /etc/cmh-ludl/
That way, everything has to be upoloaded in /etc/cmh-ludl/ and a script launched automatically by the plugin at startup is in charge to uncompress PNG/MP3 files and to create symbolic links for the PNG files. And no difference between a manual install and an installation from the store, except that from the store the decompression step will even not be necessary.
Title: Re: Say action - different problems and solutions
Post by: guessed on February 25, 2013, 05:46:22 pm
There are multiple bugs in this space.

The first is that, locally, file upload is corrupting BINARY Uploads.
The second is that, via apps.mios.com, the UI is forgetting about file location upon repeat upload (patching) of source files.

I updated the Wiki instructions to clarify that the MP3's need to be uploaded via WinSCP (et-al).  This instruction was there before, but it wasn't clear in the previous instructions.

My changes were mostly to handle the 2nd case above, since we cannot "uncorrupt" a binary loaded via the UI, that's something the MiOS team will have to address.... one day.

Quote
For the icons, we could even automatically create a symbolic link to the files in /etc/cmh-ludl/
Yes, symlinks would be a better option.  I'm hesitant to put actual content into /www (generally) since they might not be "found" when a problem occurs (file system limits, etc) and they're not guaranteed to succeed across production releases.  But symlinks would be recreated when not there (if we code it that way), and the SAY Output would also be recreated.

Something that symlinks when we startup would "fix" this in all cases....  including someone migrating across Vera hardware (where they might not be expecting to copy stuff under /www)

Title: Re: Say action - different problems and solutions
Post by: lolodomo on February 25, 2013, 06:04:43 pm
New feature added: Say calls are now managed in a queue. It allows you to call several times the Say actions without waiting for the ending of the previous call. While there are calls in queue, each message is said successively without restoring the context between each message.
Please note that group zones are not correctly managed in case of two successive calls with different group zones.

As an example, try this:
Code: [Select]
luup.call_action("urn:micasaverde-com:serviceId:Sonos1", "Say", {Text = "Here is a first test.", Volume="70"}, xxx)
luup.call_action("urn:micasaverde-com:serviceId:Sonos1", "Say", {Text = "Voici un second test.", Language="fr", Volume="60"}, xxx)
luup.call_action("urn:micasaverde-com:serviceId:Sonos1", "Say", {Text = "Here is a thrid test.", Volume="50"}, xxx)
luup.call_action("urn:micasaverde-com:serviceId:Sonos1", "Say", {Text = "Voici un quatri?me test.", Language="fr", Volume="40"}, xxx)