We have moved at community.getvera.com

Author Topic: Bug in handling of i18n JSON responses from the MiOS Unit  (Read 1763 times)

Offline lolodomo

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3484
  • Karma: +74/-10
Bug in handling of i18n JSON responses from the MiOS Unit
« on: December 16, 2014, 07:24:30 pm »
@guessed: I may have found a little bug in your binding. That is relative to strings containing special characters. They are not rendered correctly in UI. I can see that from the track title coming from the Sonos plugin.

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Bug in handling of i18n JSON responses from the MiOS Unit
« Reply #1 on: December 16, 2014, 09:33:31 pm »

@guessed: I may have found a little bug in your binding. That is relative to strings containing special characters. They are not rendered correctly in UI. I can see that from the track title coming from the Sonos plugin.

Can you send me a PM with
a) that snippet of our user_data JSON
b) a screenshot of the character/string in both the openHAB (Classic UI) as well as Vera's UI.

It'll need to be in a ZIP file in order to preserve the byte sequence(s) being used for the char.

Under the covers I'm just using a standard Java/JSON parser, so I'll need to isolate which component is mangling the data in its transition through the layers to the UI.

ie. I'm wondering if Vera is correctly using Unicode in their JSON :)

Offline lolodomo

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3484
  • Karma: +74/-10
Re: Bug in handling of i18n JSON responses from the MiOS Unit
« Reply #2 on: December 21, 2014, 07:39:04 am »

@guessed: I may have found a little bug in your binding. That is relative to strings containing special characters. They are not rendered correctly in UI. I can see that from the track title coming from the Sonos plugin.

Can you send me a PM with
a) that snippet of our user_data JSON
b) a screenshot of the character/string in both the openHAB (Classic UI) as well as Vera's UI.

It'll need to be in a ZIP file in order to preserve the byte sequence(s) being used for the char.

Under the covers I'm just using a standard Java/JSON parser, so I'll need to isolate which component is mangling the data in its transition through the layers to the UI.

ie. I'm wondering if Vera is correctly using Unicode in their JSON :)

@guessed: you can easily reproduce it by yourself in less than 5 minutes. Just open the Sonos control application and do a search for (TuneIn) radio "oui fm". You will see that the real station name has a special "u" character. Then you can see how it is rendered in Sonos application (OK), Vera (OK) and openHAB (not OK). In openHAB, use for example this item:
Code: [Select]
String SonosChambreTitle "Chanson [%s]" (GMusiqueChambre) {mios="unit:house,device:502/service/AVTransport/CurrentTitle"}
« Last Edit: December 21, 2014, 07:42:01 am by lolodomo »

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Bug in handling of i18n JSON responses from the MiOS Unit
« Reply #3 on: December 22, 2014, 05:12:26 pm »
@guessed: you can easily reproduce it by yourself in less than 5 minutes. Just open the Sonos control application and do a search for (TuneIn) radio "oui fm". You will see that the real station name has a special "u" character. Then you can see how it is rendered in Sonos application (OK), Vera (OK) and openHAB (not OK). In openHAB, use for example this item:
Code: [Select]
String SonosChambreTitle "Chanson [%s]" (GMusiqueChambre) {mios="unit:house,device:502/service/AVTransport/CurrentTitle"}

Perfect, thanks for the pointers to reproduce!

I traced the wire format of the data as it went across the layers, and could see where it was damaged (etc), but also a bunch of other interesting things along the way.

For Vera, it turns out that Vera is doing a couple of really dumb things:
  • a) It's returning a Content-Type of "json"
    ...not "text/json" or "application/json" (RFC 4627), just "json"
    That's not a valid response, but luckily I'm not [yet] looking at that string.

  • b) It's Content-Type response is missing it's charset designation.
    And in this case, the standards say that the response should be interpreted as Latin-1 (ISO-8859-1).

It's the second item that was tripping up the Java code for HTTP Response Body handling.  Per the RFC for HTTP, the content body must be interpreted as Latin-1 if there's no charset= designation (in the Content-Type response header)... so that's exactly what it was doing, and the 4-Byte (UTF-8 Encoded) sequence for the french "OUI" was getting mangled at that point.

I put a tweak in to my local code to force the charset to be interpreted as UTF-8, and it's now seeing the value correctly.

I've uploaded a new [1.7 base] version to the same Box.net URL.  It has the one-line correction for this problem (which have also been checked into my Fork), and is a composite of all the changes outstanding (based upon your feedback)

Thanks (again) for the pointers!

Offline lolodomo

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3484
  • Karma: +74/-10
Re: Bug in handling of i18n JSON responses from the MiOS Unit
« Reply #4 on: December 22, 2014, 05:43:29 pm »
It's the second item that was tripping up the Java code for HTTP Response Body handling.  Per the RFC for HTTP, the content body must be interpreted as Latin-1 if there's no charset= designation (in the Content-Type response header)... so that's exactly what it was doing, and the 4-Byte (UTF-8 Encoded) sequence for the french "OUI" was getting mangled at that point.

I put a tweak in to my local code to force the charset to be interpreted as UTF-8, and it's now seeing the value correctly.

I've uploaded a new [1.7 base] version to the same Box.net URL.  It has the one-line correction for this problem (which have also been checked into my Fork), and is a composite of all the changes outstanding (based upon your feedback)

Installed. Works perfectly now. Thank you for such a quick fix.

Will you declare a bug for Vera Control to fix that ?



For everyone, there was a fatal bug in the astro binding since the 21th of December (winter time). A new snapshot is now available fixing this bug.
« Last Edit: December 28, 2014, 07:49:37 am by lolodomo »

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Bug in handling of i18n JSON responses from the MiOS Unit
« Reply #5 on: December 22, 2014, 07:29:53 pm »
Will you declare a bug for Vera Control to fix that ?

I imagine most of the Control-point authors already know the problem, esp those on Android.  Part of real challenge here is that Lua doesn't have a good i18n story.  For your case, it's just lucky that glueing/formatting UTF-8 "Strings" (byte sequences, really) isn't being checked, and there's no lib to really support string-manipulation in Lua.

Anyhow, when they start fixing the bugs I've already filed, or even look at them, I'll file something for this 8)