We have moved at community.getvera.com

Author Topic: Tools: openHAB Item file generator  (Read 29866 times)

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Tools: openHAB Item file generator
« on: January 27, 2015, 09:08:07 pm »
I spent a little time today building an Item file generator, since it seems like a common stumbling block for folks getting started.  It was inspired by @Ap15e's work but does the conversion in a very different manner (no burden on Vera, and a complete list of StateVariables is converted to their Item counterparts)


Anyhow, it's a two-step process:

a) the first step ("miosLoad.sh") retrieves an XML version of the MiOS Device MetaData from your Vera Unit.
In some cases, Vera spits out "invalid" XML, so this content may have to be hand-edited prior to feeding it into the next step.  The output of this phase is a file called "user_data.xml"

b) the second step ("miosTransform.sh") converts the XML into a Textual openHAB Items file.
If this completes successfully, then you'll end up with an output file like "house.items"

This is an example of what it's like to run these command line scripts:

Code: [Select]
me$ ./miosLoad.sh 192.168.1.100
Loading MiOS Unit Metadata from 192.168.1.100...
Metadata Loaded into user_data.xml!
me$ ./miosTransform.sh house
Transforming MiOS Unit Metadata from user_data.xml...
Metadata Transformed into house.items!
Duplicate Item names requiring manual fixes:
  String   DownstairsDeviceStatus "Downstairs Device Status [%s]" (GDevices) {mios="unit:house,device:385/status"}
  Number   DownstairsId "ID [%d]" (GDevices) {mios="unit:house,device:385/id"}
  String   LivingRoomSonosPIcon "Living Room Sonos (P) Icon [%s]" (GDevices,GRoom2) {mios="unit:house,device:295/service/DeviceProperties/Icon"}
  String   MasterBedroomSonosIcon "Master Bedroom Sonos Icon [%s]" (GDevices,GRoom7) {mios="unit:house,device:331/service/DeviceProperties/Icon"}
  String   SceneControllerConfigured "_Scene Controller Configured [%s]" (GDevices) {mios="unit:house,device:394/service/HaDevice1/Configured"}
  String   SceneControllerDeviceStatus "_Scene Controller Device Status [%s]" (GDevices) {mios="unit:house,device:393/status"}
  String   SceneControllerDeviceStatus "_Scene Controller Device Status [%s]" (GDevices) {mios="unit:house,device:394/status"}
  Number   SceneControllerId "ID [%d]" (GDevices) {mios="unit:house,device:394/id"}
  Number   SceneControllerId "ID [%d]" (GDevices) {mios="unit:house,device:4/id"}
  String   SceneControllerScenes "_Scene Controller Scenes [%s]" (GDevices) {mios="unit:house,device:393/service/SceneController1/Scenes"}
  String   SceneControllerScenes "_Scene Controller Scenes [%s]" (GDevices) {mios="unit:house,device:394/service/SceneController1/Scenes"}
  String   SceneMasterBathroom "Master Bathroom Scene" <sofa> (GScenes) {mios="unit:house,scene:34/status", autoupdate="false"}
  String   SceneMasterBathroom "Master Bathroom Scene" <sofa> (GScenes) {mios="unit:house,scene:35/status", autoupdate="false"}
  String   SceneTestArmed "TestArmed Scene" <sofa> (GScenes) {mios="unit:house,scene:73/status", autoupdate="false"}
  String   SceneTestDisarmed "TestDisarmed Scene" <sofa> (GScenes) {mios="unit:house,scene:76/status", autoupdate="false"}
  String   UpstairsDeviceStatus "Upstairs Device Status [%s]" (GDevices) {mios="unit:house,device:337/status"}
  Number   UpstairsId "ID [%d]" (GDevices) {mios="unit:house,device:337/id"}
  Contact  SceneMasterBathroomActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:34/active"}
  Contact  SceneMasterBathroomActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:35/active"}
  Contact  SceneTestArmedActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:79/active"}
  Contact  SceneTestDisarmedActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:80/active"}

At this point, you'll have a fairly complete house.items file for use in openHAB.  I've included almost all of the "by-hand" rules for conversion that I've used in the past, so SwitchPower1/Status become Switch, etc, but I'm sure there's a lot more to add as people try out devices (etc) that I don't have.

The conversion script relies upon the command-line utility called xsltproc, which seems to come with Linux and Mac OSX.

Not everything is neatly parceled, but it will get you a very close starting point.  You will want to manually check the output Items file, since I don't perform any name-collision checks what-so-ever, and Item-names might have gotten duplicated (actually, I know they do in some cases)



Downloads...

Note: You can also use this trick to retrieve the MAP files.
« Last Edit: February 20, 2016, 12:01:54 pm by guessed »

Offline gduprey

  • Beta Testers
  • Full Member
  • *****
  • Posts: 122
  • Karma: +6/-1
Re: Tools: openHAB Item file generator
« Reply #1 on: January 27, 2015, 10:41:12 pm »
Howdy,

Just gave your script a run under ubuntu 14.04.  Did have to change the 1st line of both scripts to

#!/bin/bash

Otherwise, some tests won't work as the /bin/sh standard shell doesn't recognize them.

The resultant items file looks pretty complete (really complete -- prolly need to do some trimming).  I notice a lot of instances of "FIXME" in the names.  Could you give me a quick idea what the presence of FIXME may typically mean? (i.e. bad character, duplicate identifier, etc, etc).

Very promising boostrap tool!

Gerry

Offline shmixx

  • Full Member
  • ***
  • Posts: 171
  • Karma: +2/-1
Re: Tools: openHAB Item file generator
« Reply #2 on: January 27, 2015, 11:02:33 pm »
@guessed - awesome work!! I'm sort of glad I had to stumble through it because I understand it better now, but for up and comers, I think this is an awesome tool. And props to Ap15e as well for the work used in the effort. This is going to ramp up MiOS usability of OH for sure!

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #3 on: January 27, 2015, 11:40:24 pm »
Just gave your script a run under ubuntu 14.04.  Did have to change the 1st line of both scripts to

#!/bin/bash

Otherwise, some tests won't work as the /bin/sh standard shell doesn't recognize them.

:)  That's what I get for using a Mac...   I've patched the scripts and repushed to Box, thanks for the heads-up!

Quote
The resultant items file looks pretty complete (really complete -- prolly need to do some trimming).  I notice a lot of instances of "FIXME" in the names.  Could you give me a quick idea what the presence of FIXME may typically mean? (i.e. bad character, duplicate identifier, etc, etc).

So the FIXME's are just in the Label of each Item.  For the most part I don't always have a good Label to apply to stuff, so I glue something together from the bits I have.

In this case, I'm glueing together the Name of the Device (eg. "Kitchen Lights") and the State Variable name (eg. "HaDevice1/LastUpdate").

In that example, because this StateVariable maps to an openHAB DateTime, the result is a Label something like:

Code: [Select]
DateTime KitchenLightsLastUpdate "Kitchen Lights FIXME LastUpdate [%1$ta, %1$tm/%1$te %1$tR]" <calendar> {mios=unit:...HaDevice1/LastUpdate"}
and I figured people would go back over their Labels and fix them up, so they'd need something to search on.


The XSLT is in the file transform.xslt and, for the most part, is readable.  I'm sure there's a lot more rules to put in there, the most obvious being to move to using the UPnP Service Aliases ("SwitchPower/Status") instead of their fully-qualified counterparts ("urn:upnp-org:serviceId:SwitchPower1/Status")... for brevity/readability, if nothing else.

Anyhow, we can grow it as people come across other use-cases.


Thanks for the feedback!

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #4 on: January 27, 2015, 11:52:06 pm »
@guessed - awesome work!! I'm sort of glad I had to stumble through it because I understand it better now, but for up and comers, I think this is an awesome tool. And props to Ap15e as well for the work used in the effort. This is going to ramp up MiOS usability of OH for sure!

There's something to be said for doing it by hand, esp if you want to understand how to do more with the Binding than it's OOBox defaults, but we're starting to see an adoption rate higher than I was expecting (if PM's are anything to go by) so the Generator is to ease the pain a little for folks wanting a more wholesale bridging of their Vera bits instead of the one-or-two Devices that some people might experiment with...

... and you can always build SiteMaps using the [fat] Designer tools or Chris's nice web-based HABmin interface.  That's a little harder to automagically build, since so-much stuff is based upon user-layout preference.

Quote
And props to Ap15e as well for the work used in the effort.

Totally.  His effort [effectively] goaded me to do this.

I used a different approach as I wanted something that was more-like my hand-built Items file, and I knew there was only one way to get it ;)

Offline d55m14

  • Sr. Member
  • ****
  • Posts: 449
  • Karma: +8/-1
Re: Tools: openHAB Item file generator
« Reply #5 on: January 28, 2015, 06:50:44 am »
Hi guessed,

I've tried to excecute your tool . The first step returned the following msgs and the .xml was created.

root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142...
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied

The second step returned the following msgs and the items was created too :

root@raspberrypi:/opt/tool# sh transform.sh DBHomeVeraEdge
Transforming MiOS Unit Metadata from user_data.xml...
transform.sh: 16: [: 0: unexpected operator
Failed to Transform, Check for bogus XML in user_data.xml.


I've attached the two files created.

Any idea ?

tnks donato

Offline gduprey

  • Beta Testers
  • Full Member
  • *****
  • Posts: 122
  • Karma: +6/-1
Re: Tools: openHAB Item file generator
« Reply #6 on: January 28, 2015, 08:42:08 am »
root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142...
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied


Instead of 'sh load.sh ....' use 'bash load.sh ....'

There are operators in there that require bash and not the standard shell.

Offline d55m14

  • Sr. Member
  • ****
  • Posts: 449
  • Karma: +8/-1
Re: Tools: openHAB Item file generator
« Reply #7 on: January 28, 2015, 08:53:07 am »
root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142...
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied


Instead of 'sh load.sh ....' use 'bash load.sh ....'

There are operators in there that require bash and not the standard shell.

thank you very much..

donato

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #8 on: January 28, 2015, 11:08:47 am »
The newer version of the script, uploaded at 8:40pm PDT (about 90 minutes after the originals) has the #!/bin/bash directive at the top (thanks goto @gduprey for catching that).

You should only need to set the executable bit on the scripts, after untarring, and leave out the "bash" prefix when executing it.

I tested these as working on both MacOSX Yosemite 10.10.1 and on Ubuntu/Linaro 14.04.1 LTS (on my ODroid C1).
« Last Edit: January 28, 2015, 12:16:41 pm by guessed »

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #9 on: January 28, 2015, 03:17:36 pm »
The newer version of the script, uploaded at 8:40pm PDT (about 90 minutes after the originals) has the #!/bin/bash directive at the top (thanks goto @gduprey for catching that).

You should only need to set the executable bit on the scripts, after untarring, and leave out the "bash" prefix when executing it.

I tested these as working on both MacOSX Yosemite 10.10.1 and on Ubuntu/Linaro 14.04.1 LTS (on my ODroid C1).

A new-new version uploaded, changes include:
a) Rename load.sh -> miosLoad.sh
b) Rename transform.sh -> miosTransform.sh
c) Support for generating all UPnP ServiceId Aliases defined in the MiOS Binding Wiki page.

It now generates something a lot closer to the hand-built version that I'm using.

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #10 on: January 29, 2015, 05:19:54 pm »
Just tweaked the script a little, and re-up'd to Box.net. 

Turns out not every MiOS Device has the category_num attribute in the XML.  If a particular Plugin wasn't configured to put this value in, then the attribute would be missing altogether and the XSLT Filter rules would skip over generation for that Device.

In my environment, this was skipping the Device -> Item generation for the following Plugins:
  • ListMasterGarageDoor
  • SystemMonitor
  • WebPowerSwitch (the root/parent Device)
  • Nest (the root/parent Device)
  • Weather
  • ParadoxAlarm (root/parent Device)

Others may have been impacted also, so best to re-run it, and manually "merge" the new results.

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #11 on: January 30, 2015, 01:06:50 pm »
New version uploaded, now incorporates removal of most special characters in the Item name normalization process.

eg. _#-!@$%^&*=+~`[]{}\\|:;"<\>?/.()

There are others that likely should be removed, but these can be generated from my [US] Keyboard.
« Last Edit: January 30, 2015, 01:13:58 pm by guessed »

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #12 on: January 30, 2015, 09:08:59 pm »
New version uploaded, this one:
  • Filters some of the Legacy Sonos ServiceId's (long since changed)
  • Handles duplicate "LastUpdate" (preference to HaDevice1)
  • Handles duplicate "Target"  (preference to SwitchPower1)
  • Handles duplicate "Status"  (preference to SwitchPower1)
  • Handles duplicate "ZoneName" (preference to UPnP DeviceProperties)
  • Suffixes "Heat" on all TemperatureSetpoint1_Heat Items
  • Suffixes "Cool" on all TemperatureSetpoint1_Cool Items
  • Suffixes "Fan" on all HVAC_FanOperatingMode1 Items
  • Suffixes "User" on all HVAC_UserOperatingMode1 Items
  • Filters "urn:schemas-micasaverde-com:device:avmisc:1" Items (not a ServiceId, old Sonos code throwback?)
  • Filters "urn:schemas-upnp-org:service:RenderingControl:1" Items (not a ServiceId, old Sonos code throwback?)

This gets a lot closer to a "duplicate-free" Item declaration file (at least for my configuration).  More data will be needed to know how much better it is in the wild.

Offline gduprey

  • Beta Testers
  • Full Member
  • *****
  • Posts: 122
  • Karma: +6/-1
Re: Tools: openHAB Item file generator
« Reply #13 on: January 31, 2015, 09:38:05 am »
Runs great and I like the changes.  The items file is looking better and better and requiring less manual tweaking.  I've said it before, but this is a great tool for the new OH/MiOS user to get going (especially if you have a lot of devices).

Not that you're looking for more work, but a few thoughts:

It might be nice for temperature items (i.e. '*CurrentTemperature*") to have a format more like "Temperature..... [%.1f ?F]" instead of the %d.  Not hard to fix at all, but for OH newcomers, all the hints that can be put in for formats help.

On my thermostats, I'm seeing one of the setpoints items marked as String (others are correct).  Again, super easy to fix, but thought you might want to know.  Item is:
Code: [Select]
String   DownstairsThermostatSetpointTarget "Downstairs Thermostat FIXME SetpointTarget [%s]" {mios="unit:nightmare,device:16/service/urn:upnp-org:serviceId:TemperatureSetpoint1/SetpointTarget"}         /** SAMPLE=[67] **/

Not sure how possible this is, but as there is a catalog of room groups in the items file, it might be nice to add the room group to each item that is in a room.

These are all pretty minor changes (the result, not speaking of the work), and the tool is pretty great as is.

Gerry

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #14 on: January 31, 2015, 05:40:20 pm »
New version uploaded, this one:

  • Added Group definition GDevices
  • Changed Room-based Group generation to GRoom<roomId> "<roomDescription>"
  • Reduced the number of FIXME strings in Device-Item Descriptions
  • Changed Device-Item generation to use Group "GDevices" instead of "GDevice"
  • Changed Device-Item generation to add Group "GRoom<id>" if the device is in a Room (not for ID and DeviceStatus attr)
  • Changed Scene-Item generation to add Group "GRoom<id>" if the scene is in a Room
  • Add " Scene" at the end of the Main Scene-Item description
  • Use "[%.1f F]" format for various Temperature-related Device-Items
  • Fix Typo in Service Alias TemperatureSetpoint1_Cool and TemperatureSetpoint1_Heat
  • Dumps a list of Duplicate Item definitions that need to be manually fixed by the user