We have moved at community.getvera.com

Author Topic: Tools: openHAB Item file generator  (Read 28915 times)

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #15 on: January 31, 2015, 05:45:17 pm »
It might be nice for temperature items (i.e. '*CurrentTemperature*") to have a format more like "Temperature..... [%.1f ?F]" instead of the %d.  Not hard to fix at all, but for OH newcomers, all the hints that can be put in for formats help.

Done for most of the Temperature-related items I could find valid ServiceId's for.  I hardcoded the UoM to "F", so apology upfront to those in the Metric world ;)

Quote
On my thermostats, I'm seeing one of the setpoints items marked as String (others are correct).  Again, super easy to fix, but thought you might want to know.  Item is:
Code: [Select]
String   DownstairsThermostatSetpointTarget "Downstairs Thermostat FIXME SetpointTarget [%s]" {mios="unit:nightmare,device:16/service/urn:upnp-org:serviceId:TemperatureSetpoint1/SetpointTarget"}         /** SAMPLE=[67] **/

I'm not sure where that comes from.  In Vera, there are _Heat and _Cool variants of the ServiceId as well as "CurrentSetpoint", but I've not seen "SetpointTarget" anywhere.

For now, I've left it off the magic fix-it list that I maintain inside the XSLT file.


Thanks for the suggestions, most of them are now inside the latest version that I just published.
« Last Edit: January 31, 2015, 05:55:14 pm by guessed »

Offline gduprey

  • Beta Testers
  • Full Member
  • *****
  • Posts: 122
  • Karma: +6/-1
Re: Tools: openHAB Item file generator
« Reply #16 on: February 01, 2015, 12:32:45 pm »
Howdy,

Didn't realize I had so many duplicates!  I wrote a gawk script that 'fixes' duplicates.  Much like your FIXME references, this just finds all duplicate item names and prefixes them with a 'DUP_x' where x is the device/scene # (there is logic for duplicates within a single device too -- which I had for a device with two different level values).  Makes it immediately parsable for mios testing and still easy to find the dups for manual fixup.

I'm not 100% sure this will work on mac.  It should work with any modern awk implementation, but I had seen a few old, conflicting reports that certain more recent (i.e. last 15 years) awk features didn't work under MacOSX.  Hopefully that is no longer the case. 

I added some logic in the miosTransform to direct duplicate output to a file and only warn if there are actually any dups.  Then check to see if they have gawk and run the de-duplicator if so (if they don't, it pretty much works as before, printing the list of duplicates and noting that installing gawk and re-running will get de-duplication).

Feel free to merge this into your tool or not (I won't be offended in the slightest if not). 

Attached is a .zip (this forum wont let me attach a .tar??) with a modified miosTransform.sh and the awk script.

Gerry
« Last Edit: February 01, 2015, 01:53:28 pm by gduprey »

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #17 on: February 01, 2015, 01:41:48 pm »
If duplicates, due to Device label-strings, are really common then I can add the DeviceId to ALL Item names that are emitted.

It'll make them uglier than the current format, but it'll be a lot closer to unique output from the script... without having to repair afterwards.

This would also break any existing dependencies (Sitemaps, Rules, etc) that are based upon the old naming model.  Not sure how much of a problem that really is at the moment, but it might be worth doing before too many users experience the issues.

Let me know what you think about the above change.

Anyhow, I've attempted to do any automated fixes directly in the XSLT.  My logic is that the XSLT is i18n enabled, and Shell typically isn't (without great care), so there's a better shot at creating a valid i18n-compatible Items file if all the logic can be pushed inside.

That may/may-not be the right approach, but it's the one I've taken so far ;)

Quote
there is logic for duplicates within a single device too -- which I had for a device with two different level values
For these cases, I've slowly been cataloging the Dup StateVariable names (that I'm aware of) and putting in a preference as to which one "gets" priority.  All others get the "state" Id added into their names to make them more unique.

See the XSLT Variable section "ItemName" to see what I mean.

If you know of specific others that should be added here, then let me know.


PS: gawk isn't on the Mac by default.  It has standard awk, but I don't know the functional differences between the two, and whether your script has a strong dependency upon stuff in gawk that's not in regular awk.

Offline gduprey

  • Beta Testers
  • Full Member
  • *****
  • Posts: 122
  • Karma: +6/-1
Re: Tools: openHAB Item file generator
« Reply #18 on: February 01, 2015, 02:11:17 pm »
Howdy,

First -- there was a typo in my uploaded .zip when I went and quoted all the env var references.  I've corrected that and updated the attachment in my previous post.

My thinking would be to not "penalize" the non-duplicate, non-troublesome item names.  Depending on how someone has maintained their Vera setup, all their names may come out fine, where as having all the device IDs means that someone will very likely want to edit every single item they keep.

I have about 320 out of 1300 item definitions with dups, but most are things I don't think I care about nor will likely leave an item defined for.  Most of the things I actually care about are fine.

I've checked and while gawk may not ship default on Mac, it is a one-line install command to add it.  I'm am depending on it for device/scene ID extraction as doing it without some new features that awk never had is a giant pain.  I had to install xsltproc on my stripped down ubuntu box, and for the benefit, doing either is a pretty minor additional step.  I would expect gawk (being pretty new and updated regularly) is pretty i18l compliant and I'm not doing anything in there that would typically break strings with i18l chars in them.

As for integrating it all into xslt, that would be great!  I am not super up on xslt (I've written a few minor transformations and can follow what you did mostly) and wouldn't quite know how to do duplicate checking and resolution.  But I'm all for a one-stop shop style experience and minimal dependencies/files.

My (so far, one) duplicate name within the same device is my power meter.  For reasons that make no sense, the device seem to be publishing a LightSensor (it has no light sensor, so this is bogus) and a generic sensor level (which doesn't seem to match anything reported in the more power-specific items).  The items are:

Code: [Select]
Number   DUP_30_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%d]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:GenericSensor1/CurrentLevel"}             /** SAMPLE=[48] **/
String   DUP_30_1_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%s]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:LightSensor1/CurrentLevel"}             /** SAMPLE=[0] **/

If the de-dupper didn't add the 'DUP_30_1' to the second one, then they'd have duplicate item names.

Gerry

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #19 on: February 01, 2015, 04:44:56 pm »
New version uploaded, this one:

  • Looks for duplicate Device names, when building Item names, and appends DeviceId for duplicates.
  • Looks for duplicate UPnP Variable names, when building Item names, and appends StateId for duplicates
  • Treats a fixed list of UPnP Variable names as "special" where it'll never append a StateId (eg. SwitchPower1/Status) but the other duplicates, within the same Device, will have it appended
  • Imported some shell script tweaks (quotes around variable expansions from @gduprey)

This should eliminate the bulk of duplicate situations.  If folks are still seeing them, please PM (privately) me a copy of your user_data.xml file (or the output of the list of Duplicates generated)

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #20 on: February 01, 2015, 05:02:21 pm »
My thinking would be to not "penalize" the non-duplicate, non-troublesome item names.  Depending on how someone has maintained their Vera setup, all their names may come out fine, where as having all the device IDs means that someone will very likely want to edit every single item they keep.

I have about 320 out of 1300 item definitions with dups, but most are things I don't think I care about nor will likely leave an item defined for.  Most of the things I actually care about are fine.

The latest version will append the various (scoped) "id"s only when a duplicate is detected.

I'm hoping it will fix all of the cases occur in practice but there are some off-beat cases where it won't so I've left in the duplicate-detect-and-print stuff.

Quote
I had to install xsltproc on my stripped down ubuntu box, and for the benefit, doing either is a pretty minor additional step.

That's odd.  My default Ubuntu/Linarto on the Odroid C1 had it pre-installed.  There must be install variations on Ubuntu 14.04 LTS.

Quote
I would expect gawk (being pretty new and updated regularly) is pretty i18l compliant and I'm not doing anything in there that would typically break strings with i18l chars in them.

Well, with the version I have now, it's all contained in the XSLT.  Only duplicate-reporting is done outside, so it could potentially break with non Latin-1 assumptions.  To be fair, I'm not even sure that Item names can be anything other than ASCII-7, so it could all be moot anyhow.

Quote
As for integrating it all into xslt, that would be great!  I am not super up on xslt (I've written a few minor transformations and can follow what you did mostly) and wouldn't quite know how to do duplicate checking and resolution.  But I'm all for a one-stop shop style experience and minimal dependencies/files.

Well, there's a version now.  Hopefully it's better than the last one (at generating unique-ish Item names)... time will tell.

Quote
My (so far, one) duplicate name within the same device is my power meter.  For reasons that make no sense, the device seem to be publishing a LightSensor (it has no light sensor, so this is bogus) and a generic sensor level (which doesn't seem to match anything reported in the more power-specific items).  The items are:

Code: [Select]
Number   DUP_30_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%d]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:GenericSensor1/CurrentLevel"}             /** SAMPLE=[48] **/
String   DUP_30_1_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%s]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:LightSensor1/CurrentLevel"}             /** SAMPLE=[0] **/

I have a few of those myself, where Vera has "added" Thermostat StateVariables to Lights (etc).  Fun.   Lovely little corruptions like that used to occur frequently in UI4, due to threading/concurrency problems that [ultimately] caused LuaState corruptions.


Thanks for all the help in testing this, and in pushing me to do better! 

Offline AgileHumor

  • Hero Member
  • *****
  • Posts: 984
  • Karma: +51/-27
  • KISS
Re: Tools: openHAB Item file generator
« Reply #21 on: February 01, 2015, 08:50:23 pm »
Is there a solution for non-linux or mac users?

I have 500 devices and looking for a simpleway to Jumpstart OpenHab conversion.

Thanks!

WMC Leviton:18xVPE06,8xVRS15,3xVRP03-W,2xVRR15,4xVRCS4,2xVRCS2,VP00R,8xVRS15 Aeon:5xDSC06106,4xDSC24,4xDSC25,12xDSB29,2xDSC11,4xDSB54,DSB05,3xDSA22,DSA38,2xDSA03202B,DSB09104,HEM Other:3xYale,12xHSM100v3,7xSP103,45604,WDHA-12,SSA2USR,EVLCD1T,6xWWA02A,7xIPC-HFW2100,URTSI,Hue,Russound,OpenSprinker

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #22 on: February 01, 2015, 09:02:16 pm »
Cygwin maybe?  The script is fairly simple and there is definitely a Cygwin version of the core xsltproc routine.

Alternatively I can give you instructions so you can send me a copy of your user_data (privately) and I'll run the process over it.  It has sensitive data in it, so this isn't the safest model, but I'll treat it with care.

Offline BOFH

  • Sr. Hero Member
  • ******
  • Posts: 2409
  • Karma: +112/-140
Re: Tools: openHAB Item file generator
« Reply #23 on: February 01, 2015, 09:11:44 pm »
Is there a solution for non-linux or mac users?

I have 500 devices and looking for a simpleway to Jumpstart OpenHab conversion.

Thanks!

Perhaps use one of the Linux live file system CD's? Boot from it into a live linux environment and run it from there?

http://www.linux.com/learn/tutorials/377416:the-five-best-linux-live-cds

Vera3 UI5 UI7 Edge Plus
Trane TZEMT400AB32 | Schlage BE369 FE599 | GE 45601 45602 45603 45604 45606 45609 45631 | Intermatic HA01C HA03C HA05C HA07C CA600 CA3000 | Aeon DSC06106 | Telguard GDC1 | Foscam FI8910W FI8905W FI9821W | D-Link 930L | Wanscam JW0011 | ZModo ZPIBH13W

Offline AgileHumor

  • Hero Member
  • *****
  • Posts: 984
  • Karma: +51/-27
  • KISS
Re: Tools: openHAB Item file generator
« Reply #24 on: February 01, 2015, 09:27:17 pm »
Great Idea!

I enabled the Windows 8.1 Feature "Hyper-V" and was able to boot from this ISO without needing to install anything.
http://www.sysresccd.org/Download

Thanks!



Is there a solution for non-linux or mac users?

I have 500 devices and looking for a simpleway to Jumpstart OpenHab conversion.

Thanks!

Perhaps use one of the Linux live file system CD's? Boot from it into a live linux environment and run it from there?

http://www.linux.com/learn/tutorials/377416:the-five-best-linux-live-cds


WMC Leviton:18xVPE06,8xVRS15,3xVRP03-W,2xVRR15,4xVRCS4,2xVRCS2,VP00R,8xVRS15 Aeon:5xDSC06106,4xDSC24,4xDSC25,12xDSB29,2xDSC11,4xDSB54,DSB05,3xDSA22,DSA38,2xDSA03202B,DSB09104,HEM Other:3xYale,12xHSM100v3,7xSP103,45604,WDHA-12,SSA2USR,EVLCD1T,6xWWA02A,7xIPC-HFW2100,URTSI,Hue,Russound,OpenSprinker

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #25 on: February 01, 2015, 09:31:11 pm »
If you can find some sort of XSLT Processor for Windows (eg SAXON), then the steps are really just:

a) In a browser, run http://MIOS_IP:49451/data_request?id=user_data&output_format=xml
Where MIOS_IP is the 192.168.x.x type of IP address of Vera.  Save the results to a file (my script calls this user_data.xml)

b) Run this XML through an XSLT Processor, using the supplied Transform file (miosTransform.xslt).
If it's done without parameters, it'll assume your config is called "house" (this is the default inside the top of the .xslt file)


The output of this, if the XSLT Processor is working correctly, will be the Text file itself.  I looked for a "safe" online site to do it, but it was never clear how they treated (and how long they kept) any files/content uploaded.

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #26 on: February 02, 2015, 12:03:20 am »
@AgileHumor was nice enough to PM me his details, and I converted both XML's into their counterpart openHAB Item files. 

Since he runs 2x Vera's, it highlighted a problem with the uniqueness-detection I was using - so I'll have to give some thought to how to address multi-Vera, multi Item file, situations a little better.

On a plus note, he uses a lot of Plugins that I've not been exposed to yet, so I should be able to glean some better defaults, and ServiceId aliases, for the 1.7 version.


I think the conversion tools are sufficient for check-in at this point, so I'll go ahead and make a PR to get "this version" into 1.7

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #27 on: February 03, 2015, 12:50:42 pm »
New version uploaded, this one:

  • Omits several internal ServiceId/StateVariable combinations (usernames, passwords, Plugin config, Plugin internals/implementation data)
  • Makes Camera1/Timeout a Number Item
  • Makes the "System" attributes (at the end of the file) a commented-out block per this thread.

Offline guessed

  • Moderator
  • Master Member
  • *****
  • Posts: 5301
  • Karma: +92/-22
  • Release compat is not a bolted-on afterthought
Re: Tools: openHAB Item file generator
« Reply #28 on: February 04, 2015, 01:37:59 pm »
These tools have been submitted to openHAB for formal inclusion under PR #2080.

I'll continue to use Box.net for the "latest" versions, with periodic check-pointing back to openHAB's GitHub account as needed.  Once approved, and merged into openHAB, others will have a more standard mechanism to add extra rules to the conversion/generation system.

Offline Brientim

  • Sr. Hero Member
  • ******
  • Posts: 2497
  • Karma: +78/-7
Tools: openHAB Item file generator
« Reply #29 on: February 05, 2015, 04:30:38 am »
@guessed, finally got a moment to implement OH and the item file generator.  Even though it will be a few days before I will get back to it...

Whoever may decide to venture...

How simple has @guessed made this process? Extremely!

Thanks mate and I look forward to getting looking at this instead to work stuff.