Author Topic: Vera Plugins - Development environment and tools needed  (Read 5606 times)

Offline Dragos Perju

  • MiOS Official
  • Global Moderator
  • Sr. Newbie
  • *****
  • Posts: 48
  • Karma: +4/-0
  • Think different
Vera Plugins - Development environment and tools needed
« on: August 21, 2018, 12:53:52 pm »
This forum is one of the greatest forums (if not the greatest) in the industry of home automation. A lot of things happened in the past that made some of you to be on the edge of switching to another product. Some of you already did that but you are still visiting the forums. One of the biggest challenges we had were low resources and starting with this week this changed. We now have the numbers to make things happen.

Our vision is to offer our plugin developers all the necessary tools to build great apps (plugins) that our users will enjoy. We want to build a new app store that will support mobile applications and that will keep backwards compatibility (as much as possible with JSON interface descriptions). We want to build a visual tool that will help designers to define advanced UX for your apps and to be able to drag & drop controls and connect them to variables. We are going to build these tools for you developers out there are. And by doing that for you we will help all our clients by offering them new great apps. But even if we are more resourced now, this will take time. Until we will get to the first fixes of what's broken, to the first version of a new tool we would like to have your wishes.

Tell us what you wish for as developers to build great modern apps ready for mobile interaction, tell us what you would wish from future plugin apps as users.
We know the forum is full of older requests and there are so many things that can be pulled off from different topics but please take this new one as a restart.

Thank you
« Last Edit: November 29, 2018, 04:52:00 am by Sorin »
Dragos Perju,
Product Manager @ MiOS

Online akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6206
  • Karma: +276/-70
  • "Less is more"
Re: Vera Plugins - Development environment and tools needed
« Reply #1 on: August 21, 2018, 01:45:13 pm »
In the absence of this kind of support in the past, several of us have had to develop our own tools.

Development environment:

Of course, we are not talking about your own IDE, but some tools to help would be good.  I'm thinking of the excellent LuaTest, by @RexBeckett, which added the sadly lacking UI5/7 print and pretty statements.  A further evolution of this is given by the LuaTest window in the most excellent AltUI interface by @amg0.  For a good IDE, it's very hard to beat ZeroBrane Studio, which also has a third-party add-on to debug Vera in situ.

Frustrated with what was once incredibly slow approval in the MiOS App Store, not to mention the most appalling UI, we made our own 'Alternative App Store', which you'd also need AltUI to access, but which works on both Vera and openLuup installations.  The key feature here is that it is not a repository, it simply holds the metadata for GitHub, which, arguably, is THE go-to place for open-source code.  Additionally, this is self-policing.  Users can add plugins in an instant, with no vetting.  You may not like this approach.

The Luup environment itself is quite incomplete in terms of things you can do programmatically (versus HTTP requests) and this, itself, needs improvement.  No decent API should return invalid language objects (but Luup does, in at least on instance.)

These are just initial thoughts.  For me, the Vera UI is of no interest.  Real HA should be automatic - you shouldn't often need to look at the configuration interface.
3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Online rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 767
  • Karma: +116/-1
Re: Vera Plugins - Development environment and tools needed
« Reply #2 on: August 21, 2018, 03:03:58 pm »
My biggest gripe about the current environment is that it is difficult to gather the full scope of information available about a device. For example, the service files (those named S_blahblahblah.xml by convention) describe (or can) in detail the state variables (with data type and default value, and even range of legal values), and actions that the service can perform. It is extremely difficult to get to this information in the current Luup environment.

In general, I'd like to have access to EVERYTHING the system knows about a device, room, scene, etc., without having to make an HTTP/Luup request (lu_sdata, user_data, scene, room, invoke etc.) to 127.0.0.1 and parse the (often large) response, which is just crazy inefficient. It's all in there already. Why isn't just exposed in the "luup" global (or whatever mechanism you might ultimately replace that with)?

[As I wrote this, @akbooer said the same, in effect. You know. Great minds think alike. ;) ]

It would be useful to know for any particular device, not just if it implements a service, but what elements of the service does it implement? As an example, not all devices using the Dimming1 service implement the ramp parameters. Not all Color1 service devices may support color temperature (or do temperature but not RGB). Not all devices implementing the HVAC_UserOperatingMode service implement energy modes. I'd like to be able to query that and alter behavior of my plugins accordingly. This means it also has be defined for each device/service, but that could be as simple as the existence of an action in the implementation file corresponding to the action definition in the service file (i.e. if the service file defines a DoFoo action, but the implementation doesn't have it).

This leads to even your UI having a very schizophrenic set of rules for what to present for a device. For any given device, your UI code may look at device category, or device type, or in the worst of cases, the filename of the static JSON file. This has created a morass of exceptions in your own code, and cuts off plugins from showing basic UIs. For example, an instance of the old and ubiquitous VirtualSwitch plugin should be recognized as a binary switch and presented as such, with no further effort than setting its category to 3. A broader view of device capability needs to be taken, with more abstraction, and a consistent rule decided upon for determining what "basic" (default) UI should be present, such as using the device category, or defining the notion of a "primary" service which, from among several services the device may implement, is the one used to present a defaut UI if the plugin doesn't provide one itself). In my perfect world, the VirtualSwitch plugin should be able to exist as a definition only (e.g. "VSwitch1 implements SwitchPower1" and nothing more), and have no code behind it all.

Make the mobile apps (Android, iPhone) do it the same way. If a plugin declares itself to be a binary switch, the app should present a binary switch UI by default.

The static JSON files are too limited in their capability, and contain too many omissions and redundancies, and actually end up being one of the most time-consuming parts of developing a plugin if you care what the UI looks like. Worse, the UI is far too limited on the plugin side, providing inadequate validation of input data, insufficient variety of controls for a modern UI, etc. There is nothing at all wrong, and everything to recommend IMO, ALTUI's model of just offloading that task entirely so that the plugin developers provide small JavaScript fragments for (non-default) dashboard cards and control panels/tabs. It's super easy. It will make your life easier too. Just make sure we have access to more current revisions of jQuery, the entirety of jQuery-UI, and Bootstrap.

Plugins should not be lumped into a single directory where they can step on each other. If two plugins both use the same external library and include it, but require different versions, one will always be stomping on the other. Plugins should live in package files, or separate subdirectories.

Creation of child devices should not require a Luup restart (and if I can call luup.create_device() with a parent ID, why do we have luup.chdev?).

The UI needs to do a better job of seeing device changes. However it is that the LoadTime and DataVersion mechanism ends up being insufficient in the current UI to detect many device changes and update itself properly... those problems need to be drilled out and fixed. Hard-reloads of the browser should not be necessary except in the most dire circumstances. I've developed my own Dashboard UI and I never have to refresh it, so I know it's possible. The LoadTime/DataVersion mechanism is one of the most ingenious and useful things you've done in the API/requests interface, and it works great, but somehow your own UI manages to not handle it well.

I'd like to be able to watch a device and receive an event notification for when it goes offline or ready, without having to poll is_ready() for it. Elaborating on this, I'd like to be able to tell Luup not to call the plugin's startup function until all device network startup has finished (ZWave, Bluetooth, etc.). Extra points for being able to establish dependencies between devices and/or plugins, and start them up in dependency order.

Jobs, and watching jobs should be more robust. The current job watch callback is called for every job everywhere when used, and the callback has to determine if the subject job belongs to the plugin. It should more closely model variable callbacks, where the callback is only called for the device that made the watch call. Likewise, when a job is started (by calling luup.call_action()), I should be able to put a watch on just that job. Super bonus points: I'd like to have: (a) luup.start_job( function, {option_name=value} ), where any named or anonymous function can be submitted as a job; (b) add flags (maybe using the {option_name=value} construct) to luup.call_action() so that any action can be started as a job; and (c) the job actions should include timeouts for job start and completion (i.e. job gets a status=8 (start delayed) or 9 (completion timeout) watch callback if it hasn't started or completed, respectively, in the given time); and (d) kill_job(), to stop a running/waiting job.

Speaking of which, the variable watch callback should receive the device number of the device that made the watch call (be it a parent or child, as context may dictate). Look at "luup.device" you say? Well...

luup.device should go away. Don't have a changing global for a single device in what is otherwise a library context. Fix the places in the API where it's hard to figure out what context device we are talking about (like the variable watch callback above), and get rid of luup.device. In fact, the "luup" module should be free of all device context, and all functions reviewed and extended or fixed so that they, and any callbacks they make, can be fully reentrant (i.e. all Luup functions need to be given whatever info they need to work, and all callbacks need to pass in whatever device they apply to).

I'd like a luup function to tell me what the children of a parent device are without having to go through the entirety of luup.devices iteratively. I'd like a function to find a device by name. And one to give me all the devices in a room, or a named device in a room. Maybe a device search, with variable parameters, and wildcards? But efficient, because the days of 400-500 devices and more will be coming. I'd like this both in Lua and JS API.

I'd like to be able to delete a state variable without having to use an HTTP request. This may already exist, but it's not documented that I've seen, and I sure haven't found another way.

I'd like the option of separate logging per-plugin.

I'd like to see a mechanism for storing larger bits of data and structured data persistently for plugins that isn't state variables, or going to the filesystem directly. Right now, many plugins JSON-stringify the data and write the string to a state variable, but this always feels like a string size limit waiting to happen. It's also the case that we may want to make sure the data persists more often than user_data is rewritten. I think the general understanding is that user_data is written every six minutes, so a power event, for example, is pretty much guaranteed to lose data, it's just a question of how important that data is. You can't answer that question for all current and future devices and scenarios, so the approach taken should be as defensive as possible (assume every change is mission-critical). A mechanism should be provided for a plugin to store important state as immediately as possible.

Speaking of user_data... I generally view this as a fragile monolithic data store not unlike the Windows registry (think back to NT and before--it was easily damaged, with disastrous results). And the more devices you have, the bigger it is, and the longer it takes to store it, and the worse the pain if it gets damaged. I think this needs a rethink. I personally feel that a per-device store in fast, non-volatile storage would be vastly superior. Even changing it from a single file to a series of files per device in a subdirectory on JFFS2 (each file written as its device posts changes) would be better than the current monolith saved every six minutes. This would also facilitate implementing the immediately above request.

I also recently discovered that each instance of plugin loads not just itself, but any modules it uses, into separate memory space. This is apparently why plugins consume so much memory in the (real) Luup environment. I think it should be required that future plugins be fully re-entrant (supported by above-requested changes needed), so that only one copy of the plugin implementation needs to exist in memory for all instances (as is the case with child devices when "handleChildren" is used in the parent). And, all library modules loaded by plugin implementations should be shared. There's no reason two different plugins can't share the same copy of the dkjson, socket, or mime modules. Lua is designed to work that way, and somehow, your implementation breaks this. This inefficiency isn't welcome in a constrained-memory environment, and adds no discernible benefit (it was probably just expeditious to do it that way, which I understand, but it's bad). While the modules referenced may be small, modules introduced by the plugins themselves may not be. For the record, openLuup does not do this, it follows the regular Lua model--a module loaded is shared by all. It works great, with no special changes in the plugin code (unless you've done something inadvisable/generally regarded as poor style).

You need a PR tracking system that plugin developers have access to, and people to support the developer community.

That's it off the top of my head. I'm sure I'll think of more.
« Last Edit: August 21, 2018, 03:37:15 pm by rigpapa »
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3 sandbox.

Offline Dragos Perju

  • MiOS Official
  • Global Moderator
  • Sr. Newbie
  • *****
  • Posts: 48
  • Karma: +4/-0
  • Think different
Re: Vera Plugins - Development environment and tools needed
« Reply #3 on: August 21, 2018, 04:14:19 pm »
In the absence of this kind of support in the past, several of us have had to develop our own tools.

Development environment:

Of course, we are not talking about your own IDE, but some tools to help would be good.  I'm thinking of the excellent LuaTest, by @RexBeckett, which added the sadly lacking UI5/7 print and pretty statements.  A further evolution of this is given by the LuaTest window in the most excellent AltUI interface by @amg0.  For a good IDE, it's very hard to beat ZeroBrane Studio, which also has a third-party add-on to debug Vera in situ.

Frustrated with what was once incredibly slow approval in the MiOS App Store, not to mention the most appalling UI, we made our own 'Alternative App Store', which you'd also need AltUI to access, but which works on both Vera and openLuup installations.  The key feature here is that it is not a repository, it simply holds the metadata for GitHub, which, arguably, is THE go-to place for open-source code.  Additionally, this is self-policing.  Users can add plugins in an instant, with no vetting.  You may not like this approach.

The Luup environment itself is quite incomplete in terms of things you can do programmatically (versus HTTP requests) and this, itself, needs improvement.  No decent API should return invalid language objects (but Luup does, in at least on instance.)

These are just initial thoughts.  For me, the Vera UI is of no interest.  Real HA should be automatic - you shouldn't often need to look at the configuration interface.

Yes, we should have provided the necessary tools. Having developers build own tools so they can develop apps it's something that we want to change.
We want to feature a very fast approval process with this new app store but indeed I don't think that an un-curated environment of apps will benefit users. From a security stand point of view we are now part of a bigger family that takes security really serious.

The Luup environment needs improvement. No question about that, especially on validating input and both on respecting error codes or return invalid objects. If you have specific examples that you have in mind for certain calls please let me know. The next sprints will be about fixing things.

In terms of UI that's something that we still need and users use a lot. We want to remove hard-coded things, expose the right documentation and standardise   dynamic definitions for UX elements and the binding with environment and device variables.

Thank you for your feedback and feel free to add more.
Dragos Perju,
Product Manager @ MiOS

Offline Dragos Perju

  • MiOS Official
  • Global Moderator
  • Sr. Newbie
  • *****
  • Posts: 48
  • Karma: +4/-0
  • Think different
Re: Vera Plugins - Development environment and tools needed
« Reply #4 on: August 21, 2018, 04:38:20 pm »
My biggest gripe about the current environment is that it is difficult to gather the full scope of information available about a device. For example, the service files (those named S_blahblahblah.xml by convention) describe (or can) in detail the state variables (with data type and default value, and even range of legal values), and actions that the service can perform. It is extremely difficult to get to this information in the current Luup environment.

In general, I'd like to have access to EVERYTHING the system knows about a device, room, scene, etc., without having to make an HTTP/Luup request (lu_sdata, user_data, scene, room, invoke etc.) to 127.0.0.1 and parse the (often large) response, which is just crazy inefficient. It's all in there already. Why isn't just exposed in the "luup" global (or whatever mechanism you might ultimately replace that with)?

[As I wrote this, @akbooer said the same, in effect. You know. Great minds think alike. ;) ]

It would be useful to know for any particular device, not just if it implements a service, but what elements of the service does it implement? As an example, not all devices using the Dimming1 service implement the ramp parameters. Not all Color1 service devices may support color temperature (or do temperature but not RGB). Not all devices implementing the HVAC_UserOperatingMode service implement energy modes. I'd like to be able to query that and alter behavior of my plugins accordingly. This means it also has be defined for each device/service, but that could be as simple as the existence of an action in the implementation file corresponding to the action definition in the service file (i.e. if the service file defines a DoFoo action, but the implementation doesn't have it).

This leads to even your UI having a very schizophrenic set of rules for what to present for a device. For any given device, your UI code may look at device category, or device type, or in the worst of cases, the filename of the static JSON file. This has created a morass of exceptions in your own code, and cuts off plugins from showing basic UIs. For example, an instance of the old and ubiquitous VirtualSwitch plugin should be recognized as a binary switch and presented as such, with no further effort than setting its category to 3. A broader view of device capability needs to be taken, with more abstraction, and a consistent rule decided upon for determining what "basic" (default) UI should be present, such as using the device category, or defining the notion of a "primary" service which, from among several services the device may implement, is the one used to present a defaut UI if the plugin doesn't provide one itself). In my perfect world, the VirtualSwitch plugin should be able to exist as a definition only (e.g. "VSwitch1 implements SwitchPower1" and nothing more), and have no code behind it all.

Make the mobile apps (Android, iPhone) do it the same way. If a plugin declares itself to be a binary switch, the app should present a binary switch UI by default.

The static JSON files are too limited in their capability, and contain too many omissions and redundancies, and actually end up being one of the most time-consuming parts of developing a plugin if you care what the UI looks like. Worse, the UI is far too limited on the plugin side, providing inadequate validation of input data, insufficient variety of controls for a modern UI, etc. There is nothing at all wrong, and everything to recommend IMO, ALTUI's model of just offloading that task entirely so that the plugin developers provide small JavaScript fragments for (non-default) dashboard cards and control panels/tabs. It's super easy. It will make your life easier too. Just make sure we have access to more current revisions of jQuery, the entirety of jQuery-UI, and Bootstrap.

Plugins should not be lumped into a single directory where they can step on each other. If two plugins both use the same external library and include it, but require different versions, one will always be stomping on the other. Plugins should live in package files, or separate subdirectories.

Creation of child devices should not require a Luup restart (and if I can call luup.create_device() with a parent ID, why do we have luup.chdev?).

The UI needs to do a better job of seeing device changes. However it is that the LoadTime and DataVersion mechanism ends up being insufficient in the current UI to detect many device changes and update itself properly... those problems need to be drilled out and fixed. Hard-reloads of the browser should not be necessary except in the most dire circumstances. I've developed my own Dashboard UI and I never have to refresh it, so I know it's possible. The LoadTime/DataVersion mechanism is one of the most ingenious and useful things you've done in the API/requests interface, and it works great, but somehow your own UI manages to not handle it well.

I'd like to be able to watch a device and receive an event notification for when it goes offline or ready, without having to poll is_ready() for it. Elaborating on this, I'd like to be able to tell Luup not to call the plugin's startup function until all device network startup has finished (ZWave, Bluetooth, etc.). Extra points for being able to establish dependencies between devices and/or plugins, and start them up in dependency order.

Jobs, and watching jobs should be more robust. The current job watch callback is called for every job everywhere when used, and the callback has to determine if the subject job belongs to the plugin. It should more closely model variable callbacks, where the callback is only called for the device that made the watch call. Likewise, when a job is started (by calling luup.call_action()), I should be able to put a watch on just that job. Super bonus points: I'd like to have: (a) luup.start_job( function, {option_name=value} ), where any named or anonymous function can be submitted as a job; (b) add flags (maybe using the {option_name=value} construct) to luup.call_action() so that any action can be started as a job; and (c) the job actions should include timeouts for job start and completion (i.e. job gets a status=8 (start delayed) or 9 (completion timeout) watch callback if it hasn't started or completed, respectively, in the given time); and (d) kill_job(), to stop a running/waiting job.

Speaking of which, the variable watch callback should receive the device number of the device that made the watch call (be it a parent or child, as context may dictate). Look at "luup.device" you say? Well...

luup.device should go away. Don't have a changing global for a single device in what is otherwise a library context. Fix the places in the API where it's hard to figure out what context device we are talking about (like the variable watch callback above), and get rid of luup.device. In fact, the "luup" module should be free of all device context, and all functions reviewed and extended or fixed so that they, and any callbacks they make, can be fully reentrant (i.e. all Luup functions need to be given whatever info they need to work, and all callbacks need to pass in whatever device they apply to).

I'd like a luup function to tell me what the children of a parent device are without having to go through the entirety of luup.devices iteratively. I'd like a function to find a device by name. And one to give me all the devices in a room, or a named device in a room. Maybe a device search, with variable parameters, and wildcards? But efficient, because the days of 400-500 devices and more will be coming. I'd like this both in Lua and JS API.

I'd like to be able to delete a state variable without having to use an HTTP request. This may already exist, but it's not documented that I've seen, and I sure haven't found another way.

I'd like the option of separate logging per-plugin.

I'd like to see a mechanism for storing larger bits of data and structured data persistently for plugins that isn't state variables, or going to the filesystem directly. Right now, many plugins JSON-stringify the data and write the string to a state variable, but this always feels like a string size limit waiting to happen. It's also the case that we may want to make sure the data persists more often than user_data is rewritten. I think the general understanding is that user_data is written every six minutes, so a power event, for example, is pretty much guaranteed to lose data, it's just a question of how important that data is. You can't answer that question for all current and future devices and scenarios, so the approach taken should be as defensive as possible (assume every change is mission-critical). A mechanism should be provided for a plugin to store important state as immediately as possible.

Speaking of user_data... I generally view this as a fragile monolithic data store not unlike the Windows registry (think back to NT and before--it was easily damaged, with disastrous results). And the more devices you have, the bigger it is, and the longer it takes to store it, and the worse the pain if it gets damaged. I think this needs a rethink. I personally feel that a per-device store in fast, non-volatile storage would be vastly superior. Even changing it from a single file to a series of files per device in a subdirectory on JFFS2 (each file written as its device posts changes) would be better than the current monolith saved every six minutes. This would also facilitate implementing the immediately above request.

I also recently discovered that each instance of plugin loads not just itself, but any modules it uses, into separate memory space. This is apparently why plugins consume so much memory in the (real) Luup environment. I think it should be required that future plugins be fully re-entrant (supported by above-requested changes needed), so that only one copy of the plugin implementation needs to exist in memory for all instances (as is the case with child devices when "handleChildren" is used in the parent). And, all library modules loaded by plugin implementations should be shared. There's no reason two different plugins can't share the same copy of the dkjson, socket, or mime modules. Lua is designed to work that way, and somehow, your implementation breaks this. This inefficiency isn't welcome in a constrained-memory environment, and adds no discernible benefit (it was probably just expeditious to do it that way, which I understand, but it's bad). While the modules referenced may be small, modules introduced by the plugins themselves may not be. For the record, openLuup does not do this, it follows the regular Lua model--a module loaded is shared by all. It works great, with no special changes in the plugin code (unless you've done something inadvisable/generally regarded as poor style).

You need a PR tracking system that plugin developers have access to, and people to support the developer community.

That's it off the top of my head. I'm sure I'll think of more.

I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

While the first few iterations of our new UX will just add some info on top of already definition of static data so it is backwards compatible, this is where we want to go in the future and we hope that you are going to support us. We will make sure that tools and necessary hardware will be at your disposal so we can make your plugins available on the next generation of engine.

Regarding UX our strategy changed to a mobile first one. This means that we will implement a mobile oriented interface so that not only our Vera app, but also third party apps can display consistent information in fast way.

We will let you know the inputs that we gathered from you feedback and the status as they go into development. Thank again for such a fantastic answer and don't be shy on adding more if you have.
Dragos Perju,
Product Manager @ MiOS

Online rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 767
  • Karma: +116/-1
Re: Vera Plugins - Development environment and tools needed
« Reply #5 on: August 21, 2018, 04:54:38 pm »
Quote
I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

You have an app store full of plugins that are already dead or questionable on current firmware, and those should be culled. Looking at reviews in stores (Amazon, etc.) and here in the forums, Vera already gets a lot of kicks from new users about the swamp of dated plugins in the store that are "hit and miss" to figure out which work--they are a minefield for new users to walk through. And it's funny, in the poll thread, a lot of the plugins mentioned are in that group, they've just endured some level of community hacking to keep them on life support, and/or the community hasn't upgraded to more recent efforts that would better serve them.

But with that said, I would give you an emphatic YES. I will do anything and contribute any amount of effort required to keep up, and not just with my own plugins. I have also received many requests from within the community to update or replace older plugins (I did that just this weekend with a new Venstar ColorTouch thermostat plugin), or create new ones (the Intesis thermostat interface). I'll happily continue to do that, as long as Vera makes it possible and supports the effort.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3 sandbox.

Offline Buxton

  • Full Member
  • ***
  • Posts: 146
  • Karma: +9/-0
Re: Vera Plugins - Development environment and tools needed
« Reply #6 on: August 21, 2018, 05:23:59 pm »
" a fragile monolithic data store not unlike the Windows registry"

I believe this to be a core, foundational landmine in the luup model that if left as is, will inevitably detonate.   The model definitely needs a re-think and IMHO decentralization.  I've had to edit a corrupt user_data file by hand several times and it's always an unpleasant experience. 

One can make the argument that Windows survives with its core registry, but from what I understand, there are layers of protections to shield the registry from corruption.  So if Mios stays with a centralized model, at the least they need to emulate this type of protection.  On the other hand, if MIOS moves towards a decentralized Linux type model, to implement the change, while maintaining existing plugins (especially those that read and write directly to user_data), will be a challenge.

Offline Sammy2

  • Hero Member
  • *****
  • Posts: 870
  • Karma: +5/-5
Re: Vera Plugins - Development environment and tools needed
« Reply #7 on: August 21, 2018, 05:54:18 pm »
I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

While the first few iterations of our new UX will just add some info on top of already definition of static data so it is backwards compatible, this is where we want to go in the future and we hope that you are going to support us. We will make sure that tools and necessary hardware will be at your disposal so we can make your plugins available on the next generation of engine.

Regarding UX our strategy changed to a mobile first one.

This means that we will implement a mobile oriented interface so that not only our Vera app, but also third party apps can display consistent information in fast way.

We will let you know the inputs that we gathered from you feedback and the status as they go into development. Thank again for such a fantastic answer and don't be shy on adding more if you have.

NO! I do not want to be doing set up on a mobile device. I still want to do it with a keyboard and a mouse. Going mobile may be considered a move forward, but for speed and ease it is a move backward.
« Last Edit: August 21, 2018, 05:56:42 pm by Sammy2 »

Online akbooer

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6206
  • Karma: +276/-70
  • "Less is more"
Re: Vera Plugins - Development environment and tools needed
« Reply #8 on: August 21, 2018, 06:15:25 pm »
If you have specific examples that you have in mind for certain calls please let me know. The next sprints will be about fixing things.

Since you ask, the simple request:

Code: [Select]
luup.attr_get "foo"

when "foo" does not exist as an attribute, returns... NOTHING, apparently.  I don't mean that it returns the Lua representation nil, I don't mean an empty string, I mean nothing.  This gives unexpected, syntax-related, results when the value is tested.  This is an unforgivable mistake, that really does need fixing.  It should, of course, return nil.

3x Vera Lite-UI5/Edge-UI7, 25x Fibaro, 23x TKB, 9x MiniMote, 2x NorthQ Power, 2x Netatmo, 1x Foscam FI9831P, 9x Philips Hue,
Razberry, MySensors Arduino, HomeWave, AltUI, AltHue, DataYours, Grafana, openLuup, ZWay, ZeroBrane Studio.

Offline rstrouse

  • Hero Member
  • *****
  • Posts: 836
  • Karma: +30/-9
Re: Vera Plugins - Development environment and tools needed
« Reply #9 on: August 21, 2018, 06:58:26 pm »
@Dragos

I have developed both of the current Vera pool plugins and I want to chime in on my perspective of the UI and the device contexts.   The mobile apps seem to focus on a single control per device which is not the case for many types of devices and I sincerely hope that this is not the direction Vera is headed.  I have included some items related to the pool control plugins to give you an idea of what I am talking about.  There are others including chemistry control, scene lighting, MagicStream, and pumps that have multiple controls affiliate with the single device.  If each of the controls had its own generic device panel the experience would just plain suck as my pool, for instance would have 50+ panels to control the pool and manage its functions.

The up/down increment control was broken by an errant css background setting in your core stylesheets so that is why the chlorinator now displays with goofy looking buttons. But this screensot closely represents what is required to control two types of devices attached to most pool automation.

Intellichlor --40 is a chlorinator device.  Associated with this are settings for both pool and spa mode indicating how much power to apply to the chlorinator as well as a shock cycle countdown that runs the chlorinator at full power for a period of time.

Spa Heat and Pool Heat are heater destinations not sources.  Although these look like thermostats they are not.  In most pool automation panels the heat source will switch depending on the current body being fed as well as the variants in the temperature and the equipment installed.  This also means that the thermostat device type is just wrong for both control and management.  For instance, if I had a heat pump tied to the heating panels would need to include the preferences for the operating heat pump to maintain the temperature of both the pool and the spa.  This is how Jandy and Pentair (the two most common) have automated their controls.  I presume this to be the same with Hayward but I have no experience with it.

At some point in the past it looks like the static JSON was going to contain all the metadata required to render the controls and define the behavior for the device, but somewhere along the line the focus changed.  Shortcomings in the static JSON were augmented with display and behavior based upon hardcoded functions of particular DeviceTypes.  A quick perusal of the javascript code reveals all these special cases.  Start at returnDeviceContainer then follow on from there as even Virtual Switch has special handling that changes the css, script, and the binding.

I guess what I am trying to say is that as a plugin developer I can never create a new device type that has features or functions needed for a complex device given the amount of effort that internal staff have placed into View.js & Interface.js.  Since DOMPurify is implemented getting over things like the broken css and extending the controls are impossible without rewriting a full UI as amg0 has done.  Who has time for that except maybe amg0. 

My hope is that the css gets fixed in the future and is part of your sprint.  If it doesn't I'll need hack the css file (The hover state works by the way).  Like rigpapa alluded to, not all the parameters in the static JSON are honored or do not operate the same based upon the device type.  When you are building a custom device type this also means that you can be pretty sure it won't work on the mobile device.

I am encouraged and hope to create plugin devices that will work on the mobile platform.  I can't wait to rework my plugins to try it out.  I hear people say that home automation isn't about control so you shouldn't need an interface.  There are things that Vera should do that involve control.  Here's just a few scenarios;  You are sitting in the backyard and simply want to turn on the water features.  You need to run a shock cycle overnight after 20 kids swam in the pool all day.  You want to turn on some lights because you have friends coming over.  That is control and not automation and these all require an effective interface.  It sounds like you are headed down the Bootstrap and maybe even the TypeScript path for your tech stack given your comments below.  Perhaps some theming is in our future as well as real powerful expansion of the UX. 

In the end, I created my plugins because I needed them myself.  I shared them in the hope that someone else might need them thus expanding the Vera herd.  There is strength in numbers so long as you are standing somewhere near the middle.  And I don't use the app store because turned out to be too much of a PITA to get the code published and I felt like it was time I would never get back.

 
1xVera3 1.7.619, 4xLinear WT00Z, 3xLinear WS15Z, 1xLeviton VRCZ4, 10xCooper RF9540, 1xLeviton VRFI10, 1xLeviton VP00R, 2xLinear GD00Z-4, 1xGE/Jasco 45612, 1xGE/Jasco 45610,  4xGE/Jasco 45605, 2xYale YRD220-ZW-619, 1xCaddx NX584, Autelis Intellitouch Pool Control, OpenSprinkler

Offline melih

  • Administrator
  • Full Member
  • *****
  • Posts: 130
  • Karma: +34/-5
Re: Vera Plugins - Development environment and tools needed
« Reply #10 on: August 21, 2018, 07:38:22 pm »
I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

While the first few iterations of our new UX will just add some info on top of already definition of static data so it is backwards compatible, this is where we want to go in the future and we hope that you are going to support us. We will make sure that tools and necessary hardware will be at your disposal so we can make your plugins available on the next generation of engine.

Regarding UX our strategy changed to a mobile first one.

This means that we will implement a mobile oriented interface so that not only our Vera app, but also third party apps can display consistent information in fast way.

We will let you know the inputs that we gathered from you feedback and the status as they go into development. Thank again for such a fantastic answer and don't be shy on adding more if you have.

NO! I do not want to be doing set up on a mobile device. I still want to do it with a keyboard and a mouse. Going mobile may be considered a move forward, but for speed and ease it is a move backward.

I agree!!!

We should give equal weight to both Mobile and Desktop!

Offline Sammy2

  • Hero Member
  • *****
  • Posts: 870
  • Karma: +5/-5
Re: Vera Plugins - Development environment and tools needed
« Reply #11 on: August 21, 2018, 07:44:05 pm »
I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

While the first few iterations of our new UX will just add some info on top of already definition of static data so it is backwards compatible, this is where we want to go in the future and we hope that you are going to support us. We will make sure that tools and necessary hardware will be at your disposal so we can make your plugins available on the next generation of engine.

Regarding UX our strategy changed to a mobile first one.

This means that we will implement a mobile oriented interface so that not only our Vera app, but also third party apps can display consistent information in fast way.

We will let you know the inputs that we gathered from you feedback and the status as they go into development. Thank again for such a fantastic answer and don't be shy on adding more if you have.

NO! I do not want to be doing set up on a mobile device. I still want to do it with a keyboard and a mouse. Going mobile may be considered a move forward, but for speed and ease it is a move backward.

I agree!!!

We should give equal weight to both Mobile and Desktop!

That said, getting the android mobile app to match the web interface is necessary. Right now, custom Plugins are unusable on the app in android but seem just fine in Apple. Good thing too as my wife would not be pleased with having to log in to home.getvera.com everytime she wanted to use the app.. of course we just use ImperiHome instead because the Vera Interface is not native to use. It's okay for setting things up but not for using it to control when you want to do something outside of what is already automated, like turn on the pool or the spa or change the AC temp.

Offline melih

  • Administrator
  • Full Member
  • *****
  • Posts: 130
  • Karma: +34/-5
Re: Vera Plugins - Development environment and tools needed
« Reply #12 on: August 21, 2018, 08:08:24 pm »
I am profoundly impressed by the level of detail that you provided as feedback. On some things you mentioned (and believe me we will take them one by one in R&D and create tickets) it sounds that backwards compatibility would brake in order to fix them the right way. @rigpapa and @akbooer you would take the time to update the plugins so they run on such a firmware (of course you would have early access so you can test and develop your modifications)?

While the first few iterations of our new UX will just add some info on top of already definition of static data so it is backwards compatible, this is where we want to go in the future and we hope that you are going to support us. We will make sure that tools and necessary hardware will be at your disposal so we can make your plugins available on the next generation of engine.

Regarding UX our strategy changed to a mobile first one.

This means that we will implement a mobile oriented interface so that not only our Vera app, but also third party apps can display consistent information in fast way.

We will let you know the inputs that we gathered from you feedback and the status as they go into development. Thank again for such a fantastic answer and don't be shy on adding more if you have.

NO! I do not want to be doing set up on a mobile device. I still want to do it with a keyboard and a mouse. Going mobile may be considered a move forward, but for speed and ease it is a move backward.

I agree!!!

We should give equal weight to both Mobile and Desktop!

That said, getting the android mobile app to match the web interface is necessary. Right now, custom Plugins are unusable on the app in android but seem just fine in Apple. Good thing too as my wife would not be pleased with having to log in to home.getvera.com everytime she wanted to use the app.. of course we just use ImperiHome instead because the Vera Interface is not native to use. It's okay for setting things up but not for using it to control when you want to do something outside of what is already automated, like turn on the pool or the spa or change the AC temp.

what do you think about UI8 proposal we have? http://forum.micasaverde.com/index.php/topic,103076.msg412963.html#msg412963

Online rigpapa

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 767
  • Karma: +116/-1
Re: Vera Plugins - Development environment and tools needed
« Reply #13 on: August 21, 2018, 08:47:38 pm »
what do you think about UI8 proposal we have? http://forum.micasaverde.com/index.php/topic,103076.msg412963.html#msg412963

Melih, I think the design is clean and attractive, so that's off to a good start. For me, the key is speed, because I have over 100 Z-wave devices and 50+ plugin/virtual/child devices, and the current UI really bogs down, even on my desktop i7. Another system I use with Vera (Hass) allows you to create views, and lets you navigate between views, and choose which view is shown by default. If you're going in to turn on one light, it's very quick to jump in, switch to the view where you know the device is (because you put it there), and then command that device. The current Vera UI takes 30-40 seconds on my phone just to load and become responsive to touches/clicks, and then I can start digging through Vera's fixed organizational structure of my devices to find the one I want. So if that clean, sleek interface appearance is coupled to clean, sleek usability, you'll be leaps ahead of what's there today.

By the way, your link is to a thumbnail that's too small (no bigger when clicked). I had seen a larger image elsewhere in another announcement, so I knew what you were referring to, but you may want to go back and post a larger image in your linked thread.
Author of Reactor, DelayLight, SiteSensor, Rachio, Deus Ex Machina II, Intesis WMP Gateway, Auto Virtual Thermostat and VirtualSensor plugins. Vera Plus w/100+ Z-wave devices. Vera3 sandbox.

Offline Dragos Perju

  • MiOS Official
  • Global Moderator
  • Sr. Newbie
  • *****
  • Posts: 48
  • Karma: +4/-0
  • Think different
Re: Vera Plugins - Development environment and tools needed
« Reply #14 on: August 22, 2018, 03:49:03 am »
@Dragos

I have developed both of the current Vera pool plugins and I want to chime in on my perspective of the UI and the device contexts.   The mobile apps seem to focus on a single control per device which is not the case for many types of devices and I sincerely hope that this is not the direction Vera is headed.  I have included some items related to the pool control plugins to give you an idea of what I am talking about.  There are others including chemistry control, scene lighting, MagicStream, and pumps that have multiple controls affiliate with the single device.  If each of the controls had its own generic device panel the experience would just plain suck as my pool, for instance would have 50+ panels to control the pool and manage its functions.

The up/down increment control was broken by an errant css background setting in your core stylesheets so that is why the chlorinator now displays with goofy looking buttons. But this screensot closely represents what is required to control two types of devices attached to most pool automation.

Intellichlor --40 is a chlorinator device.  Associated with this are settings for both pool and spa mode indicating how much power to apply to the chlorinator as well as a shock cycle countdown that runs the chlorinator at full power for a period of time.

Spa Heat and Pool Heat are heater destinations not sources.  Although these look like thermostats they are not.  In most pool automation panels the heat source will switch depending on the current body being fed as well as the variants in the temperature and the equipment installed.  This also means that the thermostat device type is just wrong for both control and management.  For instance, if I had a heat pump tied to the heating panels would need to include the preferences for the operating heat pump to maintain the temperature of both the pool and the spa.  This is how Jandy and Pentair (the two most common) have automated their controls.  I presume this to be the same with Hayward but I have no experience with it.

At some point in the past it looks like the static JSON was going to contain all the metadata required to render the controls and define the behavior for the device, but somewhere along the line the focus changed.  Shortcomings in the static JSON were augmented with display and behavior based upon hardcoded functions of particular DeviceTypes.  A quick perusal of the javascript code reveals all these special cases.  Start at returnDeviceContainer then follow on from there as even Virtual Switch has special handling that changes the css, script, and the binding.

I guess what I am trying to say is that as a plugin developer I can never create a new device type that has features or functions needed for a complex device given the amount of effort that internal staff have placed into View.js & Interface.js.  Since DOMPurify is implemented getting over things like the broken css and extending the controls are impossible without rewriting a full UI as amg0 has done.  Who has time for that except maybe amg0. 

My hope is that the css gets fixed in the future and is part of your sprint.  If it doesn't I'll need hack the css file (The hover state works by the way).  Like rigpapa alluded to, not all the parameters in the static JSON are honored or do not operate the same based upon the device type.  When you are building a custom device type this also means that you can be pretty sure it won't work on the mobile device.

I am encouraged and hope to create plugin devices that will work on the mobile platform.  I can't wait to rework my plugins to try it out.  I hear people say that home automation isn't about control so you shouldn't need an interface.  There are things that Vera should do that involve control.  Here's just a few scenarios;  You are sitting in the backyard and simply want to turn on the water features.  You need to run a shock cycle overnight after 20 kids swam in the pool all day.  You want to turn on some lights because you have friends coming over.  That is control and not automation and these all require an effective interface.  It sounds like you are headed down the Bootstrap and maybe even the TypeScript path for your tech stack given your comments below.  Perhaps some theming is in our future as well as real powerful expansion of the UX. 

In the end, I created my plugins because I needed them myself.  I shared them in the hope that someone else might need them thus expanding the Vera herd.  There is strength in numbers so long as you are standing somewhere near the middle.  And I don't use the app store because turned out to be too much of a PITA to get the code published and I felt like it was time I would never get back.

@rstrouse the tiles are a tap away from the detailed control panel of the device. All plugins you guys wrote are going to have such a tile. But for the detailed panel of that device this is where on all platforms, mobile and web, plugins are going to shine. Agnostic for the client apps, they will display the detailed panels lazy, when user wants more control. We are not doing that blindly and we believe that advanced controls should be a tap away. Most users use the client apps as remotes. And what you usually do with a remote are binary actions. The tiles are going to be binary for a VirtualSwitch for example. Pressing on a tile will reverse the state.

So to put it more simpler: The entire UI plugins have now will be moved on a tap away. Access via the little 3 dots on the bottom right corner of the tile or by force touch. Having complicated layout in mobile interface and even on desktop will always make things lag. But why do that? Why not have a clean look and drill down to what's needed with just a tap. This is what we are trying to accomplish.

The approval process and the necessary steps to publish an app will change for the better. It's vital to us from now on to have all the tools in place to make your life easier. Regarding the hard codings that were made along the way, we are going to fix that. It's going to be put on R&D's table.
Dragos Perju,
Product Manager @ MiOS