Author Topic: Vera Controler Down... For Real!  (Read 4129 times)

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #30 on: May 16, 2018, 07:28:54 pm »
It's interesting Vera say the Fibaro lag fault (another thread and ticket) is Fibaro's bad support of secure mode, as that's what they are saying about Aeotec now.

Has made me think back to my original setup of the Z-UNO - they were adamant that if you are using Vera do NOT enable Z-UNO's secure mode.  I thought that was odd, why do non-secure only for Vera... but I'm starting to think Vera is immensely flawed when it comes to secure inclusions, everything about every problem I'm having is all secure related (both before the system imploded and after)...

If Vera could provide a hack to prevent secure inclusion, I would happily wipe my whole system and set everything up from scratch... as that's clearly going to solve everything and make my entire setup so much better.  Without that, there is no incentive for me to invest such vast amounts of time and effort when down the track they are likely to go this path anyway to solve their Fibaro compatibility issues and thus would have to set everything up from scratch yet again...

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #31 on: May 16, 2018, 08:45:33 pm »
I have done some reading through a paper detailing reverse engineering and exploit efforts of the Z-Wave protocol.  This has further convinced me of my theory.

Devices that are securely paired do indeed only employ encryption for specific command classes.  My Fibaro switches and dimmers for example are only using encryption for turning the light on and off, but the wattage reading is not encrypted.  Likewise the Aeotec MultiSensor's use encryption for the movement sensor, but the temperature and LUX are sent unencrypted.

Also, it has been found that Z-wave devices and controllers will simply ignore any encrypted packets for which they do not have an encryption key.

These facts combined further support my theory that Vera has lost all the encryption keys it negotiated during the pairing of each device.  Vera and all my devices are happily exchanging non-secure information, but all secure-related functions are being totally and utterly ignored.

Presumably configuring a device that has been securely paired would absolutely have to be done using encryption.  So this would again also explain why Vera is going into a total and utter meltdown - all the attempts CC made to reconfigure my non-working devices have caused configurations to be sent, but like turning things on/off these were ignored because of the missing or messed up encryption/keys so Vera is going crazy trying to understand why the configuration is not being acknowledged.... Vera is never ever ever letting go of that, it is permanently days later still wrestling over those unapplied configuration changes.

So overall this is going absolutely nowhere because:
1. CC don't even understand the situation that exists, so have repeatedly concluded things are okay because they toggle devices on and off and think it is working but in fact the entire system is ignoring all such packets in both directions
2. CC have caused all the red bars by playing with device configurations while investigating the above, causing the system to go into further meltdown because those configurations are likewise being ignored but Vera is waiting on them to complete
3. Last night CC decided to simply turn off auto-configuration because of the above configurations problems, and then concluded all was now well

In a nutshell the latest advise is that everything is now okay... but in fact what has happened is highly flammable materials have been swept underneath a highly flammable rug and I've been told it's okay to go light my campfire on top of it now.

The present situation could be more opposite.

Turning off auto-configuration simply hid one of the problems being created as a result of the existing problems that still exist.  Doing further configurations, unpairings, pairings, etc is just going to keep making things worse and worse... the whole thing is on fire, pushing that underneath something is just going to flare up and throwing more crap into it will just make the fire bigger.

The actual root problem is with the encryption keys... everything else is new problems being created as a result of fiddling without understanding this root problem.

So again I'm left with my original thought... I need the ability to force non-secure inclusion, wipe everything, and then re-set everything up from scratch.  Or that the encryption keys somehow be restored back into the system.

Offline rafale77

  • Hero Member
  • *****
  • Posts: 1186
  • Karma: +56/-22
Re: Vera Controler Down... For Real!
« Reply #32 on: May 17, 2018, 12:17:33 am »
Thank you for sharing what you have learned.

It?s interesting that your Fibaro light switches use security keys. None of my light switches do. (Mostly Levitons) and that your multisensors can be paired unsecured (I don?t have any that can)

I don?t quite understand what the disabling of the auto configuration does to help though. I have had issues with encryption keys but not to this disastrous degree.

Wish CC could figure out at least how the security keys got wiped.
openLuup (97 devices, 134 scenes, 20 apps) controlling HomeAss + VeraPlus (134 zwave nodes, 8 Zigbee nodes, 200 devices, 20 scenes , 2 app) Bridged to Homekit and Alexa. VeraPlus ExtRooted!

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #33 on: May 17, 2018, 04:56:28 am »
Same here - but once again I have got home from work and haven't heard from them nor has anything changed what-so-ever... everything is still precisely as it was, either half on/half off, red bars, nothing responding, nothing reflecting actual status, etc.

Only thing that works is my blinds which I wired and programmed myself with a Z-UNO and is non-secure paired.

Offline kwieto

  • Hero Member
  • *****
  • Posts: 557
  • Karma: +26/-12
Re: Vera Controler Down... For Real!
« Reply #34 on: May 17, 2018, 07:29:57 am »
I have impression that last backups, done under 7.0.26 were not really complete.
When I tried to restore online backup it didn't work, so I had to use the one saved offline (I save backup offline doing Z-wave backup before any upgrade of the firmware).
Also change of the Z-wave chip firmware may play some role. In my case CC upgraded this firmware long before 7.0.26b/7.0.26 was released, so there was no risk of losing something due to that switch.

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #35 on: May 17, 2018, 09:42:36 am »
Is it normal that tickets will only get followed up once every 24 hours?  Sincere question as this is my first ticket and I'm quite alarmed at only getting help once every 24 hours.

Like yesterday, CC haven't been on or in touch all of today and tonight... so wondering if there's any chance they might jump on again overnight.

I can't believe I'm going into day 4 now of Vera being totally and utterly self-imploded...

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #36 on: May 17, 2018, 09:45:54 am »
Okay credit where credit is due, ping tests and stress tests have just suddenly commenced on my Vera... someone must be on her.

So it must be that tickets get reviewed every 24 hours.

Offline Sorin

  • Administrator
  • Hero Member
  • *****
  • Posts: 937
  • Karma: +80/-8
Re: Vera Controler Down... For Real!
« Reply #37 on: May 17, 2018, 10:03:14 am »
Hey Tillsy, indeed. Although CC is here 24/7 , because it's more of a corner case issue where one agent is familiar with your setup, the case tends to stick to one agent.

The guys are on your unit as we speak.
Sorin M. ▾ Community Manager
Vera Control, Ltd. ▾ Smarter Home Control  ▾ support@getvera.com ▾www.getvera.com ▾ +1 (866) 966-2272

HOURS OF OPERATION for Customer Care Department - 24/7

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #38 on: May 17, 2018, 11:09:01 am »
OMFG I am going to loose my sh#t in a minute...

So because the ping tests work, because the tech can (so they think) turn lights on and off, apparently everything is fine.

If I am having any problems it is because there must be a baby monitor causing interference.  Seriously...

What do I need to do?!?!?!?!?  Record a video showing that when they are clicking on those lights that NOTHING HAPPENS.  Record a video showing that the light I am sitting under right now is ON, that Vera shows is burning 27W of power, but Vera also claims is OFF...

NOTHING WORKS..... Vera thinks it is doing stuff, but the encrypted comms are being ignored by the devices.  Likewise Vera is ignoring the encrypted comms from the devices.  The keys are either screwed or missing.

OMG I am so angry right now, 4 days and apparently this totally and utterly not working Vera is NORMAL.

I'm going to bed or I'm going to end up grabbing that thing and smashing the bejesus out of it... this is beyond insane :(

Offline jeubanks

  • Full Member
  • ***
  • Posts: 195
  • Karma: +6/-2
Re: Vera Controler Down... For Real!
« Reply #39 on: May 17, 2018, 12:48:33 pm »
OMFG I am going to loose my sh#t in a minute...

So because the ping tests work, because the tech can (so they think) turn lights on and off, apparently everything is fine.

If I am having any problems it is because there must be a baby monitor causing interference.  Seriously...

What do I need to do?!?!?!?!?  Record a video showing that when they are clicking on those lights that NOTHING HAPPENS.  Record a video showing that the light I am sitting under right now is ON, that Vera shows is burning 27W of power, but Vera also claims is OFF...

NOTHING WORKS..... Vera thinks it is doing stuff, but the encrypted comms are being ignored by the devices.  Likewise Vera is ignoring the encrypted comms from the devices.  The keys are either screwed or missing.

OMG I am so angry right now, 4 days and apparently this totally and utterly not working Vera is NORMAL.

I'm going to bed or I'm going to end up grabbing that thing and smashing the bejesus out of it... this is beyond insane :(

Hey Tilsey... I've been watching this and ohhh the memories...

Do yourself a favor and take a look at Homeseer while it's on sale.

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #40 on: May 18, 2018, 07:02:21 am »
So this morning I awoke, walked down the passage, and noticed the lights automatically turned on at correct brightness... that got my attention as they have been stuck at a really low level since my Vera imploded, with no way to change them.

As I walked down I noticed a dim light coming from our kids bathroom vanity - I normally have that come on automatically overnight until sunrise at minimum brightness as a night light for my girls, however it has been stuck at full brightness (and then for some reason half brightness) with no automation all week.

So I went into the kitchen and looked at the iPad on the wall - it normally shows the status of all my Siri integration with Vera, but has been blank all week because Vera has had absolutely no clue what-so-ever of what it and its devices were and weren't doing.  It was now showing that the passage and vanity were lit, and then the passage turned off and the iPad updated its status accordingly.

To that end I opened up my Mac and immediately saw Vera had no red bars anymore and was indeed showing correct status.  I then told Vera to turn on the kitchen lights and, for the first time all week, she not only claimed they were on but they actually did turn on.  From a distance I could see the iPad was lighting up their status as well.

Then I noticed some other lights in the house turn on, turn off, etc - looking at Vera I could see those commands were being manually sent.  So a CC tech must have still been signed on.

Have come home from work tonight (sorry for no update all day, had a really crap day ending a really crap week) and everything is still working perfectly now - lights are working, dim levels are correct, even the TV has power!

A few things of note...
Logging into my Siri/HomeKit interface I can see Vera has some hidden baggage/damage, as there is a whole heap of stuffed devices behind the scenes that exist now but Vera is supressing in its GUI.  For now I just don't care, everything is working!

My wife commented that she still has to flick light switches multiple times now.  It's annoying her, actually I know it's really really annoying her, but I think she's just happy that things are working again now.

I feel a bit bad that I let rip last night, not just here but back to the tech who told me everything was fine, so I might have to review what I said... but at the same time my limits and patience were hit and, ultimately, I obviously then re-re-re-re-explained the problems in a way that must have finally clicked with someone for a silent super-hero to have fixed it all up.  Thank you :)

The other problem I had during all this was that I couldn't get Vera into pairing mode anymore (except one occasion, but when I tried to do the pair it failed).  So I have just tried to pair the SmartSwitch I deleted from the system earlier in the week - it successfully paired.................. ooooooooooohhhhhhhhhhh noooooooo, now things are starting to go whacko again.... multiple devices I didn't even touch are saying they are doing configuration, a light that was turned on won't turn off (has been sitting there for 10 minutes saying Sending the Z-Wave command after 0 retries..., there is a blue bar stuck saying Configuring Z-Wave devices, nope I can't even turn anything on anymore it's stuck sending Z-wave command, oh god now I have a red bar appearing.... good god all I did was pair, name, that's it... okay now that new device, which was sitting there happy with no statuses or anything, has now lit up red saying failed setting special association.

Okay after waiting 10 minutes it has settled down and things are working again, albeit the smartswitch still has a red bar.

So this confirms what my Siri/HomeKit is seeing - things are working but there's clearly some messed up stuff behind the scenes that still causes some havoc to break lose whenever it needs to start talking to devices.

I've only had a few hours sleep each night this week so I'm really just not up for this right now, so might just catch up this week's episode of FTWD and hit the sack.

And as I got to post this, Vera is trying to re-do the special association with the smartswitch... I think it's going to keep repeatedly trying to do that, clearly still has some issues it is struggling with.... yep its red again.  Ugh, logging off that'll do.............

Offline Don Phillips

  • Hero Member
  • *****
  • Posts: 1289
  • Karma: +33/-32
Re: Vera Controler Down... For Real!
« Reply #41 on: May 18, 2018, 07:14:41 pm »
The other problem I had during all this was that I couldn't get Vera into pairing mode anymore (except one occasion, but when I tried to do the pair it failed).  So I have just tried to pair the SmartSwitch I deleted from the system earlier in the week - it successfully paired.................. ooooooooooohhhhhhhhhhh noooooooo, now things are starting to go whacko again.... multiple devices I didn't even touch are saying they are doing configuration, a light that was turned on won't turn off (has been sitting there for 10 minutes saying Sending the Z-Wave command after 0 retries..., there is a blue bar stuck saying Configuring Z-Wave devices, nope I can't even turn anything on anymore it's stuck sending Z-wave command, oh god now I have a red bar appearing.... good god all I did was pair, name, that's it... okay now that new device, which was sitting there happy with no statuses or anything, has now lit up red saying failed setting special association.

Okay after waiting 10 minutes it has settled down and things are working again, albeit the smartswitch still has a red bar.

About a year and half ago, I had a rouge z-wave device take down my whole mesh. After weeks of monitoring, I was able to notice the CT30 thermostat was the issue. It would drop from the mesh, and within an hour or so, my other devices started to go offline. I would re-pair it, and it fixed the mesh for 5 days. Exactly 5 days, and then the thermostat would drop again, and the mesh would be gone again within an hour. Very predictable and annoying.

During this time, my mother-in-law passed away and we were out of the country for a week. I had re-paired the thermostat just before leaving for the airport and z-wave was down before we returned.  Very frustrating.

Bought a new z-wave module for the thermostat, but it acted the same. I tried a straight switch, leaving both in, pairing both, and the thermostat dropped after 5 days and z-wave down within an hour.

After a few months of dinking around, I replaced the thermostat and my Vera has been humming for over a year.
Vera 3, 1.7.1030, CT101 t-stat, Everspring motion detector, GE/Jasco switch, Leviton outlet, AeonLabs sensor, NuTone garage door, Blue Iris, Sricam SP011, iPhone locator, APCUPSD, VeraMate, VeraAlerts, PLEG, House Modes, Countdown Timer, DVR, Virtual/Multi Switch, Weatherunderground, LB60Z-1 bulb

Offline Tillsy

  • Full Member
  • ***
  • Posts: 203
  • Karma: +9/-4
Re: Vera Controler Down... For Real!
« Reply #42 on: May 18, 2018, 07:37:10 pm »
CC got back to me last night and confirmed that when they did the restore only the user data (and something they call dongle.dump?) was brought back - NO security keys.  Once they properly understood my dilemma they then restored the security keys, and hence everything is now working.

So it is reassuring to know that it all makes sense.

Albeit the original implosion is still a mystery.

They've offered to do another full restore - I might take them up on this given all the original messing about after the original restore has obviously created a lot of broken devices/stuff behind the scenes that is causing problems whenever I try pairing something.

Offline HSD99

  • Full Member
  • ***
  • Posts: 245
  • Karma: +7/-0
Re: Vera Controler Down... For Real!
« Reply #43 on: May 18, 2018, 08:59:53 pm »
Glad you're back! 

The dongle dump is the Z-Wave network which is stored on the Z-Wave co-processor (called the dongle from early Veras where it was an external USB Z-Wave stick). Another clean full restore with user data, Z-Wave and security keys sounds like a good idea to clean up any loose ends. I would have assumed that security keys would be part of the backup, but it sounds like that might not be the case. Go to know.

Don makes an excellent point---a malfunctioning Z-Wave device can wreak havoc on your mesh---I've seen this myself, and it can be tough to track down.

The only question left: what happened to cause this?

Offline markd

  • Sr. Newbie
  • *
  • Posts: 47
  • Karma: +1/-1
Re: Vera Controler Down... For Real!
« Reply #44 on: May 19, 2018, 06:22:04 am »
I've recently installed an Aeotec nano dimmer and after 5 days it reports 'can't find device' and then other devices drop off so definitely something that still needs to be looked into.

Vera support logged into my system and have 'fixed' it for now.  Fingers crossed it will last.