The Vera Community forums have moved!

Advanced => Security => Topic started by: niharmehta on January 18, 2014, 01:08:19 pm

Title: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 18, 2014, 01:08:19 pm
On 1/13/14 at around 8:00AM PST , while poking around the my Vera, I found the static credentials used by Vera to archive logs to PUBLIC FTP servers. 

I tested these credentials manually, I was surprised to see that I was able to traverse the system and download ALL LOG FILES FROM ANY VERA THAT HAD "ARCHIVE TO SERVER" enabled.  The directory was organized by the Vera ID number. I found my own Vera directory and downloaded many of my logs.  I could easily see the directories of thousands of other Vera. However, I did not download any other users logs.

I verified my findings first with a senior member of the board as well. Using their contacts within MCV, the log server was secured within a few hours. I also opened a case, and left a VM and PM with MCV on that day to understand the scope of the issue, and how users will be notified. 
I have heard nothing useful back at all. Four days later, the only thing they told me was that the settings on the server were open for just a short time while the servers were migrated and a setting was missed.   However, no details were provided on time frame or impact to users.  I have repeatedly asked for notification to these forums to alert others of this mistake with no response.

Checking my log files, I have found security system PIN  and other information that would pretty easily identify the location of my Vera. If you are integrating your Vera with any security devices  I would change these codes immediately and no longer archive to MCV servers.

I delayed making this public until the servers were secured or offline and was hoping that MCV would do the right thing and let people know of this error and provide recommendations to reduce risk .

At this point, it seems the servers are indeed restricted from downloads or directory browsing. 

However, it does bring up another question. Right now, the logs use simple FTP over an unencrypted path to MCV servers. It does not use any sort of encryption nor does it use  the SSH-VPN tunnel that MCV uses for remote access.   The log information with associated potentially security codes are sent in the clear. This needs to be changed ASAP or everyone should consider no longer archiving to MCV servers if they are worried about security .
Using the tunnel would be an easy first step in securing this data while being transported.


Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: guessed on January 18, 2014, 02:01:27 pm
That's the second time they've done that with the log servers.  The last one was about 2yrs ago, I really wish they would learn to lock these down, and implement consistently.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: TC1 on January 18, 2014, 03:06:41 pm
From what I've seen in my short time in these forums and using the product, there's a lot of things they need to learn, and not just about security. If it wasn't for the generous developers and senior members like yourself spending your personal time in these forums, this product would be dead, IMO.

-TC
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: futzle on January 18, 2014, 04:37:36 pm
Good on you, niharmehta. And thanks for posting such a comprehensive summary.

There must be some jurisdiction where MCV operates that has mandatory data breach reporting laws. This is precisely the kind of scenario that those laws are designed for.

For those following along at home, the exact UI5 method to disable log uploading is: Setup > Logs > Archive old logs on server (recommended) [sic].

I wonder if Vera online backups are transferred using a similar mechanism and have a similar weakness?
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: knewmania on January 18, 2014, 07:35:39 pm
Thanks for pointing this out Niharmehta.

I'm no longer archiving old logs to MCV. For what it's worth, I opened a bug reports on MantisBT pointing out the parts of this report related to the unencrypted transfer of log data.

http://bugs.micasaverde.com/view.php?id=3808
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: S-F on January 18, 2014, 09:06:06 pm
WTF? So because I use the gcal plugin I need to change my Google login information? And change all of the PIN codes on my lock? And whatever else I've entrusted to Vera? Probably my Vera Alerts ID's too. So since Monday people could have been reading my calendar (I keep a lot of personal data on there) and receiving my Vera Alerts? And knowing where I live? And who knows what else.

This really sucks.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 18, 2014, 10:03:18 pm
Well, it has been locked down since monday afternoon some time. I do not know when the server migrations occurred when the error was introduced.   MCV did not share that information.  So it may have been open for hours or days.   MCV should let us know how long this was open and which users files were downloaded as per their logs. Maybe (hopefully) nobody had their logs downloaded. Who knows.   Obviously the clear FTP is another issue, but other than the NSA or other government entities , it is unlikely a backbone router has been compromised in the middle.

As for the backup of the configurations to the backup storage servers,  the transport does seem to be secured via SSL as it is using HTTPS from some cursory connection traces I did.   However, who knows how secure the storage is once it is backed up.

What is interesting is that the log data is hosted in a data center located in Northern California, and the backup files in another data center in Southern California.  I would assume that they would potentially be under the data breach disclosure laws in California .

Personally I am disappointed.  I really like the MCV platform and had no ill will towards MCV. I just want them to do the right thing .
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: S-F on January 18, 2014, 11:30:30 pm
  I would assume that they would potentially be under the data breach disclosure laws in California .


I have just had a brief discussion with an attorney and while he mainly works in international law he seemed to think that if Vera users have no agreement with MCV in regards to data security they can do whatever they like so long as there are no proven damages. And in this case it, so far, appears that there have been no damages. So unless someone can prove that their house got broken into or something like that due to this oversight MCV could plain deny the whole thing to everyone if they so desire. I'm of a like mind with you in not wanting to vilify MCV. Frankly I want them to succeed as a company and product. But I do think that they should fess up to their customers on this one. If they want to leave us in the dark regarding software and hardware updates that's one thing. Their products are still working as advertised.... for the most part. But if they want to neglect our personal information we trust them with we should be informed so we can make other arrangements.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: TC1 on January 19, 2014, 12:07:54 am
They could be sued, there's no EULA available to say otherwise, ie we agree to hold them harmless, look at the EULA tab on this page (hint, it's missing):

http://getvera.com/legal/

But the bottom line is that they are a Hong Kong based company and don't even list a U.S. office. So good luck with getting any answers from them.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 19, 2014, 12:45:54 am
 Security issues are found all the time in the software world.  Most of the time when companies at least take the effort to communicate with their end users to the risk, impact, and mediation plan. Customers can  implement the fix and move on. No long term harm done as long as the company takes the view that responsible disclosure is right for both their customers and themselves. 

No point even considering a lawsuit or legal liability. Regardless if it is possible or not.    It is in nobody's best interest to even really consider it. An adversarial relationship with MCV is in nobody's best interest.

Hopefully they can implement a truly secure logging mechanism before the next release. For now, just stop archiving. Change your pin codes, device/plugin passwords,  and hope that MCV decides to comment here.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: S-F on January 19, 2014, 01:04:32 am
@niharmehta,

I'm in no way trying to bring the law down on MCV. That wasn't my point. I was just trying to clarify if they were legally obligated to inform us. I tried to make that clear at the end of my last post but maybe I didn't succeed.

Though I am pissed!

Sorry.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: TC1 on January 19, 2014, 01:05:14 am
Security issues are found all the time in the software world.  Most of the time when companies at least take the effort to communicate with their end users to the risk, impact, and mediation plan. Customers can  implement the fix and move on. No long term harm done as long as the company takes the view that responsible disclosure is right for both their customers and themselves. 

No point even considering a lawsuit or legal liability. Regardless if it is possible or not.    It is in nobody's best interest to even really consider it. An adversarial relationship with MCV is in nobody's best interest.


I work in the software and website world, and in security. I agree with your first paragraph.

I respectfully disagree with your second paragraph. Governmental laws and punishments, and civil lawsuits, are not always about extracting punishment or damages, but to effect change. To make sure it doesn't happen again and to send a message that these types of actions are unacceptable. We have laws to help create the type of world we all want to live in.

As you began to allude to, when companies communicate with their customers and get in front of the issue (ie, the Target breach for example), things work out better for everyone. Fears are quelled and any possible damages that might have occurred are minimized. All MCV has to do is communicate with their users. They don't do that, and when they do, it seems reluctantly.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 19, 2014, 01:43:31 am
@niharmehta,

I'm in no way trying to bring the law down on MCV. That wasn't my point. I was just trying to clarify if they were legally obligated to inform us. I tried to make that clear at the end of my last post but maybe I didn't succeed.

Though I am pissed!

Sorry.

@S-F.  What you say makes sense. In fact, any company really need to consider their potential liability any time they decide to store end user sensitive information. Not just from civil liability but also  potential brand issues.  I know you were not threatening a lawsuit, I just wanted others to reconsider that train of thought .
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Sender on January 19, 2014, 01:55:42 am
I would want to hear from MCV personally if my files were downloaded from a non MCV source.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 19, 2014, 02:05:25 am
Quote
I respectfully disagree with your second paragraph. Governmental laws and punishments, and civil lawsuits, are not always about extracting punishment or damages, but to effect change. To make sure it doesn't happen again and to send a message that these types of actions are unacceptable. We have laws to help create the type of world we all want to live in.

As you began to allude to, when companies communicate with their customers and get in front of the issue (ie, the Target breach for example), things work out better for everyone. Fears are quelled and any possible damages that might have occurred are minimized. All MCV has to do is communicate with their users. They don't do that, and when they do, it seems reluctantly.

In a bigger picture. Completely agree.. Especially when the damages are large, and the "little guy" has minimal leverage. 

In this case, MCV is a small company, their ability to even defend themselves is probably minimal.  More than likely they would just end the Vera line and we are done with our cool little, cheap, no recurring cost, extendible, flexible,  zwave controllers.  Nobody is interested in that. 

They do need to do a LOT of work to rebuild trust. Threatening with lawyers is not the first step in this process.

Now this is public, lets give them a chance to do whats right. Although I could not get them to do it privately, I assume that if this has made enough users angry, that they will respond..

What would make me happy is:
Disclosure of the issue and affected time.
Private notification of any users that had their information downloaded. (if any)
Steps and timeframe to secure logging archiving and transport.
Documentation on how private information is collected,  transported, and protected. (logs, backups, etc).
How they plan to keep this from happening again.

I have heard security is a big focus of the upcoming release. So hopefully they have changed their internal processes and will jump on these issues.





Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Sender on January 19, 2014, 02:07:37 am
Could my login password in any way be disclosed?
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: TC1 on January 19, 2014, 02:11:40 am
Quote

Now this is public, lets give them a chance to do whats right. Although I could not get them to do it privately, I assume that if this has made enough users angry, that they will respond..

What would make me happy is:
Disclosure of the issue and affected time.
Private notification of any users that had their information downloaded. (if any)
Steps and timeframe to secure logging archiving and transport.
Documentation on how private information is collected,  transported, and protected. (logs, backups, etc).
How they plan to keep this from happening again.



Bingo. If they do the steps you outlined, then things are back on track.
I'll give them until Monday, business hours. Though a real company would have responded already.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: S-F on January 19, 2014, 01:50:31 pm
There may be no business hours tomorrow as it's a holiday.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: TC1 on January 19, 2014, 01:59:52 pm
Federal holiday in the U.S., banks and post offices closed, many businesses (including the one I work at) remain open.

But the reality is that security breaches and privacy issues should not be governed by normal business hours.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Da_JoJo on January 21, 2014, 12:42:29 pm
considering one would still need a vera and a root login to that and skill to be able to trace the routes used i think in the end it isn't all that bad as it looks. thing is that mcv should make this log uploading also a safe construction so there is no risk anymore, which presumably they allready did . the ftp servers have logging and they probably know who got in there. might as well look into the fact that breaking in such system is legally protected and they can sue you as well.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 21, 2014, 01:45:31 pm
The issue is not that the Vera itself is open for compromise. ( I don't think system passwords are in the logs). The issue is that the logs themselves were left open  as well as sent in the complete clear.

The system has been changed to only allow PUT requests now which helps initially. However, that is a stop gap for now. 

1) The logs are sent in the clear using FTP on the public side.  It does not use the management tunnel or any other authenticated crypto system to ensure that the system the logs are going to, is actually MCV. 

2) A real hacker would easily mask their identity. So I doubt MCV could ever find WHO for either criminal action or civil lawsuit.


A quick fix should be to use their management tunnel for transport. 
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Da_JoJo on January 22, 2014, 07:10:46 am
got a point there. it's something that needs securing and MCV should look at it.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Colin Burke McClure on January 29, 2014, 12:36:50 pm
Thank you niharmehta. This was brought to my attention this morning. Rest assured we are taking this event very seriously.

We're currently reviewing the specifics of this event and conducting a comprehensive audit.

We will respond with a detailed postmortem analysis in the next 12 hours to provide clarity on what transpired, as well as the actions have been taken to prevent this from recurring.

Thank you for your patience and support.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on January 30, 2014, 01:03:31 pm
Colin,
Thank you for finally jumping in with an official public recognition of the problem that occurred . Good to see your team responding to this.   The audit and prevention plan is exactly what we are looking for in this.   Looking forward to your results and continuing dialog.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: Colin Burke McClure on January 31, 2014, 10:42:12 pm
We wanted to get back to the group with an update, albeit more than a few hours later than we would have preferred. Our apologies. Suffice to say, we are very, very busy at present.

After reviewing the documentation surrounding this event, the root cause and resolution are clear. This was a simple human error, resulting from the misconfiguration of our servers.

As previously disclosed, there were only 2 external connections. Both brief. Further the situation was remedied immediately upon being brought to our attention. Doesn?t excuse the lapse, but on the upside, no Personally Identifiable Information was downloaded by those outside third parties.

That said, we?ve immediately implemented a 3 step check and balance process (which requires sign off of senior staff member) before deployment of any hardware that may contain PII. This will ensure that this type of oversight does not happen again.

Further, our new MMS backend is far better automated, hardened, and scalable, which further reduces/eliminate the risk of these kinds of errors in the future (as well as enabled a host of exciting new features and capabilities).

We are working triple time to get everyone updated and migrated over, but as Aaron has previously posted, this is not a trivial task. We will keep you posted.

//And yes, I did just say that... ;)

Again, on behalf of everyone here at MiOS and Vera, please accept our apologies for this error.

@niharmehta Thanks for the kinds words and welcome. I know you guys have been starving for attention here in the forums and this is long overdue. You will be seeing a lot of of me, as well as the rest of the staff, in coming weeks and months. Our lack of attention here (however inexcusable) has not been due to lack of action. Quite the opposite actually. More to follow. Soon.
Title: Re: MCV Log servers left open- potential compromise of sensitive info.
Post by: niharmehta on February 01, 2014, 08:40:48 pm
Hi Colin,
Great response. Thank you for following up. Appreciate the openness that MCV seems to have lately on the forums. Great engagement between the community and the developers generally means a great product.

Since there were only two logins to the server, that seems to indicate that it was just myself and the other user on the forum I tested with.   That is fantastic news that in this case no PII was downloaded.

One open item is the transfer of the logs via clear FTP.  This can be captured via many MITM attacks.  Does the MMS infrastructure secure this transfer or is this something that will come later?

Thanks!!