Author Topic: Memory Vera plus  (Read 3077 times)

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Memory Vera plus
« on: March 10, 2018, 04:33:15 pm »
Greetings,

LAst couple of weeks I had an issue with virtual switches which didn't wanna toggle anymore. After tech support looked at my system they said I was running out of memory and they freed up some memory.

Since I also installed Datamine2, I log the memory now as well since last week.

But how bad is this for a vera plus?

-every day around 0300 I do a scheduled reboot ( I see on the graphs the "memory free" jumps up at that time).
-but during the day the memory used goes up.

Do I need to get a second vera plus and balance the load?, do I have to many devices/scenes/PLEG?

Thanks,
Cor

Offline tomtcom

  • Hero Member
  • *****
  • Posts: 987
  • Karma: +23/-31
Re: Memory Vera plus
« Reply #1 on: March 10, 2018, 05:00:33 pm »
Hey Cor,

I think your plugins and virtual switches are the largest culprits. I have watched the virtual plugin threads over a few years and their memory use versus a standalone straight z-wave device is vastly different.

There have been recent topics on ways to flush memory without rebooting but you might have to do some experiments and I'm not sure of the side effects. I would suggest determining a few virtual switches and plugins you can live without and check your stats again.

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Re: Memory Vera plus
« Reply #2 on: March 11, 2018, 04:12:08 am »
@ Tomtcom:

I have read the rebooting thread as well , that hinted me on using datamine to log the memory usage, so I have a bettert overview.

The graphs just now shows the same as yesterday , after the scheduled reboot ( at 3:15) memory free goes up quite a bit and memory used goes down.

I do not want to get rid of devices I have , als not scenes/PLEG , virtual switches etc. I even want to expand a bit more, my house is still not automated enough  ;D

So the question remains , is my vera plus "full" , do I need to get a second one if I want expand , or is it not that bad "yet".
Or do I even have to get a second one and balance the load a bit of what I have now?

Thanks,
Cor

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Memory Vera plus
« Reply #3 on: March 12, 2018, 06:22:46 am »
Fist, check if your Datamine2 plugin really saves the data to the USB stick or to the internal memory (you can tell it going to the Datamine2 settings ("Datamine2" section) and checking if amount of total memory matches the size of USB stick or Vera internal memory.
Also make sure that you have local saving of Vera logs disabled. This doesn't go well with Datamine2, at least if you save data from both sources to the same stick. They are how-to's about saving logs from Vera and informations from Datamine2 to separate sticks or to separate partitions from the same stick, but I didn't try it. Most of the time I have only remote access to my Vera, so I can't access local Vera logs anyway.

Then you can check your plugins, as they are most probable source of data consumption (at least that's what Customer Care told me).

In normal operation, Vera should clear the cache by itself on regular basis (every 1-2h), and the graph should look like in attached picture.
I don't know how many devices/plugins you have so I can't judge if your Vera is overloaded or you have some misbehaving device/plugin. I have about 40 nodes (120 "devices") and 6 plugins and my system if fine, except occasional memory drains (i.e. once per month).

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Re: Memory Vera plus
« Reply #4 on: March 12, 2018, 03:47:14 pm »
@ Kwieto,  Thanks for your reply.

I have a 8GB USB stick in the vera plus  , and attached a screenshot , it looks like Datamine sees the 8GB stick.
BUT , I also attached a second screenshot where you tell me to check "Also make sure that you have local saving of Vera logs disabled" , but when I check the logs section ( see attached screenshot) it says :"USB statusDisabled Not mounted "   What is going on?

And which one do I uncheck?

*edit* and this is what is on my vera plus:
-16 plugins
-64 physical devices, from about  half of them have child devices. (fibaro double relais and universal sensors with temp probes)
about
-20 virtual switches
-about a totall of  160 devices
-44 scenes
-6x PLEG: shitloads of conditions between 10 and 30 per PLEG instance.

There is one plugin I don't use ( Sercomm IP Camera) but I am unable to remove that one somehow.

Thanks,
Cor

« Last Edit: March 12, 2018, 03:55:28 pm by Cor »

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Memory Vera plus
« Reply #5 on: March 12, 2018, 06:09:36 pm »
but when I check the logs section ( see attached screenshot) it says :"USB statusDisabled Not mounted "   What is going on?

That is OK.
If you have local logging enabled (checked "store logs on USB device" option) then Vera erases USB stick to its format (it is no longer readable by computer).
If you don't have it enabled, then you'll see "Disabled not mounted" as a USB status. Datamine can access the stick anyway.

Seeing list of your devices, I would first try to decrease number of plugins (you have a lot of them) and Virtual Switches (maybe replace at least part of them with multiswitch, or variables within scenes? Do you really need that many? As far as I remember each virtual switch consumes about 2MB of space, 20 of them is about 40MB which is almost 1/5 of your total Vera Memory)
I would also consider reducing number of PLEG instances: I've read some comments that overuse of PLEG can affect your system speed, so after my first controller got bricked I decided to avoid installing it untill it will be really necessary (same logic for most of other plugins I've had before).
For now I use Ping sensor, Netatmo weatherstation, Datamine, System Monitor, 2 virtual switches (one can be probably replaced by global variable) AltUi (mostly to have access to datamine remotely), HouseModes and VirtualPanel (installed for testing, but probably will remove it as I don't have time to dig in it) plugins.
Second controller has even less of them (Netatmo, AltUI, System Monitor, Datamine and Virtual Rain sensor)
« Last Edit: March 13, 2018, 03:52:19 am by kwieto »

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Re: Memory Vera plus
« Reply #6 on: March 13, 2018, 04:54:24 am »
@ Kwieto:  I have been looking at my system. I have again tried to remove the "sercom Ipcamera plugin" but it comes back every time. Also the "alexa helper" plugin I will try to remove.

For  virtual switches , there are 3 installed which I can remove And PLEG instances ( I have a totalll of 6 PLEG) 1 I can remove , and there is one Test PLEG. Which I can remove if really neccesary.

If I remember well , the multiswitch have 6 or 8 switches , that is much better as 6-8 virtual swicthes?     Because , that I can easily change ( If Imperihome can handle them  :-\)

To be sure: With a PLEG instance is meant how many time is is installed as a device? , not the amount of conditions in the PLEG?

Quote
Also make sure that you have local saving of Vera logs disabled.
Do I need to uncheck here anything (see attachment) ? , because I see there also "Lock Log Levels (not recommended)" is checked in my case, all options are checked. :o

It looks like I will havr to install a second vera  , although I still have a vera light unused here......  ???..... choices choces choices...

*edit*: after ca 20 minutes a luup restart and I see the "Sercomm IP Camera" has installed itself again  ???

Many thanks for you help,
Cor
« Last Edit: March 13, 2018, 05:24:03 am by Cor »

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Memory Vera plus
« Reply #7 on: March 13, 2018, 07:52:55 am »
@ Kwieto:  I have been looking at my system. I have again tried to remove the "sercom Ipcamera plugin" but it comes back every time. Also the "alexa helper" plugin I will try to remove.

This is known issue, I also have these plugins and they can't be removed. Didn't count them as I consider them as a "bug" I have no use of them

For  virtual switches , there are 3 installed which I can remove

Maybe you can replace some of them with variables stored within a scene, or global ones? I can't give you really advice since I don't know what you use them for.

If I remember well , the multiswitch have 6 or 8 switches , that is much better as 6-8 virtual swicthes?

As far as I understand there is some "minimum" amount of memory which have to be used for each device, so 1 multiswitch (capable keeping 6-8 "switches") use similar amount of memory as 1 virtual switch.
Changing 8 virtual switches to 1 multiswitch should free amount memory equal to removing about 7 virtual switches (let say 7 x 2MB = 14MB).


To be sure: With a PLEG instance is meant how many time is is installed as a device? , not the amount of conditions in the PLEG?

I'm not an PLEG expert, but as far as I understood from reading here and there, both are important.
As I mentioned above, I took general approach to keep my system lean (which includes: as little plugins as possible), so when I weighted profits (easy managing "and" conditions, continuing schedules in case of luup reloads) and loses (at least two plugins needed to be installed: PLEG and PLC) I decided that for those couple of scenes where it would be useful I can live with writing in lua. If necessary I split scene in two (like: if something has to be started at X and finished at Y I have one scene for "ON at X" and second "OFF at Y" instead of making a schedule for a duration between X and Y).
I also manage many things via Home Modes, which brilliantly cover most of my needs for virtual switches (though I have couple of scenes which use home mode AND state of the virtual switch)


Do I need to uncheck here anything (see attachment) ?

No. you can disable lock log levels and verbose logging as by default these options are disabled, but I left these options checked after support requested me to do this when they were working on some issue couple of moths ago.

It looks like I will havr to install a second vera  , although I still have a vera light unused here......  ???..... choices choces choices...

Or you may reconsider using of some tools you have. I wonder for example why do you need as many virtual switches, as in my case I have only 2 of them, and one could be in fact changed into global variable. I use it to run a geofence based scene, to track the direction I am moving (so the scene is triggered only if I move forth, but not back the way). Since the switch changes state for very short period (like: 5-10 minutes) I can set a global variable and change its value for this period, as the risk of reboot or luup reload in such short time is really low.
The other is used to track more permanent condition (if I have guests at home).

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Re: Memory Vera plus
« Reply #8 on: March 13, 2018, 04:07:13 pm »
@ Kwieto , Thanks for your big reply.

I need most virtual switches , I use them for example for:
- Every Family member has a virtual switch to switch the security system on or off , switching it on or of sends a message to everyone , saying that person deactived or actived the system.
-switching the heating system on and off , this is for PLEG  , which activates lots of other sub systems.
- other systems which I switch on and off via PLEG , via virtual switches.

I will look if I can change some to the multiswitch.

Lua is unfortunately not for me , PLEG is very difficult as well , but at least there is some logic in there for me.

No clue how to work with global viariables....... I even don't know what they are  ???

At least I know nothing is wrong with my system ( probably)  , but most likely overloaded.

The following days I will take a look into multi switches and remove the virtual switches I don't use. I can also delete some scenes.

The other think which is a bit wierd in my system is that when there is a LUA restart , the DATAmine2 takes ages to load: "dataMine 2 graphing [307] : Running Lua Startup " something like 15-20 minutes mosty of the time.  During that time I notice that virtual switches can not be toggled anymore.


Also I will consider adding a third controller in my system.

Thanks,
Cor

« Last Edit: March 13, 2018, 04:10:00 pm by Cor »

Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Memory Vera plus
« Reply #9 on: March 13, 2018, 06:28:40 pm »
- Every Family member has a virtual switch to switch the security system on or off , switching it on or of sends a message to everyone , saying that person deactived or actived the system.

You can use geofencing for that, but you will loose ability to be informed who entered home.

-switching the heating system on and off , this is for PLEG  , which activates lots of other sub systems.

Isn't the heating system turned on/off by some z-wave switch? Then why using separate virtual switch instead of just monitoring state of that z-wave switch?
I steer my heating with house modes and it works pretty well. If you have thermostat devices, you can set certain state (like Heat/Cool/Off/Auto) and temperature to be set when certain house mode is activated. If your heaters work as simple on/off devices, you can make specific scenes to be run according to the certain modes. For more fine-tuning scenes can be used as well (like: set certain temperatures for specific time periods).
I have two locations: one has central (hot water) heating so I have z-wave valves installed on each radiator. I have two scenes, one set lower temperatures for each valve at certain time in the evening, second sets higher temperatures in the morning. The rest is steered by house modes (i.e. lower temperatures when I'm away or on vacations).
At second place I have electrical heaters which work basically as on/off devices. Then I have temp sensor in each room and two scenes for each heater: one turns it on if temperature drops below certain level (different for different times of day), second turns it off if temperature increases above defined level. Scenes work in house mode home or night. For "away" and "vacation" modes I have similar duo of the scenes respectively, but with different temperatures.
Now I experiment with Qubino flush ON/OFF thermostat, which works as ON/OFF switch, but the only thing you need is to set desired temperature (works similar as radiator valve). It can be steered via house modes as well (i.e. "keep temp 15C if I'm away, turn off when I'm on vacation, etc)

- other systems which I switch on and off via PLEG , via virtual switches.

Similar question as above, can't you use states of physical z-wave devices for that?

Lua is unfortunately not for me , PLEG is very difficult as well , but at least there is some logic in there for me.

I found Lua less difficult than PLEG as it uses typical logical operators and you have everything (conditions, triggers, logic, actions) in one place while PLEG organizes them in separate sections which makes it more difficult for me to orient which is for what.


No clue how to work with global viariables....... I even don't know what they are  ???

You can define variable as local (to be used only within certain scene) or global (accessible from other scenes). For example you may define variable "X" to store status of a switch (or any other data you define), to be used as a condition (if X == n then...) or to be replicated to other devices (i.e. "check the stat of switch 1 and then set states of switch 2, 3, 4 to match it)
If variable is defined as global one, you can use it in various scenes, just referring to its name. If it is local, works only within the scene where it is defined.


At least I know nothing is wrong with my system ( probably)  , but most likely overloaded.

I don't think overload is the only issue. If you've just had too many things, you would expect your amount of free memory to be very low and/or slow speed of your system. But amount of free memory should be more or less stable at certain level, while in your case it decreases continuously. This suggests that something in your system is not right and too many things is not the only trouble.

The other think which is a bit wierd in my system is that when there is a LUA restart , the DATAmine2 takes ages to load: "dataMine 2 graphing [307] : Running Lua Startup " something like 15-20 minutes mosty of the time.  During that time I notice that virtual switches can not be toggled anymore.

Then maybe something is wrong with datamine plugin settings, or the USB stick to which datamine saves data. In my case if it takes seconds for such message to dissapear, that's all. And you have only 28 channels monitored, while I have 110.
« Last Edit: March 13, 2018, 06:31:46 pm by kwieto »

Offline zedrally

  • Hero Member
  • *****
  • Posts: 1122
  • Karma: +11/-5
  • Black Cat Control Systems
Re: Memory Vera plus
« Reply #10 on: March 13, 2018, 07:07:35 pm »

snip

Also I will consider adding a third controller in my system.

Thanks,
Cor


I've never heard of anyone adding a third controller as another Slave. Can it be done?
Living in the Land of Oz, give me a vegemite sandwich. Home Seer, Vera Lite & Edge, Popp, Black Cat Smart Hub & Vera G, Black Cat Lite 1 & 2's a Black Cat Dimmer or 2, Fantem Tec and then some  Black Cat Cat's Eye PIR's & Door-Window Sensors, RFXComm, Broadlink RMPro & Mini plus a Z-UNO or 2.

Offline Tillsy

  • Full Member
  • ***
  • Posts: 207
  • Karma: +12/-4
Re: Memory Vera plus
« Reply #11 on: March 13, 2018, 11:04:59 pm »
Mmmmmm, a Vera threesome... :-*

Offline Cor

  • Hero Member
  • *****
  • Posts: 1205
  • Karma: +7/-4
Re: Memory Vera plus
« Reply #12 on: March 18, 2018, 03:57:13 am »
Stil trying and looking what I can delete ( or change) and it isn't much.

Attached a screenshot from the memory the past 24 hours.
I see there totall memory , which is the top red Line.
memory available, the green line.
Memory used: Blue line.

Abouth half in the day , the memory used goes above the memory available, but still under "totall memory"

How do I interpret this? 
Around 14:00 it uses more memory than there is available?? is this correct? because that would be very very bad I guess. Although it seems my system is very stable . Almost no luup restarts ( maybe a bit on the slow side.)

What is your opinion?

Thanks,
Cor


Offline kwieto

  • Hero Member
  • *****
  • Posts: 570
  • Karma: +27/-14
Re: Memory Vera plus
« Reply #13 on: March 18, 2018, 05:28:08 pm »
Around 14:00 it uses more memory than there is available?? is this correct?

No. Memory available partially overlaps with memory used parameters
Calculating roughly, memory free + memory used = memory total.
Rougly, because your graph miss at least two additional parameters: cached memory and memory buffers.
If you add these, then you'll see that memory available contains memory cached so it is partially overlapping with memory used (the logic behind is that memory available = "freeable" and cached memory is able to be freed by clearing process).
For tracking how your system is loaded and how much memory left, free memory is a better parameter to track.

Offline niharmehta

  • Sr. Member
  • ****
  • Posts: 352
  • Karma: +15/-0
Re: Memory Vera plus
« Reply #14 on: March 18, 2018, 10:01:01 pm »
Just a thought, have you tried putting Vera on its own subnet/vlan off you router?

A couple of years ago when I still had my VeraLite,  I was having constant memory exhaustion and subsequent LUUP reloads.  I tried for months to track down what is draining the memory over time such as plugins, logging, etc.    On a whim, I just decided to put the vera on its own isolated subnet away from the DLNA and other broadcasting going on with the dozens of other things on my network.  This fixed my issue and even since migrating to the Plus, I have kept it on its own vlan with just another device and my unplanned LUUP reloads only happen every few weeks. 

I too have dozens of zwave, IP based, and cloud connected  devices integrated , with about 15 plugins, and a decent amount of PLEG logic including timers.
2x VeraLite; 2xTrane Tstats; 45 x Switches/Dimmers/Appliance Modules; 4x Everspring Water Sensors; DSC Integration; 2 x Zwave Door Locks; 1x Ted5K; 1x Rainforest Eagle; Onkyo AVR; 6x Squeezebox;