The Vera Community forums have moved!

Advanced => Plugins & Plugin Development => Programming => dataMine Plugin => Topic started by: d55m14 on February 15, 2015, 05:15:41 am

Title: DataYours on Raspberry Pi
Post by: d55m14 on February 15, 2015, 05:15:41 am
Hi akbooer,
 
after some tests it seems to work fine with the data stored locally on VeraEdge. I notice that the variable "ChildrenSameRoom" is equal to 1 while I've active Cache, Dash, Watcher and Graph.

Then I've tried to use a dedicated USB stick connected to VeraEdge with a USB HUB . To USB HUB is connected a USB stick for VeraEdge log and the Serial Current Cost. I got some problems when VeraEdge restart because the dev name changes. Have you any experience with this configuration ?

Alternative to external storage seems CIFS but not working yet (Correct ?).

I've a spare Raspberry pi : do you think is possibile to use it for Datayours with the Watcher deamon running on Veraedge and store the data on a storage connected to Raspberry ?

tnks

donato

--------------------

Edit:  The original post above was torn from another thread as it was the first suggestion that DataYours could be run on hardware other than Vera.  Full credit to @d55m14 for that.  Two things have happened since then:

If you want to run DataYours, then this thread is the one: DataYours - App Store Release (http://forum.micasaverde.com/index.php/topic,30808.msg220683.html#msg220683) - the first post pointed to has a link to the latest (or thereabouts).

If you want to try running any plugin on non-Vera hardware, then openLuup is for you and this is the place: openLuup - running unmodified plugins on any machine (http://forum.micasaverde.com/index.php/topic,32315.msg235161.html#msg235161)

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on February 15, 2015, 08:06:50 am
after some tests it seems to work fine with the data stored locally on VeraEdge.
Excellent.

Quote
I notice that the variable "ChildrenSameRoom" is equal to 1 while I've active Cache, Dash, Watcher and Graph.
That flag is set to indicate that if you put the master DataYours device into a room, then the children will follow it.  They can't be placed in separate room (I couldn't see any reason why you would want to do this.)

Quote
Then I've tried to use a dedicated USB stick connected to VeraEdge with a USB HUB . To USB HUB is connected a USB stick for VeraEdge log and the Serial Current Cost. I got some problems when VeraEdge restart because the dev name changes. Have you any experience with this configuration ?
No, none at all.  Sorry, can't advise on that.  DataYours is specifically designed NOT to mount any devices or do anything special with storage other than access it through a local path.  I saw the pain that this caused the erstwhile dataMine plugin and resolved not to go down that path.  This definitely falls into the SEP category ("somebody else's problem".)

Quote
Alternative to external storage seems CIFS but not working yet (Correct ?).
Indeed, this doesn't seem to install correctly on VeraEdge/UI7.  Again, SEP.

Quote
I've a spare Raspberry pi : do you think is possibile to use it for Datayours with the Watcher deamon running on Veraedge and store the data on a storage connected to Raspberry ?

This would be fun to do and is certainly possible.  There's two obvious approaches:
The whole point of basing DataYours on Carbon/Graphite was that it is an open system, not rooted in any particular architecture, so that you could do exactly this sort of thing.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on February 15, 2015, 09:23:50 am
Quote
Quote
I've a spare Raspberry pi : do you think is possibile to use it for Datayours with the Watcher deamon running on Veraedge and store the data on a storage connected to Raspberry ?

This would be fun to do and is certainly possible.  There's two obvious approaches:
  • Install Lua on the Rpi and run the DataCache daemon on it
  • Install a complete Graphite/Carbon system on the Rpi: http://graphite.readthedocs.org/en/latest/install.html (http://graphite.readthedocs.org/en/latest/install.html)
The whole point of basing DataYours on Carbon/Graphite was that it is an open system, not rooted in any particular architecture, so that you could do exactly this sort of thing.

Hi akbooer,

tnks for your reply .

For running your app on Rpi have you any documentation/instructions ? On Rpi do I have to run Graph and Dash daemon too ?

tnks again

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on February 15, 2015, 11:11:23 am
For running your app on Rpi have you any documentation/instructions ? On Rpi do I have to run Graph and Dash daemon too ?

I have no instructions, because I have never done it.  I have every intention of doing this, though, on my BeagleBone Black board.  Assuming that you can install Lua, and the LuaSocket library, then there's only one routine (the Luup HTTP callback handler) which has to be substituted, and I already have the code for that.

I would have thought that you would want to abandon Graph and Dash outside of the Vera environment.  However, Graph is just a subset of the Graphite Webapp and would run just as well as Cache (with the same change.)  The dashboard interface, though, is much more intimately connected with Luup and Vera, so is not so easily ported.  There are other third-party dashboards and renderers that may, perhaps, also be used.

Without the dashboard interface you could still issue HTTP requests to the renderer and get back CSV, JSON, and also SVG graphics (if the Rpi has access to the internet.)
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on February 15, 2015, 11:32:08 am
I'd like basically to store data outside VeraEdge possibly without installing extra module (CIFS for example) on it.

I like the idea of having VeraEdge managed only with the UI7 and plugins so that to have less problem during upgrade . The Dash and Graph module on the VeraEdge will be fine for me.

tnks

donato

Title: Re: DataYours on Raspberry Pi
Post by: airedale on February 16, 2015, 08:57:26 pm
Sounds like a great idea - I have a Raspberry PI, so I could imagine taking advantage of this too.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on February 17, 2015, 03:22:44 am
Sounds like a great idea - I have a Raspberry PI, so I could imagine taking advantage of this too.

Hi airedale,

I see that in the image of debian for  Raspberry there is lua 5.1 already installed.
Do you know how to install the luasocket  library indicated by akbooer ?


Tnks


Donato

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on February 17, 2015, 04:18:08 am
I haven't tried it, but here http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2 (http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2) looks like a good start.
Title: Re: DataYours on Raspberry Pi
Post by: airedale on February 17, 2015, 11:33:21 am
Sounds like a great idea - I have a Raspberry PI, so I could imagine taking advantage of this too.

Hi airedale,

I see that in the image of debian for  Raspberry there is lua 5.1 already installed.
Do you know how to install the luasocket  library indicated by akbooer ?


Tnks


Donato

No I haven't.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on February 17, 2015, 11:55:25 am
I haven't tried it, but here http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2 (http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2) looks like a good start.

Hi akbooer,

I've installed lua 5.1 and luasocket on Raspberry. Can I try to install DataCache on it and storing the data on a storage connected to raspberry ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 03, 2015, 06:11:27 pm
I haven't tried it, but here http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2 (http://files.luaforge.net/releases/luasocket/luasocket/luasocket-2.0.2) looks like a good start.

Hi akbooer,

I've installed lua 5.1 and luasocket on Raspberry. Can I try to install DataCache on it and storing the data on a storage connected to raspberry ?

tnks

donato

So sorry, I must have missed this post.  I'll set up a new thread for Rpi, and we can continue the discussion there!
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 04, 2015, 03:39:46 am
I'll set up a new thread for Rpi, and we can continue the discussion there!

So here we are.  What stage are you @d55m14?  If I understand correctly:
There is just one Luup call which needs to be replaced (sets up the HTTP callback handler) so I'll package a replacement and post it here - I'd rather not change the  DataCache module at all.  What this means is that a small module with just a few Luup call emulations should enable the basic Watcher / Cache / Graph combination to run.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 07, 2015, 06:33:49 am
OK, I think I've contrived to make DataCache run on a non-Vera machine without any modification of the existing files:
...and you should be up and running.

I've not tested this on a RPi, because I haven't got one, so there may be a bit of debugging needed.  I have run it successfully in a test harness on my development machine (iMac.)  The attached file simply emulates the relevant parts of Luup, although I've not yet included the HTTP callback handler, so the DataYours configuration page will not yet see the cache on the RPi.  But it should receive and store data, creating the relevant files and updating them as time goes by.

Would be grateful of any budding beta testers out there.  Good luck.
Title: Re: DataYours on Raspberry Pi (or BeagleBone Black)
Post by: akbooer on March 08, 2015, 10:44:16 am
I've not tested this on a RPi, because I haven't got one...

...but I have now tested it on my BeagleBone Black (BBB), and it runs well, taking less than 0.2% of the CPU time.  I have it set up receiving data from my Edge and it has created the relevant Whisper database files correctly.

I will add the HTTP listener code so that it becomes a fully-fledged DataYours-Graphite daemon.

The ease of doing this, and the reliability (so far) of the BBB means that I am wondering just how much of the Luup environment I can/should emulate in order to port more of my plugins to this platform.

Just a snippet of the running log and created files to prove it:
Code: [Select]
Sun Mar  8 14:41:35 2015, call: UDP_listener_2003, cpu = 1.080sec (0.155%)
Sun Mar  8 14:41:36 2015, call: UDP_listener_2003, cpu = 1.090sec (0.157%)
Sun Mar  8 14:41:37 2015, call: UDP_listener_2003, cpu = 1.090sec (0.156%)
^Clua5.1: Test_Cache.lua:76: interrupted!
stack traceback:
[C]: in function 'select'
Test_Cache.lua:76: in main chunk
[C]: ?
root@beaglebone:~# ls -l
total 112
-rwx------ 1 root root  6208 Mar  7 22:29 L_DataCache7.lua
-rwx------ 1 root root 15417 Mar  7 22:29 L_DataDaemon7.lua
-rwx------ 1 root root 27340 Mar  7 22:29 L_DataWhisper7.lua
-rwx------ 1 root root  2544 Mar  8 14:29 Test_Cache.lua
-rw-r--r-- 1 root root  6132 Mar  8 14:40 Vera-45101161.005.urn^akbooer-com^serviceId^EventWatcher1.AppMemoryUsed.wsp
-rw-r--r-- 1 root root  6132 Mar  8 14:40 Vera-45101161.005.urn^akbooer-com^serviceId^EventWatcher1.CpuLoad05.wsp
-rw-r--r-- 1 root root  6132 Mar  8 14:40 Vera-45101161.005.urn^akbooer-com^serviceId^EventWatcher1.MemAvail.wsp
-rw-r--r-- 1 root root  6132 Mar  8 14:40 Vera-45101161.005.urn^akbooer-com^serviceId^EventWatcher1.MemFree.wsp
-rw-r--r-- 1 root root  6132 Mar  8 14:34 Vera-45101161.019.urn^upnp-org^serviceId^TemperatureSensor1.CurrentTemperature.wsp
-rw-r--r-- 1 root root  6132 Mar  8 13:54 Vera-45101161.021.urn^upnp-org^serviceId^TemperatureSensor1.CurrentTemperature.wsp
-rw-r--r-- 1 root root  6132 Mar  8 14:39 Vera-45101161.047.urn^akbooer-com^serviceId^DataYours1.AppMemoryUsed.wsp
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 08, 2015, 05:46:11 pm
Hi akbooer,

I've tried Datacache on Rpi and the messages seems ok (I've attached a log).

The only error message is :

sh: 1: GetNetworkState.sh: not found

Have you any idea ?

tnks donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 08, 2015, 06:07:51 pm
Yes.  That error is because the mechanism used on Vera to recover the IP address of the machine is not available on the RPi.  It generates an error, but then goes ahead and uses the 127.0.0.1 IPv4 loopback 'localhost' address, which is fine for the purpose.

So what next?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 08, 2015, 06:29:32 pm
Yes.  That error is because the mechanism used on Vera to recover the IP address of the machine is not available on the RPi.  It generates an error, but then goes ahead and uses the 127.0.0.1 IPv4 loopback 'localhost' address, which is fine for the purpose.

So what next?

I've intended you have to add some code to Rpi to access the data from DataDash and DataGraph on Veraedge.

Is it correct ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 08, 2015, 07:04:51 pm
...you have to add some code to Rpi to access the data from DataDash and DataGraph on Veraedge.

Is it correct ?

Possibly.  I take it that your storage on the RPi is not visible from Vera (you didn't want to use CIFS)?  So I need to check that DataGraph can access DataCache remotely with some other mechanism (native Graphite cache access.)

Well, we are making some progress, anyway!

PS: I'd be interested to know the asymptotic value of the cpu percentage that you get if you leave things running a while on the RPi.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 09, 2015, 03:49:16 am
Hi akbooer,

Yes i'd like to install less extra software as possible on VeraEdge .

Porting datagraph on Rpi too will simplify the remote access of datadash ?

Ps: i'll measure the cpu Usage of datacache running.

Donato

Title: Re: DataYours on Raspberry Pi (or BeagleBone Black)
Post by: akbooer on March 10, 2015, 06:55:38 am
Porting datagraph on Rpi too will simplify the remote access of datadash ?
Yes, in fact I'll take a look at moving DataDash there as well.  I have just discovered that the BeagleBone Black comes with CIFS ready installed (not sure if this is so for the RPi) so it's trivial to have the Whisper database sitting on a NAS.

Quote
Ps: i'll measure the cpu Usage of datacache running.

You might take a look at the memory usage too... it was a delight to see enormous amounts of free space available - quite a difference from Vera.
Title: Re: DataYours on Raspberry Pi (or BeagleBone Black)
Post by: d55m14 on March 10, 2015, 08:47:45 am
Porting datagraph on Rpi too will simplify the remote access of datadash ?
Yes, in fact I'll take a look at moving DataDash there as well.  I have just discovered that the BeagleBone Black comes with CIFS ready installed (not sure if this is so for the RPi) so it's trivial to have the Whisper database sitting on a NAS.

Quote
Ps: i'll measure the cpu Usage of datacache running.

You might take a look at the memory usage too... it was a delight to see enormous amounts of free space available - quite a difference from Vera.

I've verified : Rpi comes with CIFS ready  installed too.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 11, 2015, 07:04:08 pm
Just an update on progress: I now have both DataCache and DataGraph running unmodified on my BeagleBone Black, and now see no barrier (famous last words) in doing the same for DataDash.

This means that with only DataWatcher running on Vera to pick up variable changes, the rest of the DataYours daemons could be run remotely on RPi or BBB.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 11, 2015, 07:06:24 pm
Just an update on progress: I now have both DataCache and DataGraph running unmodified on my BeagleBone Black, and now see no barrier (famous last words) in doing the same for DataDash.

This means that with only DataWatcher running on Vera to pick up variable changes, the rest of the DataYours daemons could be run remotely on RPi or BBB.

Wonderful !!

great job akbooer !!
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 12, 2015, 07:25:35 pm
DataDash is now working on my BeagleBone Black, but I do want to make some changes because it was written (not unreasonably) with the assumption that it was actually running on a Vera, and that's not now true.  So I can get you up and going and would be grateful for a test on a RPi.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 03:33:40 am
DataDash is now working on my BeagleBone Black, but I do want to make some changes because it was written (not unreasonably) with the assumption that it was actually running on a Vera, and that's not now true.  So I can get you up and going and would be grateful for a test on a RPi.
Ok akbooer,

Give me info for the installation in Rpi and I'll test .

Tnks Donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 13, 2015, 01:42:56 pm
The only error message is :

sh: 1: GetNetworkState.sh: not found

Have you any idea ?

You can suppress this message by creating the file /usr/bin/GetNetworkState.sh with the contents:
Code: [Select]
echo -n 172.16.42.88
replacing the IP address with that of your own (RPi) machine. 

Then, all the DataYours components will pick up the right address.  I'm not Unix savvy enough to make this pick up the machine IP automatically (and it seems, in fact, to be far from straight-forward.)

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 02:25:04 pm
The only error message is :

sh: 1: GetNetworkState.sh: not found

Have you any idea ?

You can suppress this message by creating the file /usr/bin/GetNetworkState.sh with the contents:
Code: [Select]
echo -n 172.16.42.88
replacing the IP address with that of your own (RPi) machine. 

Then, all the DataYours components will pick up the right address.  I'm not Unix savvy enough to make this pick up the machine IP automatically (and it seems, in fact, to be far from straight-forward.)

Hi akbooer,

I've tested and the error message disappeared. Good !! Thanks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 13, 2015, 03:20:37 pm
Give me info for the installation in Rpi and I'll test .

OK.  Here we go...

The standard Vera installation files you will need on your RPi are:
Code: [Select]
L_DataCache7.lua
L_DataDaemon7.lua
L_DataGraph7.lua
L_DataLibrary7.lua
L_DataWhisper7.lua

The attached files:
Code: [Select]
OpenLuup.lua
RPI_BBB_DataYours_Beta1.lua
L_DataDash7.lua   (very slightly modified from the App Store release)

Also
Code: [Select]
akb-json.lua
...available from here: http://forum.micasaverde.com/index.php/topic,29989.msg213050.html#msg213050 (http://forum.micasaverde.com/index.php/topic,29989.msg213050.html#msg213050)

BEFORE YOU DO ANYTHING ELSE: you must modify the RPI_BBB_DataYours_Beta1.lua file: the two parameters LOCAL_DATA_DIR and VERAS need to be changed to your particular needs.

Then you just need to run that file with the command:
Code: [Select]
lua5.1 RPI_BBB_DataYours_Beta1.lua
(or maybe just lua instead of lua5.1)

This will (should) create a virtual Luup environment, creating three devices, and away you go.

-----------

[Edit: you also need to create a /www/ directory on the RPi for the DataGraph configuration file.]

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 06:57:36 pm
Great akbooer !!

I've followed your clear indications and all seem OK.
At the moment the data are stored on the Rpi SD .

I've left on VeraEdge only Datayours7 and the Watcher.

I've added a device to VeraEdge and it seems that in Dashboard I can see it only after a restart of your lua programs.

donato

 
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 13, 2015, 07:00:34 pm
Excellent!

Yes, the device configuration is only read at initialisation.  Adding devices is not something which happens too often.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 07:27:44 pm
Excellent!

Yes, the device configuration is only read at initialisation.  Adding devices is not something which happens too often.

Ok for device configuration.

I've defined some graphs but I can't see them after a restart of lua. Repeated the process but the result is the same.

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 13, 2015, 07:34:38 pm
I've defined some graphs but I can't see them after a restart of lua. Repeated the process but the result is the same.

Ah! Well spotted!  I knew there was something else...

You need to create a /www/ directory on RPi, because this is where DataGraph keeps its configuration file with graph definitions.

We are just at the edge of being able to fool the unmodified DataYours daemons that they are still on Vera.  My current difficulties lie around the way that these different platforms deal with web servers - the location of the /www/ directory is a case in point.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 07:44:29 pm
Fine !!

www dir created and now the graphs are ok after a restart. Excellent job !!

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 13, 2015, 07:48:54 pm
I've a spare Raspberry pi : do you think is possibile to use it for Datayours with the Watcher deamon running on Veraedge and store the data on a storage connected to Raspberry ?

This would be fun to do and is certainly possible.

..and so we see that it IS possible, on both Raspberry PI and BeagleBone Black.

The whole point of basing DataYours on Carbon/Graphite was that it is an open system, not rooted in any particular architecture, so that you could do exactly this sort of thing.

...I'm delighted you asked the original question just under a month ago.  I have had fun doing it, and the choice of Carbon/Graphite as the underlying architecture was definitely the right one.   This also gives a new impetus to improving DataYours, in particular DataDash, but I'm now also very intrigued to see which other plugins can be ported: Netatmo will be next, since that just creates virtual devices and doesn't need Z-wave.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 13, 2015, 08:02:55 pm
I've a spare Raspberry pi : do you think is possibile to use it for Datayours with the Watcher deamon running on Veraedge and store the data on a storage connected to Raspberry ?

This would be fun to do and is certainly possible.

..and so we see that it IS possible, on both Raspberry PI and BeagleBone Black.

The whole point of basing DataYours on Carbon/Graphite was that it is an open system, not rooted in any particular architecture, so that you could do exactly this sort of thing.

...I'm delighted you asked the original question just under a month ago.  I have had fun doing it, and the choice of Carbon/Graphite as the underlying architecture was definitely the right one.   This also gives a new impetus to improving DataYours, in particular DataDash, but I'm now also very intrigued to see which other plugins can be ported: Netatmo will be next, since that just creates virtual devices and doesn't need Z-wave.

I completely agree with you . I think that this is a right approach : it will stabilize the zwave controller and will give more flexibility on plugins and applications and on the hw platform.



   
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on March 16, 2015, 02:44:18 pm
... I'm now also very intrigued to see which other plugins can be ported: Netatmo will be next, since that just creates virtual devices and doesn't need Z-wave.

Just as a footnote: Netatmo is now running on the BeagleBone Black, along with DataYours, and so is Rex's excellent RBLuaTest.

...but that is a story for a different thread.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on March 16, 2015, 04:30:07 pm
... I'm now also very intrigued to see which other plugins can be ported: Netatmo will be next, since that just creates virtual devices and doesn't need Z-wave.

Just as a footnote: Netatmo is now running on the BeagleBone Black, along with DataYours, and so is Rex's excellent RBLuaTest.

...but that is a story for a different thread.

fine akbooer  !!

my Datayours installation on Rpi is running with no  problems and with limited usage of computers resources.

donato 
Title: Re: DataYours on Raspberry Pi
Post by: airedale on May 10, 2015, 07:48:48 pm
I just went through the setup and all seems to be working well.

I am launching this from the command line and it is running well.

How are you guys getting it to turn into a daemon?

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on May 11, 2015, 05:54:55 am
I just went through the setup and all seems to be working well.
Excellent news!

Quote
I am launching this from the command line and it is running well.
Yes, that's the way I use it mostly because I'm still developing the openLuup framework.

Quote
How are you guys getting it to turn into a daemon?

I'm running Debian on a BeagleBone Black, and for me, this works a treat...
Code: [Select]
nohup lua5.1 RPI_BBB_DataYours_Beta1.lua > luup.log &
...then you can even tail the log file if you need to see what's going on.

The latest openLuup framework is much more capable than that posted here and runs many other plugins unmodified.  In particular, the altUI plugin by @amg0 gives a complete UI (much better than UI7.)  I'm not quite ready to start a separate thread on that, but I'm looking at it as an openHAB alternative.


Title: Re: DataYours on Raspberry Pi
Post by: sjnelle on June 25, 2015, 02:43:04 am
Nice work.

Where can I download the plugin?


What do I have to install on the vera?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on June 25, 2015, 04:52:27 am
Where can I download the plugin?
The DataYours plugin, and the associated DataYours - Graphite daemons are downloadable from the MiOS App store.  A slight change to the DataDash
code is required and attached to a few posts back in this thread (here http://forum.micasaverde.com/index.php/topic,31078.msg225202.html#msg225202 (http://forum.micasaverde.com/index.php/topic,31078.msg225202.html#msg225202)) which also has the 'openLuup' code to run on the Pi.

Quote

What do I have to install on the vera?
Just the DataWatcher daemon is all that's required, although you can do a full DataYours  install and just use it to run the watcher.

I'm assuming that you know what DataYours is all about and that you're wanting to migrate part of it to a Pi.

A much more comprehensive version of the 'openLuup' code is under construction and should be able to run almost any plugin on a Pi, or similar.
Title: Re: DataYours on Raspberry Pi
Post by: sjnelle on June 26, 2015, 06:44:50 am

Quote
The DataYours plugin, and the associated DataYours - Graphite daemons are downloadable from the MiOS App store.  A slight change to the DataDash
code is required and attached to a few posts back in this thread (here http://forum.micasaverde.com/index.php/topic,31078.msg225202.html#msg225202 (http://forum.micasaverde.com/index.php/topic,31078.msg225202.html#msg225202)) which also has the 'openLuup' code to run on the Pi.

But I have first to install them on the vera and then copy the files to the PI?
Is it possible to download the files without install?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on June 26, 2015, 04:52:07 pm
You need a minimum of files on Vera to run DataWatcher anyway.  The files are very small and do not take up significant resources.  It's really the easiest way to get hold of them, but if this gives you any trouble, I'll post a zip file.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on July 10, 2015, 12:38:35 pm
I've just released a much more general version of the openLuup system, which means that a specific version for DataYours is not required.

The release here http://forum.micasaverde.com/index.php/topic,32315.msg240753.html#msg240753 (http://forum.micasaverde.com/index.php/topic,32315.msg240753.html#msg240753), plus the following updates on that thread, should run many standard Vera plugins.

Modifications to the startup file (for openLuup) to run Datayours are here: http://forum.micasaverde.com/index.php/topic,32315.msg240753.html#msg240753 (http://forum.micasaverde.com/index.php/topic,32315.msg240753.html#msg240753).

An update to the DataYours Lua file (which still runs on a standard Vera) is attached here.  What it does it to transfer the configuration parameters to all the various daemons, so you only have to configure any given parameter (eg. LOCAL_DATA_DIR) in one place, rather than in each of the daemons.

One 'gotcha' if you choose to run DataWatcher under openLuup: you need to be sure that all of the devices variables that it is watching do already exist - this is important because devices and variables are not persistent under openLuup.  The simple solution to this is to delay the startup code until all other devices have started.  On the other hand, if you're just logging metrics from a remote system, then this isn't a thing you need to consider.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 03, 2015, 03:44:16 am
Hi akbooer,

I'm trying to replicate datayours data on a Rpi with openLuup e Datayour (Cache, Graph, Dash). The Watcher daemon is at the moment on the Veraedge controller.

Can you confirm that on the destination system (rpi) are replicated from the origin the treemap and the watched variables ? The storage schema of replicated and watched variables is for all "1h:7d average" and is different from the origin storage schema : is it correct ? If yes can I changed them on the destination system ?

tnks

donato   
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 03, 2015, 05:08:18 am
Can you confirm that on the destination system (rpi) are replicated from the origin the treemap and the watched variables ?
No, I don't think so... I'm struggling a bit with the wording, so my understanding of the question may not be correct.

If you have a complete DataYours system (or at least Watcher and Cache) on your Edge, and you're asking if the database is automatically replicated to your DataCache on the RPi, then the answer is definitely no.  DataYours is, or can be, a completely distributed system, so DataWatcher knows nothing about the database structure of where it sends the data, apart from the actual IP address and port number.  It's perfectly possible for a single DataWatcher to send data to multiple distinct DataCache instances which might have completely different archive structures.

Quote
The storage schema of replicated and watched variables is for all "1h:7d average" and is different from the origin storage schema : is it correct ? If yes can I changed them on the destination system ?

The "1h:7d average" archive structure is the default in the absence of any other information (chosen to take up little space but to capture the basics, in case someone sets up the database on, for example, the Vera itself.)  The actual archive structure, on a per metric (ie. device variable) basis, is determined at file creation.  This can either be set up manually, or using schema and aggregation rules.  There are, in fact, three ways to set up a new database with the structure you want:
I hope this clarifies things a little.  Setting up a distributed database system is not totally trivial, but it's not really very hard if you've got the basic concepts right.  Just to be clear, my recommendation is to create schema and archive rule files, which should be placed in your destination database directory (LOCAL_DATA_DIR) for your RPi.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 03, 2015, 05:18:23 am
Hi akbooer,

Tnks for your reply and sorry for my english. Now the DY functions are a bit clarified for me. I'll try some tests .

If communication between sender and receiver is down for a period is it possible to resync the data base ?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 03, 2015, 05:54:51 am
Tnks for your reply and sorry for my english.
Nothing wrong with your english at all!  Just the difference between the way that you describe the system and I describe the system.

Quote
Now the DY functions are a bit clarified for me. I'll try some tests .
OK, good luck with that.

Quote
If communication between sender and receiver is down for a period is it possible to resync the data base ?
It depends what you want.  DataWatcher itself has no way of telling if the remote system is down or not - this is because the communication is via UDP datagrams which are incredibly lightweight and part of the reason that DataYours imposes virtually no cpu or I/O overhead on the system: there is no client/server handshake.

However, if you are running DataCache locally (on the Edge) with its own database (and perhaps a rather short maximum retention time to keep file sizes down), it may be possible to arrange a subsequent merge of the missing data with that captured locally using a straight-forward Whisper database utility.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 03, 2015, 07:44:39 am
Tnks for your reply and sorry for my english.
Nothing wrong with your english at all!  Just the difference between the way that you describe the system and I describe the system.

Quote
Now the DY functions are a bit clarified for me. I'll try some tests .
OK, good luck with that.

Quote
If communication between sender and receiver is down for a period is it possible to resync the data base ?
It depends what you want.  DataWatcher itself has no way of telling if the remote system is down or not - this is because the communication is via UDP datagrams which are incredibly lightweight and part of the reason that DataYours imposes virtually no cpu or I/O overhead on the system: there is no client/server handshake.

However, if you are running DataCache locally (on the Edge) with its own database (and perhaps a rather short maximum retention time to keep file sizes down), it may be possible to arrange a subsequent merge of the missing data with that captured locally using a straight-forward Whisper database utility.

Yes I run DataCache locally so I'll ask you about the Whisper Utility.

For my check : is ok the list of Datayours files (attached) for a complete configuration (Watcher, Cache, Graph Dash) on a Rpi with Openluup ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 03, 2015, 08:03:05 am
For my check : is ok the list of Datayours files (attached) for a complete configuration (Watcher, Cache, Graph Dash) on a Rpi with Openluup ?

Ah, sorry, I never did respond to your earlier question about that.

No, this list is not quite right: is has some duplicated items from various posts, some things not needed, etc...

I'll post a complete set separately.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 03, 2015, 12:59:51 pm
Hi akbooer,

I'm doing some tests with Rpi/Openluup/Datayour but I've some questions for you .

1) To activate a complete Datayour (Watcher, Cache, Graph, Dash) on Rpi, remote to a Edge Controller,  do I have to install on Rpi at least Openluup, VeraBridge and DY ? In this case I see on DY Dash two controller (the remote one and a local controller with id 88800000) with the complete sets of device and variables replicated.  In this configuration I can activate watched variables from the devices of local controller (88800000) NOT from the remote controller  . Is it correct ? 

2) With a watched variable activated (i see the graph from the wispher DB) I've tried to define a graph but also if I don't get error msg the graph is not created.

tnks

donato

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 03, 2015, 01:25:46 pm
1) To activate a complete Datayour (Watcher, Cache, Graph, Dash) on Rpi, remote to a Edge Controller,  do I have to install on Rpi at least Openluup, VeraBridge and DY ? In this case I see on DY Dash two controller (the remote one and a local controller with id 88800000) with the complete sets of device and variables replicated.  In this configuration I can activate watched variables from the devices of local controller (88800000) NOT from the remote controller  . Is it correct ? 
Your don't need VeraBridge, it's nothing to do with DataYours per se and only there to provide a Zwave bridge through to openLuup - it gives the RPi control of devices on the remote Vera.  This only works with machines on the same LAN (or VPN).  As I understand you application, it's not what you need - I thought you needed only the most tenuous of connections using DataWatcher to send variable changes to DataCache on the RPi.

The default id for an openLuup installation happens to be 88800000, although it is configurable.

If you DID want to select remote variables for watching using DataYours, then you would need to add the Vera IP to the VERAS configuration parameter.

Quote
2) With a watched variable activated (i see the graph from the wispher DB) I've tried to define a graph but also if I don't get error msg the graph is not created.

The graph configuration file is normally stored on Vera's /www/ directory (the same is true for the DataWatcher configuration.)  I really must make this configurable, but at the moment it's fixed.  My guess is that you have not created a /www/ directory on the RPi.

Attached you will find a new startup file for openLuup, which checks the installation for anomalies like missing files or directories before launching openLuup.  It aborts if there's a fatal flaw in the setup configuration, but otherwise issues warning messages to standard output.  If you're starting openLuup with the "nohup" command, then standard output is redirected to the file "nohup.out".

Hope this helps.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 03, 2015, 07:08:32 pm
Hi akbooer,

I've installed on Rpi Openluup, a complete DY with the VERAS parameter and NO Verabridge.

I can see the treemap (see the attachments ) but when I try to activate a watched variable I get the No Handler message.

I've attached a screenshots of configuration.

Have you any idea of my mistake ?

At startup I don't have error messages.

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 04, 2015, 02:38:33 am
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 04, 2015, 03:08:20 am
  • It looks like you don't have DataWatcher running on the Vera?
  • How have you configured DataYours on the RPi?
  • What does your startup file on the RPi contain?

Datawatcher must run on Edge ? Can't I have all on Rpi included DataWatcher ?

Attached mu startup.lua on rpi

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 04, 2015, 03:32:19 am
No, you have to have a DataWatcher on every machine from which you want to receive data.  Sorry, thought this was clear from previous discussions.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 04, 2015, 04:26:46 am
No, you have to have a DataWatcher on every machine from which you want to receive data.  Sorry, thought this was clear from previous discussions.

It's my little attention you're always very clear.

So the minimal configuration of DY on Edge is Datawatcher and on Rpi I've to install datayours with watcher cache graph and dash ?

Tnks

Donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 04, 2015, 04:31:16 am
So the minimal configuration of DY on Edge is Datawatcher and on Rpi I've to install datayours with watcher cache graph and dash ?

No, the only thing you need running is DataWatcher.  It uses the DataDaemon too.  But you don't need the rest of DataWatcher and you don't even need to set it up as a device - it can just run in the background (started in Vera's startup code) and you will not even see it.

There's instructions on how to do this somewhere in the DataYours threads, but I am away for the rest of the day so unable to help further right now.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 04, 2015, 05:17:37 am
So the minimal configuration of DY on Edge is Datawatcher and on Rpi I've to install datayours with watcher cache graph and dash ?

No, the only thing you need running is DataWatcher.  It uses the DataDaemon too.  But you don't need the rest of DataWatcher and you don't even need to set it up as a device - it can just run in the background (started in Vera's startup code) and you will not even see it.

There's instructions on how to do this somewhere in the DataYours threads, but I am away for the rest of the day so unable to help further right now.

If I have the following configuration :

1) Edge with Datawatcher;

2) Rpi with Dash Graph, Cache

can I configure the watched variables from the Dash on Rpi setting the following startup.lua with NO WATCHER ?

do -- DataYours
  local dy7 = luup.create_device ('', '', "DataYours", "D_DataYours7.xml","I_DataYours7.xml")        -- create the device
  luup.variable_set ("urn:akbooer-com:serviceId:DataYours1", "DAEMONS", "Graph, Dash, Cache", dy7)
  luup.variable_set ("urn:akbooer-com:serviceId:DataYours1", "LOCAL_DATA_DIR", "/nas/whisper/", dy7)
  luup.variable_set ("urn:akbooer-com:serviceId:DataYours1", "VERAS", "192.168.xx.xxx", dy7)
end

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 04, 2015, 02:18:03 pm
Hi akbooer

attached a doc with some tests on Rpi-Openluup-Datayours configuration.
In bold char the configuration changes.
tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 04, 2015, 05:44:58 pm
can I configure the watched variables from the Dash on Rpi setting the following startup.lua with NO WATCHER ?

Yes, no need at all to have any component of DataYours running if you don't need it.  You could even, for example, configure DataWatcher on the Vera entirely remotely with HTTP calls from either a browser, or Luup code on the RPi, and also plot graphs locally without using DataDash at all.

Having had a day out to think a bit about this, I now see where you were coming from with using VeraBridge: it is indeed perfectly possible to bridge to the remote Vera and simply use DataYours to record the bridged variables locally on the RPi.  I read with interest your comprehensive test of various configurations and concur with the findings, all of these are valid, if sometimes bizarre, combinations. The fact remains, however, that VeraBridge will place a much higher demand in terms of cpu and I/O on the system than simply using DataWatcher, and if you were just using this to access the data (rather than remotely control Vera) then it would be a bit of overkill.

So really it just depends on what you want to do - DataYours is, as you have found out, very modular and very configurable.   VeraBridge adds another set of options.  You should perhaps note that whilst DataYours has logged many millions of variable changes and has been very extensively tested for well over a year, openLuup and VeraBridge are new and definitely Beta releases.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 17, 2015, 08:13:35 am
@d55m14

Does this update fix your problem with UDP_RECEIVER_PORT not being set?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 17, 2015, 09:05:57 am
@d55m14

Does this update fix your problem with UDP_RECEIVER_PORT not being set?

tnks akbooer,

I've multiple VeraEdge/DYonRpi installed on different site and a DYonRpi in a central site. I'd like to have different port number between local copy of Wispher database and the central copy .

tnks again

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 17, 2015, 09:11:14 am
Does that mean you're running two DataCache daemons on the same IP but with different port numbers?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 17, 2015, 09:21:48 am
Does that mean you're running two DataCache daemons on the same IP but with different port numbers?

No. One Datacache on Rpi is local to Veraedge and the other is remote (public IP) always on Rpi.

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 17, 2015, 09:28:54 am
OK, fair enough.  I don't see that you need to change the UDP port at all, then, but it's your call.

Did that fix work, by the way?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 17, 2015, 09:31:32 am
OK, fair enough.  I don't see that you need to change the UDP port at all, then, but it's your call.

Did that fix work, by the way?

Yes the fix works. great !! tnks
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 26, 2015, 10:30:05 am
Hi akbooer,

I've active a configuration with a central Rpi (IP address public) with Openluup e Datayours. The active daemons  of DY on Rpi are cache, graph and dash. The cache daemon receives data from multiple remote VeraEdge with Datayours (Watcher) installed.

On the central Rpi I've active a web server too to access the graph of watched variables.

My question : is it possible from a web page (html) on the Rpi to access the realtime value of a watched variable as it is captured from the cache daemon ?

Excuse me if not clear .

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 26, 2015, 12:02:54 pm
It is possible to write simple piece of Luup/Lua code which would serve up values or graphs for web pages which are refreshed automatically.  I have an intranet server at home which does this.  Examples attached.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 26, 2015, 12:08:04 pm
It is possible to write simple piece of Luup/Lua code which would serve up values or graphs for web pages which are refreshed automatically.  I have an intranet server at home which does this.  Examples attached.

Hi akbooer,

have you some simple piece of Luup/Lua code I can re-use. Only if you have already . I don't want to waste your time.

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 27, 2015, 05:03:00 am
The luup.register_handler call can service HTTP requests and deliver any information you want to a web page:  a whole page or just a data update responding to an AJAX call.

For a simple solution you could simply have a page which refreshed itself every five minutes, or whatever.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 27, 2015, 08:48:13 am
The luup.register_handler call can service HTTP requests and deliver any information you want to a web page:  a whole page or just a data update responding to an AJAX call.

For a simple solution you could simply have a page which refreshed itself every five minutes, or whatever.

Hi akbooer,

excuse for my ignorance but can I use the luup.register_handler also in my case where the Rpi with Datayours (Cache Graph Dash) and Openluup is remote to Veraedge ? From Rpi I can access Veraedge only via Mios Server.

Can you give me a bit more information ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 27, 2015, 11:28:30 am
excuse for my ignorance but can I use the luup.register_handler also in my case where the Rpi with Datayours (Cache Graph Dash) and Openluup is remote to Veraedge ? From Rpi I can access Veraedge only via Mios Server.

You original question was this:
Quote
I've active a configuration with a central Rpi (IP address public) with Openluup e Datayours. The active daemons  of DY on Rpi are cache, graph and dash. The cache daemon receives data from multiple remote VeraEdge with Datayours (Watcher) installed.

On the central Rpi I've active a web server too to access the graph of watched variables.

My question : is it possible from a web page (html) on the Rpi to access the realtime value of a watched variable as it is captured from the cache daemon ?

From reading that, I understand that all data ends up on the RPi, ergo the RPi has access to all data changes from the remote machines without resorting to the MiOS servers.  A luup.register_handler callback on the RPi can serve the latest information in response to an HTTP request.  It's what DataDash, running on the RPi, already does.

I don't see the problem?

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 27, 2015, 12:10:49 pm
Hi akbooer

is there a HTTP call, with specific parameters, I can do to DataDash to obtain the most recent value of a watched variable ?


tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 27, 2015, 12:58:58 pm
is there a HTTP call, with specific parameters, I can do to DataDash to obtain the most recent value of a watched variable ?

It's DataGraph, actually:

Quote from: User Guide, p.19
DataGraph is the Graphite Web App look-alike, see:
http://graphite.readthedocs.org/en/latest/render_api.html
It reads data from the Whisper database and plots or prints it out in a variety of formats.

http://<yourVeraIP>:3480/data_request?id=lr_render2&target=Vera-12345678.321.urn:upnp-
org:serviceId:TemperatureSensor1.CurrentTemperature

This is currently only a partial implementation, so following along with the Graphite render API documentation link, the exceptions are here below:
...

I'd recommend JSON format and a time window longer than the update period of your measurements.  You'll get back multiple values (depending on the archive structure), but just pick the newest.

With a request like
Code: [Select]
http://172.16.42.151:3480/data_request?id=lr_render&target=Vera-35104571.208.urn:micasaverde-com:serviceId:EnergyMetering1.KWH&from=-1h&format=json

I get back this
Code: [Select]
[{
  "target": "Vera-35104571.208.urn:micasaverde-com:serviceId:EnergyMetering1.KWH"
,  "datapoints": [
  [4416.164, 1440690000],
  [4416.326, 1440691200],
  [4416.498, 1440692400]
  ]
}]

It's the last line of data you need.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 27, 2015, 03:21:13 pm
Thank you very much for your support..


donato
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 29, 2015, 04:18:10 am
Hi akooer,

I'm trying to use the JSON output of Graph to get the most recent value of a variable but I'm getting a cross domain origin error . The web page e DY are on the same Rpi.

The message is :

http://192.168.55.104:3480/data_request?id=lr_render&target=Vera-45103954.0?rn:micasaverde-com:serviceId:EnergyMetering1.Watts&from=-10min&format=json. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://192.168.55.104' is therefore not allowed access.

Do you know how to enable cross domain origin on the web server that respond to port 3480 ? On the Rpi is installed Apache. Is there another web server for DY ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 29, 2015, 05:42:38 am
We seem to have drifted far away from the original topic of this thread and, at the same time, away from any expertise I might have - I really know very little about Web programming. ( I am, by the way still grateful to you for having asked whether DataYours could run on a RPi in the first place and set me off on the path to what has become openLuup.)

Anyway, observing simply that  'it works for me', won't help, and maybe we're not doing the same thing.  My previously posted pages were from a locally served webpage with embedded HTTP calls to the remote machine port 3480.  I am assuming that the URL you gave actually does work when posted into your browser.  There are, perhaps, a number of things to try:


Don't know if this helps.  (BTW, I'm intrigued by the system you are building - what/who is it actually for?)
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 29, 2015, 01:43:45 pm
Hi akbooer,

tnks for your suggestions and I'll try to remain inside the original topic.

I'm trying to setup a configuration as simple as possible to monitor some photovoltaic installation on different geographic sites.

The solution should collect  data  in a central site and should be indipendent of hw present (panels, inverter, etc) . At single remote installation there is a zwave controller (VeraEdge) with power meter (Aeon HEM G2). The zwave Controller with your fantastic DataYours-Watcher captures data and archive them locally and send them to the central system (a Rpi with DY Cache, Graph, Dash) too.

On the Central System (Rpi) I have the data of all PV installations and I'd like to activate a web page that a user can access to display the real time production power and some graphs of his PV system.

About your suggestions :

1) the page is served locally on the Rpi where is DY;
2) I've tried to activate the 3480 port on Apache web server but I get an error msg saying that the port 3480 is already active and used.
3)my security settings are the default settings of a rpi installation plus your sw (DY and Openluup);
4)can you tell me how to add the Access-Control-Allow-Origin to the Graph pages ? I haven't found where to define it.

For any more info or explanation I'm at your disposal.

tnks

donato 
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 29, 2015, 01:52:30 pm
1) the page is served locally on the Rpi where is DY;
I meant locally to the browser - maybe this is the same thing for you.

Quote
2) I've tried to activate the 3480 port on Apache web server but I get an error msg saying that the port 3480 is already active and used.
No, you won't be able to do that - openLuup has taken it already.  Simply use the openLuup server itself - its file system root is /etc/cmh-ludl

Quote
3)my security settings are the default settings of a rpi installation plus your sw (DY and Openluup);
I never fiddle with this - way beyond my expertise!

Quote
4)can you tell me how to add the Access-Control-Allow-Origin to the Graph pages ? I haven't found where to define it.
Let me read up more about this - I know where I define the reply headers, anyway - but I think using the 3480 server itself will fix the problem.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 29, 2015, 02:29:36 pm
Ok tnks I'll try the openluup server
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on August 30, 2015, 04:50:12 pm
Hi akbooer,

I used the openluup server for the web page and all works fine !! No more Access-Control-Allow-Origin error message and now I'm able to get the most recent values of watched variables using JSON.

tnks again

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on August 30, 2015, 05:10:37 pm
Excellent news - I hoped that would work.  You're close to achieving all you need to on this project?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 03, 2015, 05:01:19 am
I've in openluup/Datayours some  watched variables with the archive 1d:10y but the single item of the day is at 2 am.
Not entirely clear on this, since if you have a "1d:..." archive, the time of the data entry should, by definition, be at the start of each day.

Quote
In the startup of openluup I've :

attr ("timezone", "1")

and I'm at GMT+2 now.
This parameter is completely ignored by the openLuup system, since the time and locale should be set correctly on the host system.  The only reason to set it is if a remote UI wants to know it.

Quote
The time of the system (Rpi) is correct.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 03, 2015, 05:33:47 am
[/quote]

Yes it's seems a plotting issue.
I see the wrong time in the datapoints and so in the graph.
I've attached the datapoints e the Rpi local time
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 03, 2015, 07:30:31 am
Those datapoints are from a 10 minute archive.

The timestamp used in writing the archive is taken from the source machine, ie. not necessarily the machine which writes the data.  You should, therefore, also check that the locale, date, and time are correctly set at the origin.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 03, 2015, 10:13:46 am
Those datapoints are from a 10 minute archive.

The timestamp used in writing the archive is taken from the source machine, ie. not necessarily the machine which writes the data.  You should, therefore, also check that the locale, date, and time are correctly set at the origin.

On the  VeraEdge (the origin source) the timestamp (that I see from the UI interface) is correct.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 06:59:16 am
Those datapoints are from a 10 minute archive.

The timestamp used in writing the archive is taken from the source machine, ie. not necessarily the machine which writes the data.  You should, therefore, also check that the locale, date, and time are correctly set at the origin.

On the  VeraEdge (the origin source) the timestamp (that I see from the UI interface) is correct.

Hi akbooer,

can I do some tests to understand  where is the error ?

It seems that for 1d:xx  retention schema the daily plot item is taken at gmt time while my local time is gmt+2.

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 07:14:03 am
All timestamps should be in UTC.  This can be checked in the files.  Try plotting on the source machine and compare the difference (perhaps you don't have DataGraph there, though?)  Conversion to human readable times depends on the locale setting of the display machine.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 12:43:35 pm
All timestamps should be in UTC.  This can be checked in the files.  Try plotting on the source machine and compare the difference (perhaps you don't have DataGraph there, though?)  Conversion to human readable times depends on the locale setting of the display machine.

ok I'll check better..

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 12:49:36 pm
Hi akbooer,

I've another problem :

on the Openluup/DY I've defined :

[KWH]
pattern = \.KWH                 
retentions = 10m:1d,1d:10y

but when the wsp file is created it has

a retention of 1h:7d (image attached)

Have you any idea ?

tnks donato
 
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 01:02:34 pm
The has to be in the storage-schemas.conf file in the LOCAL_DATA_DIR where the RPi DataCache writes its files (ie. NOT on the machine running DataWatcher.)  Yes?

You should not have DataYours (specifically, DataCache) running if you delete a file and redefine it (since the headers are cached and this will screw things up.)
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 01:16:17 pm
The has to be in the storage-schemas.conf file in the LOCAL_DATA_DIR where the RPi DataCache writes its files (ie. NOT on the machine running DataWatcher.)  Yes?

You should not have DataYours (specifically, DataCache) running if you delete a file and redefine it (since the headers are cached and this will screw things up.)

Yes the storage schemas is on the LOCAL_DATA_DIR. Maybe DataCache active is the problem. Is there a specific command to close DY/DataCache and restart ?

Or if I want to change schemas or aggregation, can I set to NOWATCH the variables, make the changes, eventually delete the wsp files and then put the variables to WATCH ?


tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 01:46:49 pm
Restart Luup is safest way.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 01:56:48 pm
Restart Luup is safest way.

I did the following step in sequence :

1) changed the schemas and aggregation;
2) deleted the wsp files;
3) restarted the Rpi.

I'm quite sure that between the step 2 and 3 there weren't changes in the variables watched (KWH).

but the problem seems not resolved.

Is there something wrong in the following schemas for KWH ? :

[KWH]
pattern = \.KWH                 
retentions = 10m:1d,1d:10y

I need the 10m:1d retention to get plot point during the day so I can display the incremental value on a web page.

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 02:11:23 pm
I use:

Code: [Select]
[KWH]
pattern = \.KWH
retentions = 20m:30d,3h:1y,1d:10y

and it works for me (although there may be a problem if there's a syntax error elsewhere?)

Are any other schema rules there, and have they apparently been applied correctly?  I've never seen this problem before.

What does the Vera log show?  There should be clear messages when it reads rules and creates a new file.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 02:15:49 pm
I use:

Code: [Select]
[KWH]
pattern = \.KWH
retentions = 20m:30d,3h:1y,1d:10y

and it works for me (although there may be a problem if there's a syntax error elsewhere?)

Are any other schema rules there, and have they apparently been applied correctly?  I've never seen this problem before.

What does the Vera log show?  There should be clear messages when it reads rules and creates a new file.

Yes there are others rules that work correctly

The datayour (cache graph dash) is running in openluup and in VeraEdge is running datawatcher.

In the log of openluup there is :

2015-09-04 18:57:19.616   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:57:19.619   luup_log:3: DataCache2: created: Vera-45103914.003.urn:micasaverde-com:serviceId:EnergyMetering1.KWH
2015-09-04 18:57:19.619   luup_log:3: DataCache2: schema [default] = 1h:7d, aggregation KWH = max, xff = 0


while in the dash configuration option schemas you can see the image attached :

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 03:05:19 pm
PM me the entire openLuup log showing the creation of the problem file.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 03:25:34 pm
PM me the entire openLuup log showing the creation of the problem file.

Hi akbooer

I can't find in the PM the option to attach files

In the log :

2015-09-04 18:03:22.543   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.545   luup_log:3: DataCache2: created: Vera-45103914.025.urn:micasaverde-com:serviceId:EnergyMetering1.KWH
2015-09-04 18:03:22.546   luup_log:3: DataCache2: schema [default] = 1h:7d, aggregation KWH = max, xff = 0
2015-09-04 18:03:22.548   luup.variable_set:3: 5.urn:upnp-org:serviceId:altui1.DisplayLine1 was: 3 metrics now: 4 metrics #hooks:0
2015-09-04 18:03:22.552   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.556   luup_log:3: DataCache2: created: Vera-45103914.012.urn:upnp-org:serviceId:VContainer1.Variable1
2015-09-04 18:03:22.556   luup_log:3: DataCache2: schema KWH_Calcolato = 10m:1d,1d:10y, aggregation KWH_Calcolata = max, xff = 0

It seems that for KWH the schemas is the default while for the pattern Variable1 is correct.

Following the schemas :
#
[KWH]
pattern = \.KWH                 
retentions = 10m:1d,1d:10y
[KWH_Totale_Calcolato]
pattern = \.Variable2
retentions = 10m:1d,1d:10y
[KWH_Calcolato]
pattern = \.Variable1
retentions = 10m:1d,1d:10y
[Power]
pattern = \.Watts
retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y
[Power_Calcolata_Kwatt]
pattern = \.Variable3
retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y
[Power_Calcolata]
pattern = \.Variable4
retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 03:31:19 pm
cut and paste work?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 03:37:21 pm
cut and paste work?

I did but there is a limit in the number of chars
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 05:08:54 pm
OK, well this is the interesting bit...

Code: [Select]
2015-09-04 18:03:22.543   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.545   luup_log:3: DataCache2: created: Vera-XXX.025.urn:micasaverde-com:serviceId:EnergyMetering1.KWH
2015-09-04 18:03:22.546   luup_log:3: DataCache2: schema [default] = 1h:7d, aggregation KWH = max, xff = 0
2015-09-04 18:03:22.548   luup.variable_set:3: 5.urn:upnp-org:serviceId:altui1.DisplayLine1 was: 3 metrics now: 4 metrics #hooks:0
2015-09-04 18:03:22.552   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.556   luup_log:3: DataCache2: created: Vera-XXX.012.urn:upnp-org:serviceId:VContainer1.Variable1
2015-09-04 18:03:22.556   luup_log:3: DataCache2: schema KWH_Calcolato = 10m:1d,1d:10y, aggregation KWH_Calcolata = max, xff = 0
2015-09-04 18:03:22.565   luup.variable_set:3: 5.urn:upnp-org:serviceId:altui1.DisplayLine1 was: 4 metrics now: 5 metrics #hooks:0

which clearly shows that DataCache is using the default schema to create the .KWH metric, but uses the rule KWH_Calcolato to write the .Variable1 file.

I can only assume a syntax error in the rule configuration file before the [KWH] one.  Can you post the whole of storage-schemas.conf ?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 05:43:40 pm
OK, well this is the interesting bit...

Code: [Select]
2015-09-04 18:03:22.543   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.545   luup_log:3: DataCache2: created: Vera-XXX.025.urn:micasaverde-com:serviceId:EnergyMetering1.KWH
2015-09-04 18:03:22.546   luup_log:3: DataCache2: schema [default] = 1h:7d, aggregation KWH = max, xff = 0
2015-09-04 18:03:22.548   luup.variable_set:3: 5.urn:upnp-org:serviceId:altui1.DisplayLine1 was: 3 metrics now: 4 metrics #hooks:0
2015-09-04 18:03:22.552   luup_log:3: DataCache2: rules: #schema: 7, #aggregation: 20
2015-09-04 18:03:22.556   luup_log:3: DataCache2: created: Vera-XXX.012.urn:upnp-org:serviceId:VContainer1.Variable1
2015-09-04 18:03:22.556   luup_log:3: DataCache2: schema KWH_Calcolato = 10m:1d,1d:10y, aggregation KWH_Calcolata = max, xff = 0
2015-09-04 18:03:22.565   luup.variable_set:3: 5.urn:upnp-org:serviceId:altui1.DisplayLine1 was: 4 metrics now: 5 metrics #hooks:0

which clearly shows that DataCache is using the default schema to create the .KWH metric, but uses the rule KWH_Calcolato to write the .Variable1 file.

I can only assume a syntax error in the rule configuration file before the [KWH] one.  Can you post the whole of storage-schemas.conf ?

Attached the conf file
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 04, 2015, 06:03:48 pm
Yes, this is the problem.

Sadly the format for the file (not my design) is easily open to corruption.

Your line
Code: [Select]
pattern = \.KWH                 

actually has a large number of trailing spaces.

Replacing it with this:
Code: [Select]
pattern = \.KWH

(yes, it looks the same) should fix the problem.

I will also patch the parsing code to catch this problem.

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 04, 2015, 06:10:03 pm
Tnks akbooer,

I've to wait tomorrow morning to get some value. I'll give you a feedback.

thank you again for your precious support.

donato

Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 05, 2015, 03:15:49 am
Tnks akbooer,

I've to wait tomorrow morning to get some value. I'll give you a feedback.

thank you again for your precious support.

donato

It's working !! tnks again

have a nice we

donato
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 11, 2015, 04:29:59 pm
Hi akbooer,

I used the openluup server for the web page and all works fine !! No more Access-Control-Allow-Origin error message and now I'm able to get the most recent values of watched variables using JSON.

tnks again

donato

Hi akbooer,

I'm now using perfectly the openluup server and data from datayours for my web pages relative to photovoltaic monitor I've setup .

Do you know if is it possible to activate an authentication login for the users to web pages ?

I used the openluup server for problems with Access-Control-Allow-Origin error message when  using the web server apache on the same server where is DY/openluup ( different port number between web pages and http calls to DY-3480) .

In alternative did you have time to check where you could modify reply headers ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 12, 2015, 05:39:21 pm
Hi akbooer,

I did a test modifying the server.lua file.

I've added the following line :

"Access-Control-Allow-Origin: *",


to the following section  :

 local headers =  table.concat({
      "HTTP/1.0 " .. status,
      "Server: openLuup/" .. revisionDate,
      "Content-Type: " .. type or "text/plain",
      "Content-Length: " .. #response,
      "Connection: keep-alive",
      "Access-Control-Allow-Origin: *",
      crlf}, crlf)
--  _log ("Response headers: " .. headers)    -- **************
  sock: send(headers)
  sock: send (response)
  return #(response or {})

After some test it seems there is no more error msg due to different port between web pages (80) and datayours (3480) both on the same server.

Do you think that the change may be correct ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 14, 2015, 12:26:35 pm
I've been away for a couple of weeks, so not been able to attend to this.

Yes, the fix you made is what I had in mind - and if it fixes your port 80 issues, that's great.

I'll include it in the next release, unless someone tells me it's a serious security issue.
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 14, 2015, 01:29:44 pm
I've been away for a couple of weeks, so not been able to attend to this.

Yes, the fix you made is what I had in mind - and if it fixes your port 80 issues, that's great.

I'll include it in the next release, unless someone tells me it's a serious security issue.

ok tnks akbooer
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 21, 2015, 01:10:14 pm
Hi akbooer,

is it complicate to run on DY/Rpi/Openluup some Whisper script (https://github.com/graphite-project/whisper) ? For example whisper-merge.py, whisper-fill.py whipser-diff.py  that can be used to re-sync two or more Database with the same schema/aggregation ?

tnks

donato
Title: Re: DataYours on Raspberry Pi
Post by: Freddan101 on September 21, 2015, 11:48:20 pm
I thought I would give this a try. Which is the preferred OS/distribution for Openluup/DY?
Title: Re: DataYours on Raspberry Pi
Post by: d55m14 on September 22, 2015, 03:08:46 am
I' ve Openluup DY installed on Raspbian distribution based on Debian Wheezy. Tnks

Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 22, 2015, 03:24:47 am
It should run anywhere that you can install Lua. I have it running on Debian (BeagleBone Black), Mac OS, and OpenWRT (on Vera and also on Arduino Yun.)

It's really portable.
Title: Re: DataYours on Raspberry Pi
Post by: Freddan101 on September 22, 2015, 04:16:23 am
Wow, sounds like it could be installed anywhere. :) I'm leaning towards Raspbian or Minibian. I will use it for DY, web server and such sobdon't think I will need a fancy GUI.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 22, 2015, 04:21:14 am
It should even run on a PC!
Title: Re: DataYours on Raspberry Pi
Post by: Freddan101 on September 22, 2015, 06:47:52 am
Interesting... I have an HP Windows Home Server NAS. In the same time I need to make use of my Pi1. :)

Could you even run it on Synology?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on September 22, 2015, 07:14:49 am
Could you even run it on Synology?

Don't know for sure, but it's just Linux, isn't it?  I see other HA software (Domoticz) which also uses Lua (to some extent) runs there.

Start with your Pi and then try!
Title: Re: DataYours on Raspberry Pi
Post by: Freddan101 on September 22, 2015, 10:25:01 am
Will do! 👍🏻
Title: Re: DataYours on Raspberry Pi
Post by: airedale on October 08, 2015, 04:33:42 pm
Hi akbooer,


I read through all of this thread, along with a few other ones and it is obvious there has been multiple things making progress. The ability to offload onto another device, OpenLuup, and just general updates to dataYours.

I am looking to find all of the latest files to run this solution on a separate system. Can you help consolidate it? Even if that means updating the original post of this thread?

Also, can you confirm for me whether or not the .conf files will be created if I do not place them in the /www/ directory?

Thanks!
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on October 08, 2015, 05:56:13 pm
I read through all of this thread, along with a few other ones and it is obvious there has been multiple things making progress. The ability to offload onto another device, OpenLuup, and just general updates to dataYours.
Yes, there has been fairly steady progress.

Quote
I am looking to find all of the latest files to run this solution on a separate system. Can you help consolidate it? Even if that means updating the original post of this thread?
Fair point, but it's really hard to go back in time and massage every thread to make it reflect the latest - that's rather the nature of a forum. 

If you want to run DataYours, then this thread is the one: DataYours - App Store Release (http://forum.micasaverde.com/index.php/topic,30808.msg220683.html#msg220683) - the first post pointed to has a link to the latest (or thereabouts).

If you want to try running any plugin on non-Vera hardware, then openLuup is for you and this is the place: openLuup - running unmodified plugins on any machine (http://forum.micasaverde.com/index.php/topic,32315.msg235161.html#msg235161)

I will edit the first post of this thread to point to those places as well.

Quote
Also, can you confirm for me whether or not the .conf files will be created if I do not place them in the /www/ directory?
Not sure I understand the question.  DataYours (specifically, DataWatcher and DataDash) writes .conf files to /www/ (a poor choice, in retrospect, since it seems to get erased by firmware updates.)  There's little point in transferring files from an existing system since all the device numbers will be different.
Title: Re: DataYours on Raspberry Pi
Post by: airedale on October 08, 2015, 11:16:29 pm
Thanks for the feedback.

I did notice that the launch script that can be found throughout this thread is not included in your write up. Is it no longer required with the OpenLuup solution?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on October 09, 2015, 03:26:16 am
I did notice that the launch script that can be found throughout this thread is not included in your write up. Is it no longer required with the OpenLuup solution?

No.  openLuup is totally non-specific: it will run almost any plugin without modification. DataYours itself is now effectively a launcher plugin which can configure all its daemon processes through device variables settings.  Running it under openLuup is no different from running it on Vera.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 12, 2015, 02:31:20 pm
I've read the entire thread and I have a few questions. I setup OpenLuup and AltUI on my RPi2 running OpenWRT yesterday with CudaNet's instructions, and I'm trying to get datayours running on the RPi2. I tried datayours last year (on the Vera) but couldn't ever get it working properly so I uninstalled it. Given that I would need the datawatch plugin at a minimum to run this on RPi2, I reinstalled datayours on my Vera last night.

Attached is a screen shot of my RPi2 files to clarify what I've got sitting in my RPi2.

I'm running Test_Cache.lua on my RPi2, and I added a destinations entry pointing to my RPi2 on the datawatcher device on the Vera.

My problem is that when I try to access the datayours dashboard on the RPi2, I get a "no handler" error. Any ideas?


Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 12, 2015, 05:11:29 pm
I've read the entire thread and I have a few questions. I setup OpenLuup and AltUI on my RPi2 running OpenWRT yesterday with CudaNet's instructions, and I'm trying to get datayours running on the RPi2. I tried datayours last year (on the Vera) but couldn't ever get it working properly so I uninstalled it. Given that I would need the datawatch plugin at a minimum to run this on RPi2, I reinstalled datayours on my Vera last night.

This should work just fine - I run a similar configuration (with Open-WRT) on an Arduino Yun.

Quote
Attached is a screen shot of my RPi2 files to clarify what I've got sitting in my RPi2.

I'm running Test_Cache.lua on my RPi2, and I added a destinations entry pointing to my RPi2 on the datawatcher device on the Vera.

My problem is that when I try to access the datayours dashboard on the RPi2, I get a "no handler" error. Any ideas?

Ah, well I see the problem.   Test_Cache.lua is very old, and not required.  What you need is:

The standard DataYours runs unmodified on openLuup.  Get openLuup up and running first, with AltUI (looks like you might have that) and then there's one of three ways you can start up DataWatcher:


I'd actually suggest (1) and go for a full installation to test things out first on the RPi before sending data to another DataYours installed on your Vera.

So get openLuup from Github, and DataYours from http://forum.micasaverde.com/index.php/topic,30808.msg243487.html#msg243487

To create the DataYours plugin use this code (with appropriate parameter changes for your configuration) either in a startup.lua file or interactively in the Lua Test window of AltUI.
Code: [Select]

do -- DATAYOURS
  local dy7 = luup.create_device ('', '', "DataYours", "D_DataYours7.xml","I_DataYours7.xml")        -- create the device
  luup.variable_set ("urn:akbooer-com:serviceId:DataYours1", "DAEMONS", "Graph, Dash, Cache, Watcher", dy7)
  luup.variable_set ("urn:akbooer-com:serviceId:DataYours1", "LOCAL_DATA_DIR", "/nas/whisper/", dy7)
end

This is almost word-for-word what my system uses.

--------

Edit: I just created a startup.lua file with the above (and an AltUI creation) and it runs a system which looks like the screenshot attached.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 12, 2015, 06:29:28 pm
Thank you for the prompt response.

I installed OpenLuup from github yesterday (so it should be up to date) and replaced the DataYours files on my RPi2 with the ones you linked to just now (I had previously used Datayours files copied/extracted from my Vera which were downloaded yesterday from the app store and I have no idea if they were up to date)

I added the code you suggested in the startup.lua file (and stopped/restarted openLuuP) and still received the same results unfortunately.

Quote
I'd actually suggest (1) and go for a full installation to test things out first on the RPi before sending data to another DataYours installed on your Vera.

Can you explain this to me? I (perhaps mistakenly) assumed that the RPi2 would accept data from the Vera, not the other way around as you imply.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 13, 2015, 06:34:13 am
I installed OpenLuup from github yesterday (so it should be up to date)
OK, so I assume that's working just fine.

Quote
I added the code you suggested in the startup.lua file (and stopped/restarted openLuuP)
So I also assume you started a fresh system with
Code: [Select]
./openLuup_reload startup.lua

Does the running system show all the components of DataYours, as per my previous attachment?

Quote
and still received the same results unfortunately.

So what URL are you using?  It should be:
Code: [Select]
<VeraIP>:3480/data_request?id=lr_dashboard

Quote
I (perhaps mistakenly) assumed that the RPi2 would accept data from the Vera, not the other way around as you imply.

This is my misunderstanding, you had said "Given that I would need the datawatch plugin at a minimum to run this on RPi2" so I assumed that was what you were after.  However, you also said "I added a destinations entry pointing to my RPi2 on the datawatcher device on the Vera" so clearly you need DataCache as a minimum on the RPi to store the data.  I take it that you are planning on storing directly on the RPi.

Again, I do the same thing with some of my systems.  I actually have DataYours running both on openLuup and on Vera sending data to each other for storage in different places.


If all of the above is OK, but it's still giving "No handler", then I think looking at the whole log since the last reload is the next step.

Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 13, 2015, 04:30:35 pm
Akbooer, you are correct in your assumption, sorry if I wasn't clear before. The plan is to have datawatcher running on the Vera sending data to the RPi, which will run all the other daemons.

I re-read your instructions and I'm not sure what I was doing incorrectly yesterday but I got it working today. Thanks for the help!

I had another question regarding the relationship between OpenLuup w/Vera Bridge (running on my RPi) and the Vera. Why does datawatcher have to run on the Vera if OpenLuup / VeraBridge (RPi) mirrors the Vera (or at least is aware of device values).

Thanks.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 13, 2015, 04:50:22 pm
I re-read your instructions and I'm not sure what I was doing incorrectly yesterday but I got it working today. Thanks for the help!

Sometimes it just needs another day and a fresh look!

Quote
I had another question regarding the relationship between OpenLuup w/Vera Bridge (running on my RPi) and the Vera. Why does datawatcher have to run on the Vera if OpenLuup / VeraBridge (RPi) mirrors the Vera (or at least is aware of device values).

That's a really good question. DataYours was written before openLuup and VeraBridge, but it's certainly true that you could simply monitor the variables on the cloned devices.  It's arguable, but I believe that DataWatcher imposes less of a resource load on Vera than actually bridging to it.  You would also get the wrong machine ID on the files, but that hardly matters.  There are more things to break in the chain with the Bridge too.  The timing of variable changes would also be much closer with DataWatcher... The bridged solution could be up to 5 seconds late, but you might not worry about that either.  It also batches events, so sometimes event order might not be preserved.  Again, this might not matter to you if your application is not time critical, especially if you have coarser quantisation of the Whisper database archives.   

My preference would be to decouple the issues of bridging and monitoring variables, so would still go for DataWatcher on Vera, but it's your choice.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 13, 2015, 06:11:03 pm

Quote
My preference would be to decouple the issues of bridging and monitoring variables, so would still go for DataWatcher on Vera, but it's your choice.

I understand.  I'll stick with what works :) I was simply curious. Thanks for the explanation.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 14, 2015, 06:57:54 am
I'll stick with what works :) I was simply curious.

You could, of course, do both simultaneously and compare the results  ;)
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 15, 2015, 08:37:16 am
Akbooer, quick question for you..I'm using ssh w/a putty client to connect to my RPi from Windows machines. I'm finding that when I close out of my ssh client, datayours stops running for whatever reason. openLuup, I assume is still running though because I'm still able to access the AltUI interface on the RPi. Do you experience this as well on your mac?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 15, 2015, 11:50:44 am
Are you sure that openLuup is still running?  AltUI does most of its work in the browser, so it might look like it's working, at least for a while.  My best guess is that the openLuup process terminates when you log out.  Somebody has already posted a Windows script to keep it running, I recall.

Thing to do is to check the /id=alive request.  If it returns OK, then openLuup definitely is running.  The next thing to do would be to check the configuration URLs of the DataYours daemons.

I'm hard put to think of why or how DataYours would stop if openLuup is still running.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 15, 2015, 01:48:23 pm
Akboor, you were right (as usual), my mistake.

Killing the SSH session does terminate openLuup as well. I guess when I was trying earlier this morning it was pulling a cached version of the AltUI page. I'll look around for that windows script and see where that gets me. Thanks.

Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 16, 2015, 12:31:42 pm
To follow up on the datayours not running thing after closing an SSH session, I forgot that CudaNet's guide specifically included instructions that cause openluup to run after the RPi boots, so that takes care of my issue, as an SSH session isn't required to start openLuup (assuming you're using his guide, you just have to restart).
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 18, 2015, 03:11:24 pm
Given that Cache, Dash, and Graph do not need to be on the Vera now (since they are running on RPi), what would be the proper way to remove them on the Vera? Through a URL command giving the delete device argument  or simply deleting the proper files from Vera via SSH/WinSCP?
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 18, 2015, 03:22:32 pm
Given that Cache, Dash, and Graph do not need to be on the Vera now (since they are running on RPi), what would be the proper way to remove them on the Vera? Through a URL command giving the delete device argument  or simply deleting the proper files from Vera via SSH/WinSCP?

The file space that they take up is hardly worth bothering with.  I assume you are running with a DataYours parent device.  If so, then all you need to do is to change the device variable DAEMONS to simply read Watcher and reload Luup.  After reloading, the other child devices will be removed and another restart forced.  After it comes up again, you should be left with just DataYours and DataWatcher running.

If you need to gain more running memory (ie. RAM) you can run without the DataYours device, but it makes configuration a tiny more difficult (but will save you about 4 Mbytes.)

Strangely enough, I am, as we speak, working on an upgrade that reduces memory overhead even more and vastly reduces the number of files required for a DataYours installation, but that can be for a bit later.

-----------

Edit: You could additionally remove the "127.0.0.1:2003" in the DESTINATIONS variable, which will stop DataWatcher from trying to send the data to the local DataCache (which doesn't exist any more.)
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 27, 2015, 10:19:34 pm
Akboor, I hope you've had a good holiday with your family.

I understand what you mean about it not being worth it to delete the files on the Vera. Honestly I don't know how to view the free ram on the Vera. I assume the "free" command run only shows free space (similar to a hard drive) and not the free ram in the system. The reason I asked about getting rid of them is because my Vera is always a little sluggish, even though I've got about 5 or 6 devices and maybe 10 "virtual" devices (wunderground, etc).

I've got yet another question. I'm working on my Dad's RPi openLuup/datayours setup. I did his slightly different than mine and here is what I did:

1. Downloaded datayours and datayours graphite onto his Vera, and enabled the daemons required to make datayours work (his Vera runs UI7 if that's relevant). I was able to successfully access the datayours web interface on the Vera, but did not bother logging any devices since I only installed Datayours so that the watcher plugin would be installed, becuase I planned to run all the other daemons on the RPi2.

2. Installed openLuup and AltUI on his RPi2 and ran the openLuup_getfiles.lua scriptso that all the plugins were copied to the RPi2. OpenLuup/AltUI worked flawlessly on the the RPi2. I then moved onto to getting datayours installed on the RPi2.

3. I encountered problems when I tried to run the startup.lua script ("./openLuup_reload startup.lua"). If I create the datayours plugin with the code in the startup.lua script, datayours keeps requesting a reload (see attached image). Have you encountered this?


Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 28, 2015, 03:57:48 am
The first things to look at are:

This should give something to start with.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 28, 2015, 02:23:01 pm
The part of the log where it begins to loop is attached.

Version 7, downloaded from the App store on 12/25.

startup.lua is also attached.

Thanks.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 28, 2015, 05:00:26 pm
The problem is clear.  Your luup.create_device() call for DataYours should not, confusingly, be in the startup attribute definition, but further up the startup.lua file just after the similar call for VeraBridge.

What is happening is that EVERY startup, that attribute code is being run, and creating a new device which instantly wants to create some child devices wich requires another restart...
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 28, 2015, 08:01:39 pm
Thanks, that was an easy fix. I'm now getting the no handler response again on the Vera and RPi, and I'm fairly certain I know why... I have no DAEMONS variable on the Vera or RPi that is accessible. It was there previously, and I had the child devices setup properly on the Vera and Rpi, but they disappeared on the Vera, and subsequently the RPi. (see attached screen shot)



Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 29, 2015, 03:46:40 am
You're using a very old version.  You should, at least, be using this one http://forum.micasaverde.com/index.php/topic,30808.msg243487.html#msg243487

...although that's shortly to be superseded by the latest, currently in testing.
Title: Re: DataYours on Raspberry Pi
Post by: thegazelle on December 29, 2015, 08:18:30 pm
Sort of the same mistake I made last time... ;)

I replaced the files on Vera and the RPi, got the daemons variable to show up, and everything is up and running on my Dad's Vera and RPi (w/openLuup and Datayours).

Last time I started with an old DY version I found in the forums that I thought was the most current. This time I downloaded DY from the App store then used the file copy feature to replicate the necessary files to the RPi. I assumed that the Appstore version would be the most current.

Thanks for the help and your patience.
Title: Re: DataYours on Raspberry Pi
Post by: akbooer on December 30, 2015, 04:02:39 am
No, my apologies.  This is all because I haven't kept the App Store version up to date... the reason for that being that it's SO difficult.  But I now have in testing a new version with far fewer files (it does not create child devices for all the daemons) and it will form a firm foundation for future updates which will be both in the App Store and on GitHub.