The Vera Community forums have moved!

Advanced => Programming => General => Topic started by: mikee123 on April 13, 2015, 02:08:19 pm

Title: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 13, 2015, 02:08:19 pm
Not sure if this is the right part of the forum to ask. I want to extract data from a website, ultimately I am trying to extract a figure (in the red rectangle) to be used to trigger a scene in PLEG. I know how to do that, once I get a value into a variable. But, how can I get that website to feed its value into a variable ? Or is that possible at all ?
If I cannot do it in Vera, I could try to extract the variable from my Android tablet, which could then use tasker to trigger the scene, but ideally I would have it on Vera (and I have no idea how to do it on the Android tablet either).
I guess there might be a http get, but I do not understand it well enough to know how to use it.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: integlikewhoa on April 13, 2015, 05:07:39 pm
Not sure if this is the right part of the forum to ask. I want to extract data from a website, ultimately I am trying to extract a figure (in the red rectangle) to be used to trigger a scene in PLEG. I know how to do that, once I get a value into a variable. But, how can I get that website to feed its value into a variable ? Or is that possible at all ?
If I cannot do it in Vera, I could try to extract the variable from my Android tablet, which could then use tasker to trigger the scene, but ideally I would have it on Vera (and I have no idea how to do it on the Android tablet either).
I guess there might be a http get, but I do not understand it well enough to know how to use it.

This information needs to be pulled via HTTP from this site. How that can be done usually comes from and "API" from that system (if one is even available). The API will tell you want commands can and are used to send and receive requests. Alot of API's are not made public due to security concerns. You need to research more on that solar company and what it shares. It's also gonna be hard for most people to help as if they don't have access to that website they are SOL.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 13, 2015, 05:26:18 pm
Ok so I'll email them to see if they have a api. I thought (probably being naive and not really understanding it) it might be possible to give something like coordinates on the website and to send that to a variable. You can do something like that on Bloomberg or Reuters, so I thought it might be possible on any website. That does not seem to be the case. So if they do not have a api i'll probably have to give up on the idea.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: ddluk on April 13, 2015, 05:32:57 pm
From what I see from uncle Google they've got API available:

https://theowl.zendesk.com/hc/en-gb/articles/201284603-Multicast-UDP-API-Information
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: integlikewhoa on April 13, 2015, 05:40:34 pm
From what I see from uncle Google they've got API available:

https://theowl.zendesk.com/hc/en-gb/articles/201284603-Multicast-UDP-API-Information

Not sure where you got this, but this appears to be for heating/air condition controls maybe of the same brand/manufacture but I didn't see anything in that API that would be of use for SOLAR.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: ddluk on April 13, 2015, 05:59:08 pm
From what I see from github there are using exactly this way to grab data from home energy monitor devices developed by this company. I don't know how this device exactly looks like because I never use them but if this API is not available there is a mobile application available so it's possible that they've got different API for developers and mikee123 just need to ask them for access :)
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 13, 2015, 06:04:00 pm
I do not really understand this, but from what I can see it might be possible to do what I want. This website just takes readings from a clamp at the PV panel, and works out what power its generating (I have that data on Vera as I can see it on a device my Rfxtrx has created). But I think what they are saying is that this data can be pushed anywhere (somehow). It is sent to a website and I think it might be possible to grab it somehow. Rfxtrx can see and show the current power generated on a device in my UI5, what is missing is the total of the day which has been generated. So I think the answer is in that Network OWL API.pdf.. but getting it out is going to be far beyond my abilities...
But good find, that might be getting me another little step closer to what I am trying to achieve
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 06:12:47 am
Looking at the page @ddluk has posted, it seems there is a way of extracting the data I need

The Network OWL sends UDP packets to the following multicast group: Address:
224.192.32.19  Port: 22600

MULTICAST PACKETS

EXAMPLE
PACKET

<solar
id='00A0C914C851'>
<current><generating
units='w'>0.00</generating><exporting
units='w'>0.00</exporting></current>
<day><generated
units='wh'>0.00</generated><exported
units='wh'>0.00</exported></day>
</solar>

PARAMETERS

The main <solar> tag contains sections for the current solar readings (<current>) and the totals for the day (<day>). Both sections have <generated> and <exported> tags that contain the values for power generated by the PV system and power exported to the electricity grid.

So it looks to me as if the data I want can somehow be accessed
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 14, 2015, 09:02:38 am
Mike,
   You will have to write a little app that run's somewhere on your LAN (does not have to be Vera) that
listens for the UDP message receives it, extracts the data, and then sends the HTTP request to Vera to set some device variable.
Then this code needs to be launched ... to make sure it's running all the time. It might also need to be throttled ... I am not sure how often the UDP packets are broadcast.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: Z-Waver on April 14, 2015, 09:26:05 am
This topic should likely be moved under the programming sub. What you are trying to do is extract a "field"(it may not be adequately marked) from a webpage and stuff it into a variable.

The Multicast API information is for receiving data directly from the Owl device, not the website/page. In order to receive the information from the device, you would need to setup a receiver for the multicast packets and then a handler for the data. This is a good candidate for a Raspberry Pi with some Python, but that would be massive overkill to extract a single variable to then pass it on to Vera. It might be possible to setup such a receiver on Vera itself, but it would require a rather advanced plugin to do it and I doubt that this is the way you want to go.

It is likely possible for Lua code to scrape the webpage you posted and extract the desired information. However, it would not be trivial. By this I mean that it is unlikely that anyone here is going  to be willing to spend the time writing it for you, especially in the most painful, five forum posts per line of code way. Here is a link to a Lua webpage scraper (https://github.com/1nv1/grab4eda) if you want to see what's involved and have a sample to start from.

It would be far easier if Owl provided a JSON file with the data on their site, but you'd have to ask if they do that. It's a possibility. Even with a JSON file, you'll still be looking at a somewhat complex bit of Lua.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 09:28:01 am
hmmm that would mean having to run a pc all the time, I don't really want to do that. Could that be an app running on a Android tablet ? I have never written an app, so not sure if its even worth me trying but if it would be possible I would look into it. I have 2 Android tablets wall mounted which are connected to my LAN, and they are running tasker anyway to interface with Vera, Vera Alerts, Autovera and Authomation. I wonder if there is maybe a plugin for tasker I could utilise...
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 09:30:08 am
Yes that sounds like another idea (the web scraper) I will look into. - had a look, probably a non starter for me as I do not really understand it
enough..

Quote
you would need to setup a receiver
Not sure if that's what you mean, but I have a receiver, my Rfxtrx, that already receives the current output and displays it in a device ?


How can I (or a moderator maybe ?) move this thread to programming ?  I had a look I don't think I can move it
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: Z-Waver on April 14, 2015, 09:48:44 am
Not sure if that's what you mean, but I have a receiver, my Rfxtrx, that already receives the current output and displays it in a device ?
I was referring to a process that listens to the network for multicast packets.

But, If you already receive the information in a Vera device, why are you bothering with scraping a webpage? You should be able to get the variable that you want from the Vera device quote easily. It may already be available to PLEG as a property.


Quote
How can I (or a moderator maybe ?) move this thread to programming ?  I had a look I don't think I can move it
A moderator would have to move it. Don't worry about it, if it doesn't happen.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 10:01:55 am
I have the current output of my PV panels in the device, but not any other information. So what I need is missing. But, I could well be that it is received by the Rfxtrx but just not displayed ? I am quite sure that is the case, as its wireless transmitted to a receiver, which in turn sends it to the website. And the Rfxtrx is receiving that information. Maybe there is a way to get that information. It does not necessarily need to be displayed, all I need is the value of the total produced for the day

I think I might post a question about the Rfxtrx in that part of the forum. That might be another avenue worth pursuing.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 14, 2015, 12:01:16 pm
Quote
It does not necessarily need to be displayed, all I need is the value of the total produced for the day

If that's all you need then you should be able to calculate that inside PLEG. You may need to add a device variable (or two) to the PV where you can save the previous running totals. You can reset your daily total(s) using a PLEG Schedule at, say, 00:01:00.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: integlikewhoa on April 14, 2015, 12:57:11 pm
Thats what im thinking and sounds like what he said that page was doing. He just needs to tally up the totals with the fata hes already getting.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 01:58:28 pm
Ok theoretically that sounds easy enough. I am quite comfortable with PLEG, but not sure how I would go about it. That value can change every second  (the watts produced figure). I can get that into PLEG as a device property I guess. Setting up a schedule is easy enough. I only need to run it from 6am to lets say 11pm every day. But now the difficult part. I will probably have to take a reading every second. Thats still easy with a schedule I think. But how do I add them up to end with kWh produced ?
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 14, 2015, 02:22:31 pm
Does the device have any other variables apart from instantaneous power? Often they have a running total as well.

If you only have the current power, you could run it every second but is that really necessary? There would be much less load on Vera if you updated it less frequently - maybe every ten seconds? Just how accurate do you need this value?

However often you do it, the calculation is quite simple. KWh = KWh + ((Watts /1000) * period (seconds) / 3600).

It would be possible to do this using a Conditional expression but I would suggest using a device variable and performing the calculation in Lua attached to a PLEG Action.

Edit: 14/04/2015 21:50 Corrected for KWh instead of Wh.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 02:42:40 pm
I had a look under advanced, there is a history where I can see the last values, but no total unfortunately. I do not need it that accurate. Its only when its above 10kWh I want to trigger a scene to stop my hot water being heated by gas as its ben heated enough by the solar panels.

Ok next point performing it in Lua. From what I read its the most efficient way. So I would run a action every 10 seconds which calculates the formula and gives me a running total of kWh. How do I do that in Lua ? And then the next morning I have to reset kWh to 0. I could do that with another Lua action ?
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 14, 2015, 02:50:21 pm
To avoid a lot of polling, it may be worth exploiting the fact that PLEG will be notified whenever the power changes. This is untested but similar to something I use.

Schedules
sMidNight     On: Weekly, all days, 00:01:00    Off: Interval, 00:00:05

Device Properties
pPower     Current power variable in your device

Conditions
chgPower     lastPower; pPower
KWh     sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh )
lastPower     pPower

BTW: When you first create this in PLEG you will get a crazy number for KWh as the timestamp for lastPower will be zero. It should become a real value after the first reset happens. You could preempt that using the Do it now button for sMidNight.

Edit: 14/04/2015 21:51  Corrected for KWh instead of Wh.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 14, 2015, 05:32:50 pm
Yes i am pretty sure i understand the logic. I think that could work. It's a bit late for me now but I'll program that in tomorrow and see how it goes. I could use Variable container or something like that to see the value of the variable ... just to keep an eye for a few days and compare to the website and possibly extend the interval to reduce cpu usage.
One idea I had. Is there a way i could display that variable in my device ? The kWh in the device is blank, so maybe it could be diplayed there ? I would probably need @lolodomo to tell me which variable is being displayed in that device, then write kWh to that variable ?
Just an idea...
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 14, 2015, 06:16:19 pm
Quote
One idea I had. Is there a way i could display that variable in my device ? The kWh in the device is blank, so maybe it could be diplayed there ?

Yes - just set an Action for the Condition KWh that uses PLEG's SetVariable action to write the required variable. You would need to set Repeat on for KWh.

Quote
...and possibly extend the interval to reduce cpu usage.

The logic will update KWh whenever it gets a new power value. If you want to reduce the frequency you would need to change it a little. The following change should prevent updates faster than once per 30 seconds:

Conditions
chgPower     lastPower; pPower > 29
KWh     sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh )
lastPower     chgPower? pPower : lastPower

You could also limit updates by how much the power changes. This should only update if the power changes by at least 10%:

Conditions
chgPower     ((pPower-lastPower) >= (lastPower/10)) or ((lastPower-pPower) >= (lastPower/10))
KWh     sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh )
lastPower     chgPower? pPower : lastPower
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 03:56:04 am
This is what I've done so far:

Schedules
sMidNight Weekly 05:00:00 1,2,3,4,5,6,7 None Interval 00:00:05  None false 2015-04-15 07:55:15.182  2015-04-15 07:55:20.100 

Device Properties
pPower Power Solar Panels Watts 322 2015-04-15 08:43:21.288  2015-04-15 08:39:21.339 

Conditions
chgPower No lastPower; pPower true 2015-04-15 08:28:21.183  2015-04-15 08:21:35.355 
KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh ) 0.076620324308276 2015-04-15 08:43:21.332  2015-04-15 08:39:21.391 
lastPower No pPower 322 2015-04-15 08:43:21.344  2015-04-15 08:39:21.403 

I have also attached the full status but I think its easier to show just the relevant bits here as its a big PLEG.

I have started with the do it now on the schedule. After that I watched my KWh in ther status. That should increase with every change. It doesn't so something is slightly wrong I think.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 04:07:27 am
Try changing to:

lastPower     chgPower? pPower : lastPower
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 04:17:39 am
Also try:

chgPower     pPower != lastPower
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 04:43:00 am
Something is still not quite right. The number in KWh is changing, but it is going negative.

KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh ) -3.3149971883759

Also there must be something else slightly wrong with how we calculate KWh. I have been monitoring the changes and compared to what I can see on the website

KWh      Owl website

-2.254        0.7
-2.309        0.8
-2.944        0.8
-3.31          0.9
-4.5            1.0

I have changed lastPower to chgPower? pPower : lastPower, the figures above are from after the change
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 04:55:10 am
Quote
I have changed lastPower to chgPower? pPower : lastPower, the figures above are from after the change

Did you also change:

chgPower     pPower != lastPower
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 05:03:44 am
No I didn't see it. Changed it now, fingers crossed...  and I got the Multistring working now as well displaying the Value. Next project is to have it displayed in the PV device

Many thanks for all your work !
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 05:29:45 am
There is still a gremlin somewhere in the calculations..

KWh    OWL

-5.94     1.3
-6.7       1.5
-6.77      1.6
-7.07      1.7
-10.4      1.9

chgPower No pPower != lastPower true 2015-04-15 10:16:20.108  2015-04-15 10:15:53.652 
KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh ) -7.5627948234535 2015-04-15 10:29:20.406  2015-04-15 10:29:08.374 
lastPower No chgPower? pPower : lastPower 1288 2015-04-15 10:29:20.438  2015-04-15 10:29:08.387 
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 07:52:59 am
I think I might have found a glitch.

KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh ) -0.31325140817655 2015-04-15 12:44:55.844  2015-04-15 12:44:49.143 
lastPower No chgPower? pPower : lastPower 2392 2015-04-15 12:44:55.856  2015-04-15 12:44:43.863 
lastpowertime No (#chgPower - #lastPower) -254.7188000679 2015-04-15 12:45:43.456  2015-04-15 12:44:55.868 
kwhcalc No (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600))  -0.48249789977722 2015-04-15 12:45:43.468  2015-04-15 12:44:55.883 

I had reset sMidNight to start with 0.
(#chgPower - #lastPower) has gone negative which in turn causes KWh to go negative. I will try to change to (#lastPower - #chgPower)

Not sure how that will work but I am trying that now
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 08:34:13 am
That was one problem. What is happening though lastPower timestamp changes to the current time, and chgPower doesn't. Next time this happens this interval gets longer, and added again to KWh. So KWh increases too much. Example with a snapshot a few minutes apart:


chgPower No pPower != lastPower true 2015-04-15 13:26:15.130  2015-04-15 13:25:34.793 
KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#lastPower - #chgPower) / 3600)) : KWh ) 18.423663961577 2015-04-15 13:26:15.167  2015-04-15 13:24:32.277 
lastPower No chgPower? pPower : lastPower 2484 2015-04-15 13:26:15.188  2015-04-15 13:24:32.305 
lastpowertime No (#lastPower - #chgPower) 0.05768609046936 2015-04-15 13:26:15.210  2015-04-15 13:25:34.803 
kwhcalc No (KWh + ((lastPower/1000) * (#lastPower - #chgPower) / 3600))  18.42370376498 2015-04-15 13:26:15.227  2015-04-15 13:25:34.817 

and a few mins later:


chgPower No pPower != lastPower true 2015-04-15 13:26:15.130  2015-04-15 13:25:34.793 
KWh No sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#lastPower - #chgPower) / 3600)) : KWh ) 19.402439199988 2015-04-15 13:32:20.295  2015-04-15 13:31:20.709 
lastPower No chgPower? pPower : lastPower 2530 2015-04-15 13:32:20.309  2015-04-15 13:31:20.720 
lastpowertime No (#lastPower - #chgPower) 365.17838406563 2015-04-15 13:32:20.321  2015-04-15 13:31:20.732 
kwhcalc No (KWh + ((lastPower/1000) * (#lastPower - #chgPower) / 3600))  19.659078453234 2015-04-15 13:32:20.336  2015-04-15 13:31:20.747 

I think the problem is when PLEG updates without a change in Power. Then KWh gets updated, with the timestamp of chgPower not being changed. If PLEG would only update KWh when chgPower is triggered, it would work I think. So I could try moving this logic to a dedicated PLEG. But I think that's overkill and I would have to get another license... there must be another way but I cannot think of one at the moment...

I have tried moving the logic to a deditacted PLEG. The problem is still the same. PLEG status attached
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 11:19:33 am
Try this slightly simpler version:

KWh     sMidNight ? 0 : ((pPower != lastPower) ? (KWh + ((lastPower/1000) * (#pPower - #lastPower) / 3600)) : KWh )
lastPower     pPower

Otherwise we may have to go back to Lua...
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 15, 2015, 11:43:36 am
I am planning to modify PLEG to be able to run actions from INPUT changes (triggers, schedules, device properties) as well as conditions.

In this case you can put this type of code in the LUA action for the INPUT property. From the LUA code you can write another property, possibly one created on the PLEG device itself.
That property can be used in other PLEG logic ... via a Property input.

Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 12:51:26 pm
In case you need it, and I am not online, here's some Lua to try.

Device Properties
pPower     Current power reading from solar unit

Conditions
updkWh   [Repeat=Y]   pPower

Actions
updkWh   <Lua code>

Code: [Select]
local sID = <Service ID of power device>
local dID = <Device number of power device>
local lastp, tlastp = luup.variable_get(sID,"LastPower",dID)
luup.variable_set(sID,"LastPower",PPOWER.state,dID)
if (tlastp or 0) ~= 0 then
local kWh = tonumber((luup.variable_get(sID,"KWH",dID))) or 0
kWh = kWh + ((tonumber(lastp) or 0) / 1000) * (os.time() - tlastp) / 3600
luup.variable_set(sID,"KWH",kWh,dID)
end

You need to set appropriate values for sID and dID to suit your device. The code creates new variables LastPower and KWH using these. You could place them in another device or use MultiString variables though.

The daily reset can be done using PLEG's SetVariable action and writing zero to KWH.

Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 15, 2015, 03:03:49 pm
Thanks i might give the Lua code a try. At the moment things look good with the last modification. I only tested for the rest of the afternoon, and the result was close enough for what I want it to do. But I'll see over a full day tomorrow.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 15, 2015, 04:34:54 pm
I am planning to modify PLEG to be able to run actions from INPUT changes (triggers, schedules, device properties) as well as conditions.

In this case you can put this type of code in the LUA action for the INPUT property. From the LUA code you can write another property, possibly one created on the PLEG device itself.
That property can be used in other PLEG logic ... via a Property input.

That sounds useful, Richard. I shall be interested to see how you manage the timing. If a change in an input property causes Lua code to be run which changes another property, can that initiate the evaluation of Conditions that use it?
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 15, 2015, 07:34:39 pm
It does right now ... but that will be a separate evaluation ...
i.e. if LUA (or an Action) does a SetVariable ... the variable will not effect subsequent condition evaluations in the set of conditions.

A separate evaluation because the variable was changed will happen and ALL of the conditions will be re-evaluated with the variable change.

Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 16, 2015, 10:36:30 am
I am struggling to get my Power device to display the calculated value. I have tried endless versions, nothing works, @korttoma has tried to help me in another thread but its not working. I have not tried using the Lua version above, I am not really sure about what sid to use and the PLEG version is working well enough.

The calculated (PLEG condition which has the value to be displayed) variable is KWh
The Power device where I want to display it is device id 379
The Power device variable I want to use is KWH

I tried this (and countless other versions) in the luup section of a PLEG action

local Value = KWh
luup.variable_set("urn:micasaverde-com:serviceId:EnergyMeter1", "KWH",KWh,379)
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 16, 2015, 12:18:30 pm
First you need to find-out for sure the ServiceId and name of the variable displayed by the power device. If the variable is shown on the device's Advanced tab, hovering your mouse over the variable name will show you the ServiceId. If it isn't shown, you will need to inspect the device's json file (or post it here).

Once you have that information, you could use PLEG's SetVariable action to update the variable.

If you use Lua in PLEG, you need to use special syntax to access Inputs and Conditions. For your example, you would need to use:

Code: [Select]
luup.variable_set("urn:micasaverde-com:serviceId:EnergyMeter1", "KWH",KWH.state,379)
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 16, 2015, 04:51:25 pm
I tried to use PLEG set variable. I used this:

      
ServiceId      urn:micasaverde-com:serviceId:EnergyMeter1
VariableName      KWH
Value      {(KWh)}
Device             379

that did not work. I have the .json attached. From what I can see (but I never looked at a .json before) that looked right to me...

I have just seen, KWH does now show up in advanced (It didnt before), with the right value. But its not shown on the device. The KWH is there as before (see previous attached Power Solar Panels pic), but still no value next to it.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 16, 2015, 05:56:05 pm
Mike I looked at the JSON you provided.

The ServiceId is:  "urn:micasaverde-com:serviceId:EnergyMetering1
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 17, 2015, 02:49:12 am
What a silly mistake. I should have looked at it properly. Its working now. One thing I still would like to fix, the value displayed is with a huge amount of digits. Is there a way to round this ? Either in PLEG in a condition, or in the device itself ? I would like it to have max 2 decimal places. In multistring I can round, but looking through the PLEG manual I cannot see a possibility to do that. If there is a integer fuction, I could multiply by 100, take the interger and then devide by 100 again... but I have not seen that function
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 17, 2015, 02:59:24 am
You need to keep full precision in the PLEG Condition otherwise your results will not be accurate. It would be better to apply rounding before display using Lua. So replace your current PLEG action with this:

Code: [Select]
local kWhR = string.format("%0.2f", KWH.state)
luup.variable_set("urn:micasaverde-com:serviceId:EnergyMetering1", "KWH",kWhR,379)
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 17, 2015, 03:16:09 am
That worked. I was thinking of a separate rounding condition in PLEG for exactly the reason you stated. Thanks everybody for all your help. And I learnt a few things along the way as well...
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 17, 2015, 03:31:32 am
That worked. I was thinking of a separate rounding condition in PLEG for exactly the reason you stated. Thanks everybody for all your help. And I learnt a few things along the way as well...

I'm glad you got what you wanted. You can round down in PLEG Conditions by using the Modulus operator (%) but it gets a bit messy:

KWhR     ((KWh * 100) - ((KWh * 100) % 1)) / 100


Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 17, 2015, 03:48:38 am
I was looking at the modulus operator, But I thought it only gives me the fraction. In your formula, what is % 1 doing ? From the PLEG manual I wasn't sure how to use it, and before I could try you came up with the Lua solution.. but for future reference I would like to understand the formula
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 17, 2015, 04:51:40 pm
X % 1 gives the remainder of dividing X by 1 - so the fractional part. X - (X % 1) is therefore the integer part of X. You can use it for rounding if you add 0.5: (X + 0.5) - ((X + 0.5) % 1).

You can also use it with a decimal modulus so you could get your two decimal places using:

(X + 0.005) - ((X + 0.005) % 0.01)
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 07:43:22 am
I tried to add another logic to the calculation. What I want to do is add up KWh, but only if it is over 924 W, and I only want to add up anything above 924 W. I thought I have the solution, but I get a error, attempt to perform arithmetic on a nil value

This is the condition in question:

KWh2 No sMidNight ? 0 : ( (pPower != lastPower) ? ((lastPower > 924) ? (KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600))) : KWh2 ) null 2015-04-20 11:24:45.662  2015-04-20 12:29:51.380 
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 20, 2015, 09:22:24 am
I count 3  "?" characters and only two ":" characters ... so something is not fully defined!

Another approach:
   Create a condition:
     PowerChanged                pPower

Use LUA to do your math in this condition ... and write to a KWH2 variable.
If you also need that for other logic in PLEG than make that variable an input property.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 09:48:27 am
I am not really comfortable with Lua...  I changed the condition a few times, still the same error.
This is my latest version (not functioning). I thought I could use the ... : KWh2 as a 'no' for both cases. Obviously not. So I tried to have a : KHh2 for each , but somehow it doesn't work


KWh2 No sMidNight ? 0 : ( (pPower != lastPower) ? ((lastPower > 924) ? (KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2) : KWh2 ) null
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 20, 2015, 10:16:13 am
I looked at your REPORT ... You need to move the KWH2 condition UP to be executed before the  lastPower condition!

Otherwise  pPower will always be equal to lastPower!
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 10:43:34 am
I always get caught out with that. But the error is still there. I tried these 2 versions:

KWh2  sMidNight ? 0 : ( (pPower != lastPower) ? ((lastPower > 924) ? (KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2) : KWh2 )

KWh2  sMidNight ? 0 : ((pPower != lastPower) ? ((lastPower > 924) ? ((KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2)) : KWh2 )
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 20, 2015, 01:26:50 pm
I think you are getting the error because KWh2 is currently nil so PLEG will not add to it. This will get fixed when sMidNight fires. You could also fix it by temporarily changing the Condition to force it to have a value of 0.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 02:32:35 pm
There must be something else wrong. I have tried firing sMidNight with do it now, but i cannot even do that, I get device communication failure. Also that PLEG has a red banner on the bottom Lua failure. If I change the condition to somethin easy like KWH2  KWh, that all disappears. So something in my condition is wrong. I just tried to expand on your condition which is working fine. This is the original (your version)

KWh  sMidNight ? 0 : ((pPower != lastPower) ? (KWh + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)) : KWh )

I thought the easiest way to add another condition which counts everything above 923 Watts, and adds that to KWh2 was the way I tried... I must admitt I have never before used the ?, but I thought that was a very good way. But somewhere I have a mistake in my logic.

Testing a few different things, somehow it has something to do wit KWh2 being 0. But it worked on KWh without a problem...

I tried this (only to try and narrow down wheer the error is coming from):

KWh2   No   lastPower > 923 ? (KWh2+ ((lastPower-923)/1005) * (#pPower - #lastPower) / 3600) : KWh

and got the error. I changed it to this:

KWh2   No   lastPower > 923 ? (1 + ((lastPower-923)/1005) * (#pPower - #lastPower) / 3600) : KWh

and it works. The problem is, even if I wait until after midnight, sMidNight will not trigger as the PLEG is not running due to the error
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RexBeckett on April 20, 2015, 03:51:31 pm
Now that KWh2 has a non-nil value, you should be able to restore the expression.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 05:22:00 pm
Nope. I get a boolean error with this one

KWh2  sMidNight ? 0 : ((pPower != lastPower) ? ((lastPower > 924) ? ((KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2)) : KWh2 )

Then i tried this:

sMidNight ? 0 : ( (pPower != lastPower) ? ((lastPower > 924) ? (1+ (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2) : KWh2 )

That worked without errors, so I replaced the 1 with KWh2 and got the attempt to perform arithmatic on nil value again...

sMidNight ? 0 : ( (pPower != lastPower) ? ((lastPower > 924) ? (KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : KWh2) : KWh2 )

It cannot be that difficult, can it ??? It was working straight away with your first formula. Why is this causing so much trouble ? I have to say I am close to asking how to do this in Lua, it would probably be easier. But just from a point of having it all in PLEG as conditions under each other and if changes need to be made, its a lot easier to see whats going on. And I do not like giving up... It does really sound and look as if the KWh2 starting value is causing a problem. But as that boolean error on teh first one I tried came up, there is obviously also an error in my logic. That did not happen on the last one. But maybe that is another reason for it to fail ?
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 20, 2015, 05:27:00 pm
Look at this. I have just tried this:

KWh   No   sMidNight ? 0 : ((pPower != lastPower) ? (KWh + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)) : KWh )   0.16604957553529   2015-04-20 22:22:10.856    2015-04-20 22:19:58.900

KWh2   No   sMidNight ? 0 : ((pPower != lastPower) ? (KWh2 + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)) : KWh2 )   null   2015-04-20 22:02:45.045    2015-04-20 22:04:04.415

same formula, bottom one produces an error. Now I thought maybe its the name... so I changed it

KWh   No   sMidNight ? 0 : ((pPower != lastPower) ? (KWh + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)) : KWh )   0.16628052599776   2015-04-20 22:25:11.920    2015-04-20 22:23:22.831

Immersun   No   sMidNight ? 0 : ((pPower != lastPower) ? (Immersun + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)) : Immersun )   0.00008124469030

but if you thought now changing my formula in the same way would work, it doesnt. Its either boolean error or divided by nil value still. I'll give up for tonight.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: RichardTSchaefer on April 20, 2015, 07:27:25 pm
Use two expressions (order matters!!!!!)

newValue    KWh2 + (((lastPower-23)/1005) * (#pPower - #lastPower) / 3600)
KWH2         sMidNight ? 0 : ((pPower != lastPower) ? newValue: KWh2 )

Now to initialize ... Run your sMidNight schedule ... it should set KWH2 to zero. It was failing before because newValue will continue to get an error until KWH2 gets a non null value. By breaking this into two conditions ... it does not cause the KWH2 to get a math error during it's evaluation when KWH2 is currently null.
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 21, 2015, 02:55:00 am
I thought of splitting but could not find a solution. I only want to add up values above 923 Watts. If I adapt the newValue to

newValue    KWh2 + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)

newValue will add up negative values while lastPower is below 923. So I would need a if lastPower > 923 then ... I couldn't work out how to, I am having a mental block
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: mikee123 on April 21, 2015, 06:11:42 am
I think I might have fixed it. I changed the condition to:

sMidNight ? 0 : ((pPower != lastPower) ? ((lastPower > 924) ? (1 + Immersun + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : Immersun) : Immersun )

then after a while that lastPower was above 923, I changed it back to

sMidNight ? 0 : ((pPower != lastPower) ? ((lastPower > 924) ? (Immersun + (((lastPower-923)/1005) * (#pPower - #lastPower) / 3600)) : Immersun) : Immersun )

obviously the Immersun value now iss wrong as its been adding up 1+ ..., but no errors. The value should be ok after sMidNight has run and reset it to 0 I hope...
Title: Re: Can I extract data from a website to use in PLEG (as a variable)
Post by: dmckenna on April 23, 2015, 09:28:43 am
X % 1 gives the remainder of dividing X by 1 - so the fractional part. X - (X % 1) is therefore the integer part of X. You can use it for rounding if you add 0.5: (X + 0.5) - ((X + 0.5) % 1).

You can also use it with a decimal modulus so you could get your two decimal places using:

(X + 0.005) - ((X + 0.005) % 0.01)

Thanks Rex - you're the best.......