We have moved at community.getvera.com

Author Topic: New dataMine graphing and logging plugin  (Read 245148 times)

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #30 on: September 21, 2011, 10:30:08 am »
Quote
I have two TED 5000 Devices that are updating OK in the main UI4
I assume they have different names? As you've probably noticed, dataMine uses the device name and variable name to make up the filename. If the devices had the same name, this may cause problems (I think you would end up with a single file with both lots of data).

Quote
It took several tries to get the variable to stay green in the Config
Strange. When you click on a variable name, it sends a message back to dataMine, and dataMine should update an internal table, and rewrite the dataMineConfig.json. If it does this (actually, just the updating of the internal table), then it should stay green. I think there may still be a bug which could cause a problem if the config file was either not initially present, or corrupted - this has been fixed (I really should upload an update). I don't know if this is the problem you've seen though...


Quote
Can I just delete a .txt file for a variable without messing everything else up?
Yes. The generation of the txt files is completely independant. Each time the variables change, Vera sends a message which dataMine receives and logs the data to a file. Each time dataMine logs the data, it opens the file, writes the data, and closes the file. So, the file will be generated at that point.
However, this won't change the logging - the configuration is stored in the dataMineConfig.json file - this file stores the variable names, file names, status (logging 'green' or 'yellow'). If you download this file, you can make it look pretty by putting it into www.jsonlint.com.

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #31 on: September 30, 2011, 06:23:17 pm »
I've added a few features to dataMine. There have been a few fixes/additions under the hood, but the main changes are -:
  • Multiple channels are now graphable - currently up to 4 channels are supported.
  • Graphs are now in local time
  • A legend has been added to the graph. This can be disabled using a button on the toolbar
  • You can disable mouse tracking using a button on the toolbar

The system allows you to select multiple channels and place them onto the left or right Y axis to allow for channels with different scales. This is shown in the attachement. When you click on the "Select Channels" button, there are four fields of interest on each line. First, there's an arrow button. Pressing this will instantly graph this channel with no further action required. Next is the variable name. Clicking on this selects the channel, and enables two further buttons. The first button on the right is a "go" button - it will display all variables currently selected. The second button allows you to select two the left or right axis. This is extremely useful if you have different types of data (say, temperature and humidity) which would otherwise not graph well on the same scale.

It may take a little bit of getting used to, but it's a lot more powerful than a simple dropdown list! I'm  open to suggestions on improving this if it's deemed to be too complex.

If you're upgrading, note that there are a number of new PNG files to support the new buttons and features of the software, so please ensure you upload these as per the installation instructions. The LUA, JSON, XML and JS files also need to be updated via the "MiOS Developers" button.

A wiki has also been started at http://code.mios.com/trac/mios_datamine.

Offline Jackpost

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #32 on: October 11, 2011, 10:59:35 am »
Hi Chris,

Could it be that the dataMineConfig.json file is missing in your ZIP file?

I installed dataMine, but I can't get it to work, reading trough the posts I noticed you were referring to dataMineConfig.json file, and that it should be present in /dataMine, but I cant find it in your ZIP...

Thanks,

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #33 on: October 15, 2011, 05:19:17 pm »
Sorry for the slow response - I'm travelling in New Zealand at the moment...

You shouldn't need to have the json file now - dataMine should create it for you when you go into the config page and select variables to log. I did include a copy of the json file in an early zip file, but haven't bothered recently.

If you can tell me what's happening, and also take a look in the vera log file for and "dataMine" entries and I'll take a look into the problem. I won't be back home though until the 25th, so won't be able to do a lot until then, but I will be checking the forum semi regularly over the next week.

Cheers
Chris

Offline Jackpost

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #34 on: October 16, 2011, 02:08:03 am »
No problem Chris :)

I have installed datamine but I can't get it to work. I ran trough the instructions several times, deleted the created device and started the installation again, but can't find anything I missed...

I can create the device, but it does not start as it should, see pictures attached.

I am running 1338, but that should not be a problem, as it is referenced in this post.

I also noticed that the /etc/init.d/custom-user-startup file does not exist in my system.
Does that point to something I should be checking?

Thanks,

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #35 on: October 16, 2011, 05:19:43 pm »
Quote
I can create the device, but it does not start as it should, see pictures attached.
The first thing I noticed is that the icon is a default icon - not the graph icon that should be displaying in dataMine. This could mean that there are files missing from the installation, or that the I_DataMine1.XML file isn't specified in the UpnpImplFilename in the configuration (the picture of your config doesn't show this).

It would be worth looking in the log file to see if there are any "dataMine" entries being logged. dataMine logs a status message on startup. You could also look further down the config screen to see if dataMine has created any variables - it should create one for the data directory - this will tell you if the software has been running at all.

Quote
I also noticed that the /etc/init.d/custom-user-startup file does not exist in my system.
You don't necessarily need this file - I use it to mount the USB drive when Vera starts. I couldn't get the Vera option of using a USB stick for the log files to work, so I mounted a USB stick outside of the Vera software, and pointed my /dataMine directory into the USB stick. I think by default this file doesn't exist, and you should create it with the mount command (as per the info on the wiki) to remount the USB drive when Vera starts.

However, if you have simply created a directory in the main filesystem, and aren't using a USB stick, then you don't need the startup file as won't be mounting a USB stick. I don't recommend this as dataMine will take a lot of disk space. It also results in writing to the Vera flash memory, which has limited write cycles available. You can use it like this for short term testing, but not long term.

So, I would -:
  • Look in the log file to see if there are any dataMine entries.
  • Check you have the implementation file loaded (through the developers menu), and configured.
  • Check to see if there is a diretrory variable toward the bottom of the config screen
« Last Edit: October 16, 2011, 05:25:10 pm by Chris »

Offline Jackpost

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #36 on: October 17, 2011, 12:39:48 am »
Thanks for your help Chris,

•Look in the log file to see if there are any dataMine entries.
No entries found

•Check you have the implementation file loaded (through the developers menu), and configured.
I_DataMine1.XML file has been loaded and is specified in the UpnpImplFilename

•Check to see if there is a diretrory variable toward the bottom of the config screen
No variables have appeared, see screen attached

A USB stick has been mounted. I created the custom-user-startup file manually, but it did not mount the USB after a reboot. As you said this is only needed in case I need to reboot, I hoped it pointed to some package I missing on my system that is required to run datamine...

Where to look next?

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #37 on: October 17, 2011, 01:16:35 am »
I've just noticed one thing that could be the problem - you have the device_type incorrect. This is my fault - I've specified it incorrectly on the wiki page (I'll change it shortly).

device_type should be urn:schemas-cd-jackson-com:device:DataMine:1 - I've specified servicId.

I'm not 100% sure that this will resolve the issue as there are a few things I can see that I think should work, but I'm not really sure how Vera works under the hood so let's see how it goes.

Offline Jackpost

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #38 on: October 18, 2011, 03:18:42 pm »
The device type correction gave some progress...datamine is starting...but something is still wrong....

- Vera keeps saying;  Running Lua Startup
- The Setup Tab gives me a list of devices and their variables, and I can select them (they become green), but when I go to a different ab, and then return to the Setup Tab, they are not green green any longer. Is this as intented or should a previously selected line still appear green?
- Datamine is logging data in folder /Datamine
- When selecting the Select Channels button on the Graph Tab, an empty list is displayed
- The  L_DataMine1.lua file shows as only file in the /overlay/etc/cmh-ludl folder without .lzo extionsion, no clue if this could be related to the Running Lua Startup message.

What do you think?

Thanks,

Offline Jackpost

  • Sr. Newbie
  • *
  • Posts: 46
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #39 on: October 19, 2011, 02:21:02 am »
More news this morning...

The Select Channels drop down list has been populated with selected variables (did took a couple of hours on my system) and datamine produces a graph, but only for 1 value. Displaying 2 channels on 1 or 2 axles does not seems to be working.

Vera still is continiously notifying that it is Running Lua Startup
 
Selected values in the Setup Tab now remain green once selected. When unselecting they turn yellow.

Is there a way to remove variables from the channels drop down list?

Thanks

Offline readw

  • Sr. Newbie
  • *
  • Posts: 33
  • Karma: +0/-0
Re: New dataMine graphing and logging plugin
« Reply #40 on: October 21, 2011, 06:14:00 am »
Hi All,

I am very unfamiliar with openwrt.

In the install instruction it advises that,

md /dataMine
mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine


If I am logged onto the root directory of vera and try the command md /dataMine i get the following error

-ash: md: not found

What am I doing wrong?

thanks

Warren

Offline futzle

  • Beta Testers
  • Master Member
  • *****
  • Posts: 3260
  • Karma: +192/-9
Re: New dataMine graphing and logging plugin
« Reply #41 on: October 21, 2011, 07:09:58 am »
Instead of md use mkdir

Offline Ap15e

  • Beta Testers
  • Hero Member
  • *****
  • Posts: 1998
  • Karma: +12/-0
Re: New dataMine graphing and logging plugin
« Reply #42 on: October 30, 2011, 06:53:55 am »
dataMine is failing after changing some settings via Config tab:

Code: [Select]
10/30/11 11:49:06.837   ^[[31;1mluup_log:42: GoogleWeather: Google Weather #42 starting up with id ^[[0m <0x402>  10/30/11 11:49:06.940   luup_log:47: dataMine: Initialising Data Mining System <0x402>
10/30/11 11:49:06.986   ^[[31;1mLuaInterface::CallFunction_Startup-1 device 47 function initialise failed /usr/lib/lua/json.lua:122: Unterminated JSON
10/30/11 11:49:06.987   ^[[31;1mLuImplementation::StartLua running startup code for 47 I_DataMine1.xml failed^[[0m <0x402>
« Last Edit: October 30, 2011, 06:56:30 am by Ap15e »

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #43 on: October 31, 2011, 09:49:25 am »
Sorry for the delay in responding to this thread - I've been "down under" in New Zealand and have just recovered from the trip...

Quote
dataMine is failing after changing some settings via Config tab
I've seen this happen too - when it happened to me, it was caused by the json file being written incorrectly. I use a JSON library to turn the Lua tables into json to save the config, and if something bad happens here, then you end up with this. After it happened to me, I planned to rename the old version of the dataMineConfig.json rather than just overwrite it, but I don't recall if I have completed this or not. Take a look in the /dataMine directory to see if there's a backup, or take a look in the json file to see if it's correctly formatted (or indeed, if it has anything in it).

I'll take a look at trying to better handle this, although it's unclear to me what I can do at the moment (other than keep a backup of the config).

Offline Chris

  • Moderator
  • Hero Member
  • *****
  • Posts: 821
  • Karma: +3/-0
Re: New dataMine graphing and logging plugin
« Reply #44 on: October 31, 2011, 09:53:38 am »
@Jackpost - are you still experiencing issues with the "Running Lua Startup" message and getting multiple graphs plotted?

Quote
Is there a way to remove variables from the channels drop down list?
Yes, and no. Currently, if you change the color to yellow in the config list, then it is not being logged anymore, but you can't completely remove it from the list. The reason I don't completely remove it is to allow users to disable logging, while still keeping, and being able to graph, old data. I will probably add another button to the config screen to allow users to remove the channel completely.