In the latest version of emoncms v8.3.0 that's just been comitted PHPFiwa, PHPFina and PHPTimestore feed engines no longer record the number of datapoints in an emoncms meta file. The number of datapoints are instead calculated from the filesize as recorded by the filesystem in its inode data. clearstatcache(filename) is used to ensure that php's copy of the file stat info is refreshed prior to calling filesize().
The main advantage is that it reduces the disk write load by reducing the number of files opened, written to and closed per datapoint update. It also simplifies things quite a bit. There's more on reducing write loads mainly focusing on the buffered writing and filesystem choice here if anyone missed the post and related investigation from last week http://openenergymonitor.org/emon/node/5387
Backing up before you upgrade: Given that this is a change to the feed engine implementation I would recommend making a backup of your data prior to the upgrade, there's more on backing up here: http://emoncms.org/site/docs/backup
I've also updated the usefulscripts file to no longer create the npoints data files and to ensure that the integrity check and datarecovery tools still work. The backup script also copies the remote account inputs now and It works great for replicating a raspberrypi system, just used it to setup 2 parallel test systems today.
There are also two new scripts (Thanks for asking @engeeaitch) for converting phpfiwa and phptimestore feed engines to phpfina which has a much smaller write load and is a lot better to use if your datalogging to an SD card. The PHPFina feed engine is the engine of choice (because it doesnt have averaged layers which add additional write load) for the new in early testing SD card optimized 'bufferedwrite' branch of emoncms here https://github.com/emoncms/emoncms/tree/bufferedwrite which I've started long term testing of on my home system.
Edit by mod - Paul
Don't forget, with this update, default.settings.php has been changed, and needs to be updated with your mysql/timestore key, etc, and saved back as settings.php
Re: Emoncms v8.3.0
I am currently running v8.1.2 (from apt) on raspberry pi hard drive monitoring solar PV installation.
Although I have an emoncms folder at var/www/ it does not contain much, it seems that my program folder is at /usr/share/emoncms/www/ although I only have Events under the Extras tab.
Will an update 'fix' this? (or, indeed, does it need to be fixed?) Can you please inform me of the commands to update?
Thanks. Sorry if this has been answered before.
Re: Emoncms v8.3.0
Hi johnemo,
In short no, your version isn't broken and this won't fix it, the "git" version is installed to /var/www and the "apt" version to /usr/share, the 2 install methods are not compatible.
Normally emonCMS would get updated by just running
sudo apt-get update
sudo apt-get upgrade
but currently v8.1.2 is the latest "apt" version.
I really hope the apt version will get updated soon but in the interim I have posted how to make the switch from "apt" to "git" if you need the latest version of emoncms but this is not the "recommended" route.
Paul
Re: Emoncms v8.3.0
Hi Paul,
Very many thanks for your information on the 'git' and 'apt' version of emoncms. I don't particularly want, or need, the latest version, although I do like to keep up to date on all of my software, but I'm puzzled why there are 2 routes. Is this by design or is it a mistake?
You said that you hope that the apt version will get updated - is there any question that it won't? I'm really confused why and what route I should be following.
Regards.
Re: Emoncms v8.3.0
Emoncms has been around sometime and was always installed using git, the versions were a little unclear which made support difficult and there where a lot of steps to install, plus the steps changed with different versions. so over the last year there has been a versioning system put in place and a package installer version developed to reduce installing emoncms to under half a dozen lines (on linux).
This "apt" method is simpler and more controlled, therefore easier to support. The development of this method was done by schism who hasn't been about much recently, so the versions have fallen behind a bit. When he is able I'm sure he will update the "apt" versions, they are exactly the same as the "git" versions of the same number but installed to a different location to a) prevent mixing install methods and to comply with "debian" official standards.
If schism's unable to continue maintaining the debian "apt" packages then I hope someone will step in (or I will have to take a crash course in the subject) as it is without a doubt a superior installation method for the large part.
The original "git" method was not replaced by "apt" because some users prefer it, many already have it and it is necessary for development.
Paul
Re: Emoncms v8.3.0
Hi Paul,
Thanks for your succinct reply. I'm grateful to you and all the others working to keep this project working.
Regards.
Re: Emoncms v8.3.0
Hi Trystan
I backed up my Pi based master system to my backup emoncms on a different linux machine using the original backup.php.
That worked fine.
I upgraded the backup emoncms to 8.3.
I then upgraded the master system to 8.3.
I pulled the latest backup files onto the backup system, made the changes required in backup.php and ran backup again to check every thing. I got a string of data like this:-
{"id":"2","nodeid":"18","name":"1","description":"","processList":"2:0.01,1:42","time":"1405876162","value":"2787"}
{"id":"3","nodeid":"18","name":"2","description":"","processList":"2:0.001,1:43","time":"1405876162","value":"2987"}
Then I got this:-
22 Emoncms.org feeds found
PHPFIWA: 22
local and remote meta data do not match
PHPTIMESERIES: 37
--downloaded: 0 bytes
PHPFIWA: 38
local and remote meta data do not match
PHPTIMESERIES: 39
--downloaded: 0 bytes
PHPFIWA: 40
local and remote meta data do not match
PHPFIWA: 41
local and remote meta data do not match
PHPFIWA: 42
local and remote meta data do not match
There are then more of the same errors.
Regards
Ian
Re: Emoncms v8.3.0
Ian, this might not be the answer to your problem, but did you notice that the default.settings.php file was updated in v8.3.0, and needs to be edited with database details, timestore key etc, and saved back as settings.php
Paul
Re: Emoncms v8.3.0
Ian, I've added more error logging to the backup script, hopefully it will give us some more information as to what's going wrong.
Can you see your data through emoncms in your remote server? I wonder if its the data directory settings that are wrong as Paul suggests?
Re: Emoncms v8.3.0
I missed a bug in the usefulscripts phpfiwa backup function that read the local meta data incorrectly, I've fixed that now. The directory settings where also hardcoded in the phptimeseries export function in emoncms, I've switched to use the settings file directory settings. That brings us to v8.3.2
Re: Emoncms v8.3.0
Hi Trystan
Pulled latest versions including usefulscripts.
Backup.php ran without errors.
Regards
Ian
Re: Emoncms v8.3.0
Hi
I was hoping that the apt method was updated with x86 support as I was going to test it on a virtual machine before getting a Pi.
Is it easy to change to Git when you know nothing about linux?
Re: Emoncms v8.3.0
Is it easy to change to Git when you know nothing about linux?
Re: Emoncms v8.3.0
Do you mean this guide?
http://emoncms.org/site/docs/installlinux
Re: Emoncms v8.3.0
This is the guide you want haden: http://openenergymonitor.org/emon/node/5431
Re: Emoncms v8.3.0
Trystan; Thanks, but i'm not migrating. I'm starting from scratch and looked at the apt method as it looked as the simplest method.
Re: Emoncms v8.3.0
ah ha!
haden - Your use of the term "change to Git" is why the responses aren't quite what you expected. You do have the correct link for a fresh install to linux from Git.
You are right, apt-get is by far the simplest way to install but it's still in it's infancy with regard to this project, not all software has made it to a debian package and not all platforms are catered for unfortunately.
I recall a previous discussion and believe you are installing to a non-Pi linux machine, currently that is only possible via git until the i386 or amd64 emoncms packages are available again,
Paul