I had a powercut at home the other day and then an hour or so of intermittent supply. At first glance it appeared that the raspberrypi survived it ok but then I had a look at my historic data and the raspberrypi would freeze with the system load going through the roof.
I then tried to do a backup before I did anything else following procedure 2 here: http://emoncms.org/site/docs/backup
Attempting to copy the data files preceded very slowly around 4-5 kb/s and then resulted in filesystem input/output error's which would terminate the transfers.
I found that by writing a script I could copy the data and get past the bytes in the data file that where not reading properly, it only seemed to be a few datapoints out of millions. Ive written a little tool that can be used to copy emoncms data from a usb harddrive to another backup location wile skipping the parts of the file that where causing the problem, my system is now back up and running with no visible loss of data.
I've uploaded the recovery tool here:
https://github.com/emoncms/usefulscripts/tree/master/datarecovery
To use it just enter your source and target location in recover.php, create folders for each engine in the target location, and then run in terminal with:
$ php recover.php
Its been tested with version 8.2.3 of emoncms and recovers: phpfiwa, phpfina, phptimeseries and phptimestore engine types. MYSQL has its own repair table features. timestore feeds can be recovered with the phptimestore but you will then need to use phptimestore after the recovery.
Re: Powercut data corruption and how to fix it
I've uploaded another tool that may be of interest that I've been using myself on emoncms.org just to do a quick meta data integrity check on all feeds:
https://github.com/emoncms/usefulscripts/tree/master/integritycheck
To use it:
Open integritycheck.php in a text editor and set your feed data directories:
$dir = "/var/lib";
$engines = array(
'phpfiwa'=>array(
'dir'=> "$dir/phpfiwa/"
),
'phpfina'=>array(
'dir'=> "$dir/phpfina/"
),
'phptimeseries'=>array(
'dir'=> "$dir/phptimeseries/"
),
'phptimestore'=>array(
'dir'=> "$dir/timestore/"
)
);
Then run with:
php integritycheck.php
The output it gives looks like this if there are no errors:
CHECKING INTEGRITY OF phpfiwa META DATA:
Error count: 0
Number of feeds: 2
CHECKING INTEGRITY OF phpfina META DATA:
Error count: 0
Number of feeds: 9
CHECKING INTEGRITY OF phptimeseries META DATA:
Error count: 0
Number of feeds: 2
CHECKING INTEGRITY OF phptimestore META DATA:
Error count: 0
Number of feeds: 44
Re: Powercut data corruption and how to fix it
Interesting tool.
I tried it but I am not able to locate the needed library:
include "lib/phpfiwa.php";
include "lib/phpfina.php";
include "lib/phptimeseries.php";
include "lib/phptimestore.php";
Re: Powercut data corruption and how to fix it
Ah, you will need the files in the lib folder that is also here: https://github.com/emoncms/usefulscripts/tree/master/integritycheck
Re: Powercut data corruption and how to fix it
phpfiwa, phpfina and phptimeseries appear OK, but I get the following log from phptimestore;
CHECKING INTEGRITY OF phptimestore META DATA:
Feed 11 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 12 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 13 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 17 [npoints meta file does not exist] [27:05:2014 21:04]
Feed 25 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 36 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 70 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 90 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 94 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 97 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 98 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 99 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 101 [npoints meta file does not exist] [27:05:2014 21:10]
Feed 102 [npoints meta file does not exist] [27:05:2014 21:10]
Error count: 14
Number of feeds: 14
All the above feeds display ok in emoncms, and historical data appears fine.
Paul
v8.2.4
Re: Powercut data corruption and how to fix it
So obvious that I missed it. Now it works, but I have the same error than Paul
Re: Powercut data corruption and how to fix it
Hi Trystan,
I have just used your data recovery routines because one of my feeds (phase 1 power data) stopped working a few days ago (don't know why?). After running the routines, the phase 2 power data was shifted by almost three months into the future (same for the recovered phase 1 data). I cured this by copying the 'start date' information from one of the other meta file into the meta files of phase1 and phase2. To my own surprise this seemed to work ( I have no real idea which information is contained in all these files).
Today my son accidentally cut the power supply to my server and now the feeds of all three phases and total power ( = all feeds) are shifted (they start 'now' and go until ~end of august instead of beginning in march and ending now.).
Is there anything I can do about it? (put a different 'start date' into the meta files again?)
BR, Jörg.
Re: Powercut data corruption and how to fix it
Hi JBecker,
I have recently had similar corruption issues in my meta data and I'm wondering if you found a solution to your problem.
When I run the integritytest.php script, all my feeds (which were working file up until Oct 20, 2014) return meta errors: Here's a snippet of what it returns for my phpfiwa and phpfina feeds:
CHECKING INTEGRITY OF phpfiwa META DATA:
Feed 166 [nlayers out of range: 0] [08:11:2014 0:02]
Feed 167 [nlayers out of range: 0] [08:11:2014 0:02]
Feed 168 [nlayers out of range: 0] [08:11:2014 0:02]
Feed 172 [nlayers out of range: 0] [08:11:2014 7:35]
Feed 173 [nlayers out of range: 0] [08:11:2014 7:35]
Feed 174 [nlayers out of range: 0] [08:11:2014 1:01]
Feed 176 [nlayers out of range: 0] [08:11:2014 1:01]
Feed 177 [nlayers out of range: 0] [08:11:2014 7:35]
Feed 178 [nlayers out of range: 0] [08:11:2014 0:02]
Feed 179 [nlayers out of range: 0] [08:11:2014 7:35]
Feed 180 [nlayers out of range: 0] [08:11:2014 7:35]
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfiwa.php on line 49
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfiwa.php on line 51
Feed 181 [nlayers out of range: ] [22:10:2014 0:01]
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfiwa.php on line 49
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfiwa.php on line 51
Feed 182 [nlayers out of range: ] [22:10:2014 20:34]
...
CHECKING INTEGRITY OF phpfina META DATA:
Feed 22 [interval: 0][start==0] [25:10:2014 11:05]
Feed 255 [interval: 0][start==0] [25:10:2014 11:05]
Feed 267 [interval: 0][start==0] [26:10:2014 0:01]
Feed 269 [interval: 0][start==0] [25:10:2014 11:05]
Feed 271 [interval: 0][start==0] [25:10:2014 11:05]
Feed 273 [interval: 0][start==0] [26:10:2014 0:00]
Feed 277 [interval: 0][start==0] [25:10:2014 11:05]
Feed 279 [interval: 0][start==0] [25:10:2014 11:05]
Feed 28 [interval: 0][start==0] [25:10:2014 11:05]
Feed 281 [interval: 0][start==0] [25:10:2014 11:05]
Feed 283 [interval: 0][start==0] [26:10:2014 0:01]
Feed 285 [interval: 0][start==0] [26:10:2014 0:01]
Feed 287 [interval: 0][start==0] [25:10:2014 11:05]
Feed 291 [interval: 0][start==0] [26:10:2014 0:01]
Feed 293 [interval: 0][start==0] [25:10:2014 11:05]
Feed 295 [interval: 0][start==0] [26:10:2014 0:01]
Feed 297 [interval: 0][start==0] [26:10:2014 0:01]
Feed 299 [interval: 0][start==0] [25:10:2014 11:05]
Feed 301 [interval: 0][start==0] [25:10:2014 11:05]
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfina.php on line 49
PHP Warning: unpack(): Type I: not enough input, need 4, have 0 in /var/usefuls cripts/integritycheck/lib/phpfina.php on line 51
Feed 303 [interval: ][start==0] [22:10:2014 20:34]
...
My original post to this problem can be found here: http://openenergymonitor.org/emon/node/6055
I hope I can recover the data I've collected...about 2 months worth.
Kevin
[This is a near duplicate of this post: http://openenergymonitor.org/emon/node/6055#comment-25037 - Moderator (RW)]