Archived Forum

Hello PhilThy, did you see the backup documentation here: https://github.com/emoncms/emoncms/blob/master/docs/Backup.md

Hi Trystan, thanks for the reply. I have read that document but I got the impression that it was aimed at backing up the emoncms installation rather than the entire system. I have other mods, scripts etc running on it and so I was hoping to backup the lot.

What is the operating system on your backup computer?

Paul

Hi Paul, it's Linux (Debian Jessie 64bit)

Phil.T

I use and totally rely upon a system tool called fsarchiver which has worked brilliantly for me.
I have previously used dd, but had all sorts of problems restoring an image file due to it expecting the same physical destination size which is not always the case - especially with SD cards.

fsarchiver works differently in that it only archives actual data, and creates the file system when it extracts the data. Earlier in the year, I tried to update my Rasp Pi to 'Jessie' from 'Wheezy', (and failed twice!) but after each attempt I restored the archive successfully with no loss of data.
I also restored an archive to a replacement HDD again with no problems.

A further benefit is archive size; I archive a 10GB partition which like yours contains lots of other programs, scripts, etc, and yet the archive size is less than 2GB.

Paul

Hi Paul, thanks for the info. I'm running Debian Jessie on my laptop so I'll have a look at that program you suggested.

unfortunatley, fsarchiver doesn't seem to be able to remotely backup a running system. I think rsync is the tool that I want to use, and I have done a trial backup but with errors. I'm guessing that the errors are because these are still running:

Important Make sure you disable oem_gateway/emonhub or raspberrypi_run and any posting to the http api's (stop apache) before copying the data files so that when you make the copy the data is in a state where its not being written to.

I can find the start/stop command for apache on the web but i'm unsure about the other.

Well if you are using emonhub, it's;

$ sudo service emonhub stop

Paul

Thanks Paul. Does it matter which one I start/stop first?

It shouldn't make any difference, although I've always taken the easy route, and archived during 10 minutes of downtime.

Paul

Hi Phil,

I use rsync to make a live backup of my system. Works like a champ. Here's a link to an example of how to implement it. I run it from a cron job that copies all changed files to the backup drive every 30 minutes.

https://wiki.archlinux.org/index.php/full_system_backup_with_rsync

Regards,

Bill

Hi Phil

The comments about backing up the data in useful scripts is referring specifically to the data. As does the references to stopping Apache, Redis and emonhub/oemgateway/raspberrypi. It may make sense to back-up data and file system separately. I usually stop the emoncms server and take a copy of the data, you can actually leave emonhub running and it will buffer the new data whilst you are backing up your data files.

I have also used rsync (although not enough to offer any expertise) and can confirm it can run without stopping services. the filesytem is less dependent on rapidly changing files where as databasing can go horribly wrong if the data changes mid copy.

What errors do you get? do they relate to the data files? if so you can exclude them, which if using 

   sudo rsync -axv / /target

with a 3rd "data" partition should happen automatically as additionally mounted locations are not included.

Paul

Hi Paul, thanks for your input.

I'm getting a few permission denied errors during copying, unfortunately the info flashes by so quickly that I can't say what files cannot be copied.

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.1]

This is the info after the backup completes.

I run rsync as follows:

sudo rsync -e ssh -aAXvl --delete --stats --progress --exclude-from '/home/phil/Dropbox/pi_backup/rsync-exclude.txt' pi@192.168.1.102:/ /home/phil/Dropbox/pi_backup/pi_emoncms/

The excluded directories are:

/tmp/*
/run/*
/proc/*
/sys/*
/dev/*
/mnt/*
/boot/*
/media/*

Phil.T

Try adding something like       2> ~/rsync.log     to the command string to redirect just the error messages to a log file, saves sifting them out of pages of progress logs.

Paul
 

Thanks for the     2> ~/rsync.log      tip.

I think I have found the problem, when logging into the pi via ssh I was doing so as user 'pi' who doesn't have full read permissions - hence the permission denied error.

As I had previously enabled root login, it was simple enough to just login as root. For security reasons though, it would be better to add user pi to the sudoers file:

http://www.ustrem.org/en/articles/rsync-over-ssh-as-root-en/

Thanks to all for your help.

Phil.T

thank you very much! i have been trying to use rsync to one-way sync an itunes library from one mac to another machine. is there a way to get more details in to a log file? like date/timestamp? i’ve added >> /path/to/log.txt and it works well. just need to add date/timestamp. also would it be possible to get an email when rsync is done?