Archived Forum

Hi Paul,

Tried your script. I have a file in my dropbox emoncms-2016-04-05_173205.tar.gz 

The only thing that I can report is I had the following message presumably because I am not using emonhub:-

 Setup completed!
emonhub: unrecognized service
emonhub: unrecognized service
tar: Removing leading `/' from member names

Regards,

Ian

The reason why you are getting that error is because the script attempts to momentarily stop emonhub, to ensure that when the data directories are copied while they are not being updated by emonhub.
The first error is when it tries to stop it, and the second when it tries to start it again.

You could always comment out the references to emonhub, ie;

# sudo service emonhub stop
# sudo service emonhub start

I'm open to suggestions of how best to handle this, as data could also be written to the data folders via MQTT. 

Also, I've found a typo in the script which prevented the weeding of old archives and have just updated it.
Probably the easiest way to update is just delete the dropbox-archive directory, and git clone it again, (add your emoncms password to dropbox_filemanager.sh again).
You won't have to authorize the script with Dropbox again, as the script has already saved the authorization in a hidden file, saved at /home/pi/.dropbox_uploader

​I've also added an option (disabled by default) to backup node-red flows, credentials and configuration. To activate it, just select (Y)es in the configuration section of dropbox_filemanager.sh

Paul

Edit 15/4/2016 - now fixed

Hi Paul

As I am not using emonhub do I need a way of stopping data being posted during the backup run or is it not likely to be an issue.

Regards

Ian

Hi Ian

You may have seen my other recent post regarding this issue, and to try and overcome the possibility of a corrupted archive, I've found a solution, which doesn't involve stopping services, yet will secure a snapshot of the emoncms data.

I'm going to rewrite the script in php (if I can!), but it may take a week or so to complete.

Paul

I'm open to suggestions of how best to handle this, as data could also be written to the data folders via MQTT.

Hi Paul,

Here's some info about a method to determine, via a bash script, whether or not a process is running.
Perhaps the logic flow can be of some help if/when you rewrite the script.

Thanks Bill, I've now found a way to get the feeds data without stopping any of the services, which seems the safest and most resilient option.
I'll update this thread when I've updated the github repo.

Sounds good! Looking forward to seeing what you've come up with.

I've just updated git with a new version, which should deal with the issue described by Ian.

The main changes are;

• Script now uses some code written by Trystan to obtain the feed data by using emoncms's API, which avoids stopping and starting the emoncms services to back up the data, and therefore avoids data loss and greater compatibility with different user configurations.
• Main script now written in php
• Can be installed or run from any location in a pi
• Now uses a config file to save user settings. This will now allow fixes, enhancements, etc to be updated by doing a git pull from the installation directory from this version onwards.

I've been testing the new version for over 3 days now, running the script every 30 minutes via node-red, so have created almost 150 archives, which were automatically weeded after 12 hours. All of the archives were checked for consistency of file size, and have used 5 archives chosen at random to perform full backups onto a 2nd pi, all of which were successful for both emoncms & node-red. Just remember to check the file ownership when using the backups, ie $chown -R pi:pi .node-red (same for emoncms data directories).
Each backup comprised of 29 emoncms feeds, MYSQL dump and the essential node-red files, and was just over 60Mb in size, but was compressed before uploading to just 13Mb.

To update from the previous version, delete the original installation directory, and git clone the latest version as per the readme. (you won't have to re-authorize your dropbox API).

Paul

Hi,

Thank you for sharing your code, though I am having issues with backing up. At the end of the script, i get the following:

Dumping MYSQL data
Backing up node-red data
An error has occurred, details:tar-based phar "/home/pi/dropbox-archive/backups/archive.tar" cannot be created, contents of file "phpfina/44.dat" could not be written Checking for expired archives...
Uploading new archive to Dropbox, this may take a while...
find: `*.gz': No such file or directory

I presume there is a permissions issue?

Kind Regards,

Dave

I presume there is a permissions issue?

Doubtful as the script will be running as Root. Can you check how much free space you have left on your pi pls;

$ df -h

Also, from your feeds page, can you tell me what is reported when you 'Refresh Feed Size'.

Paul

Hi Paul,

the output is:

pi@emonpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  1.9G  1.4G  59% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M   13M  451M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p3  194M   66M  119M  36% /home/pi/data
tmpfs            40M  6.6M   34M  17% /var/lib/openhab
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcp
tmpfs            50M  8.4M   42M  17% /var/log
tmpfs            30M   40K   30M   1% /tmp
/dev/mmcblk0p1   60M   20M   41M  34% /boot

Kind Regards,

Dave

Hi Dave

As I said in my first post,it hasn't been tested on an emonpi (as I don't have one).
Presumably, you've installed the script in your writeable /home/pi/data directory which shows that you have 119Mb free.
I don't know how large your feed data is (check by running 'Refresh Feed Size' from your feeds page), but the script will need at least that amount of free space to create the archive PLUS enough space to store the archives locally (which compressed are about a quarter of the size of the feed data each).
So if your feed data size is 100Mb, you would need approx 125Mb free to create the first archive, and an additional 25Mb for each archive.
If you were keeping 7 days worth of archives, then you would need 300Mb of free space. After 7 days, the older archives are weeded, so won't increase further thereafter.

Does this appear to be the issue? If so, it's a shame that you've 1.4G unused in your root partition.

It will probably be a good idea for me to add something in my first post to check for free space before installing...

Paul

Dropbox changed it's API format 2 days which caused the script to fail.
I've now updated the script in Github, and it's back working!

Paul

Still working well! 7 days worth of emoncms & nodered archives safely stored in Dropbox. Archives older than 7 days are automatically weeded.