Archived Forum

Thanks for letting us know, is it 2 years already!?

What form is your backup in? mysqldump? and feed data files?

Yeah I had mysql dumps done, now I realise that may not have been enough :( I'm going to download some forensic tools to see if I can rescue any of the scripts and stuff I had sitting on there I hadn't backed up. 

Going forward for the new install I'm going to look into tarsnap as an offsite backup solution for the data partition, seems like it will plug in and be relatively cheap to hold. 

I've been using tarnsap (not for emoncms stuff) for well over a year now, and am very impressed with it.  I've now got "110GB" of tar files stored with them, or at least that's how it appears to me (to their servers it appears as about 850MB).  Each of my full backups is about 2.2GB, but thanks to the magic of block de-duplication and compression, a full backup takes just moments and results in just a few hundred kB being written to the cloud (subject to how active I've been).  Some days it shows an entire day's work can be summed up in just a few kBs!  My initial $5 deposit has been depleted down to $2.57.

Hello guys, sorry for the loss..

I had an idea about how to make smaller changes to the setup on the pi in order to get a more durable system. I am not a unix wiz and dont know how much of this is possible in reality. But using common services as dropbox, and OS features like RAM disk might help us out in creating systems that will run forever and never need manual backup. If this can be automated it will really be useful.

Imagine "time machine" (Built in OS X super simple backup feature) like backup from the raspberry to dropbox or any other free service.

I wrote som in this thread about the idea, feel free to comment.

http://openenergymonitor.org/emon/node/10812

Here's a daemon that would help extend flash memory life. Excerpted from the author's webpage:

Ramlog act as a system daemon. On startup it creates ramdisk, it copies files from /var/log into ramdisk and mounts ramdisk as /var/log. All logs after that will be updated on ramdisk. Logs on harddrive are kept in folder /var/log.hdd which is updated when ramlog is restarted or stopped. On shutdown it saves log files back to harddisk so logs are consistent. Ramlog 2.x is using tmpfs by default, ramfs and kernel ramdisk are suppored as well. Program rsync is used for log synchronization.

I've been running it on an early Model B Pi since Sep of 2014, and have not had any problems with it.

Site URL: www.tremende.com/ramlog/

Ramlog is GPLv3 software.

Wow, that is an encouraging insight dBC! Glad to hear that setup is working well for you. I hope that a $5 deposit will last a long time.

goodfidelty and Bill, the prospects of RAMdisk and ramlog seem interesting. Maybe Trystan can throw in his 2 cents because I remember in the early days the read/write topic received some investigation and these ideas/tools may have been considered already.

Yepp.

In my opinion the dropbox idea should be considered as an option for all users anyways, a nice online automatic backup of the database would be very nice.

I am not very well aware of other system settings that needs to be backuped, but using an automated script that keeps settings, scripts, plugins or whatever backed up together with database to dropbox account would be almost to good to be true =)

Having the same script in the pre installed SD image file with an option to recover from backup would make it just to easy when making new installations. Just put dropbox user ID and PW and everything is just where you left it.

Dropbox is free, and by inviting friends you can get as much as 16Gb of free data. I am not sure but i guess that will cover many years of usage. And for customers who like to pay for this service i am sure its still a good feature.

The whole RAM Disk idea i think needs more tinkering. But i am sure that anyone with some LINUX experience can get it done.

//J

Raspbian on the Pi is already using the RAM Disk feature for its temporary files.  The Unix name for this is "tmpfs" as described here: Wikipedia article on tmpfs

We have 6 separate tmpfs filesystems in the Pi's RAM -- /dev, /tmp, /var/log and 3 under /run.

Indeed it would be possible to use another tmpfs for data logging.  My emonPi durrently has 740 Mbytes of memory free, so it would be safe to allocate maybe 200M for this purpose.

Thank you Steve for this valuable input.

Maybe you know how much data is created in one month of average logging? Is 200Mb enough for one year or so? I think it will probably be more then enough.

How much of a hassle will it be to reconfigure the emonSD card to make all the logging to ramdisk in the low write mode?

Will there be any major savings to read/write cycles on the SD if everything is logged to the RAM and later to the SD maybe once every day? Maybe this can be done with RAR or something to save space and cycles?

I think this is one very important feature for the whole system, to be durable and redundant.

//J

I just want to add this link for cross reference:

http://openenergymonitor.org/emon/node/10812

One year of logging will be no problem.  My Pi uses the PHPFINA engine and the files named /home/pi/data/phpfina/NN.dat (where NN is the feed number) grow by 24 bytes every minute.

That's 34 kbytes for a day and 12 Mbytes for a year.  I have 3 feeds (house, solar and solar kWh) so altogether 36 Mbytes of logged data in one year.

Logging to RAM would take a small amount of scripting.  After a reboot we begin with an empty RAM filesystem, so a boot-time script must set up the logging directories.

Of course data will be lost at any power failure or reboot, so your idea to copy back to SD once a day is good.  Compression will help -- I tried gzip on one of my .dat files and the compressed data is 30% of the original.  The big saving is in not writing the directory (actually the inodes) every minute.

For my 3 feeds, Linux on the Pi reports 6 kbytes per minute written to the mmcblk0p3 filesystem, the one with the logs.  Much more than 3*24 bytes due to the directory overheads and the 1k filesystem blocksize.  I used this command to get the bytes written, then checked again 10 minutes later:

iostat -p /dev/mmcblk0

Inside the SD card the bytes written will be even more than 6k/minute due to blocksizes and wear levelling.  So a daily copy from RAM would be a big improvement.

I don't think moving the data feeds to RAM will improve much as when the existing SDcard image is using low-write emoncms the data is already buffered in RAM and then written to disk once every minute, this minimizes the data loss if there is a restart to 1 min at most.

As your calculations show the writing of the data to the data feeds is minimal and that part itself would take an eternity to wear out a SD card as the same part is never written to twice, even editing and rewriting past data has been prohibited. I believe the main cause of SDcard burnout is the use of mySQL, The indexing (and "last value" and "time of last update" type fields) will constantly be rewritten and quickly hit the wear limits for a particular location.

Perhaps it is possible to move the mySQL files around as a form of wear leveling or move the sql tables to RAM and write to disc once an hour or so???

Ultimately the way to go would be to replace the SQL tables with text based files but that probably involves a major rewrite of emoncms 

Paul

Paul,

On my Pi I can't see any significant SQL activity.  Timestamp on /home/pi/data/mysql/ibdata1 is the time of the last boot. In the emoncms subdirectory the only recent activity is in the "rememberme" table, from a few hours ago when I started a web browser session.

Unless I missed something, all the action is in the phpfina files.  (My config is simple -- 2 current sensors, logged to 3 feeds then viewed in My Solar.)

I know very little about SQL or the intricacies of writing data to the SD card and struggle to understand how writing data to the SDcard in whole blocks once and once only to any single point could significantly contribute to the SDcard wear, I understood that database activity was a main cause of SD card failure so from that I'm led to believe it is databasing but not the feeds themselves, which makes sense (to me) as I expected mySQL indexing and updating to be write intensive and occur frequently in a fairly consistent manner (location and time wise) .

If it isn't the the SQL indexes then could it be the inode metadata file being updated constantly? if so perhaps an automated process that regularly results in a new (relocated) inode could increase SD card life expectancy.

Paul

Here's an interesting read about how the way an SD card is formatted can affect not only its performance, but its lifespan too: http://3gfp.com/wp/2014/07/formatting-sd-cards-for-speed-and-lifetime/

Excerpt from the article:

Raw flash chips are divided up into several integral read and write sizes. It is important to keep these sizes and their alignment into consideration or a card may end up performing 2 write operations for each 1 write operation intended thereby cutting the life of the flash in half.

The article made me wonder if that's one of the contributing factors (among many) as to why some folks seem to burn through SD cards, while others seem to get reasonably long life from them.

Paul,

I also don't know how to reconcile the low writing rate with the topic of this thread, namely the death of an SD card after 2 years.

Much depends on the SD card's internal design.  Those once-a-minute writes could hit 6 blocks (3 files and 3 inodes) and a block could be 32k or more, so inside the chip we are writing 6 times 32k which is 32 times the rate observed in Linux.

Bill,

From that article I learned that flash controllers are optimised for FAT32.  For a Linux perpsective try Optimizing Linux with cheap flash drives (written in 2011 when the kernel programmers were working in this area).

Maybe the answer is to establish a optimum "write" regime and work backwards. eg from a quick glance through Bill's link it looks like 128kiB blocksize is an optimum size if aligned with the partition structure. each day or hour (which ever is more efficient) the "current" block gets moved to minimize wear and when the block is full append it to the main data file and create a new one, so "the feed" is actually made of 2 files a static "main" and a rolling "current"??

Just throwing ideas out there, as I see the potential loss of up to an hour or a day's worth of data a much bigger inconvenience than replacing the SD card once a year, as a preventative measure since we get quite a few power interruptions, mostly momentary, but nonetheless, enough to cause a reboot. 

Even if the ideas won't fly, if someone points that out, I consider myself a step closer to finding a solution :-)

Paul

Unfortunately the optimum blocksize may be different for each make/model of SD card.  This page has a long list from a survey in 2012-13: Flash memory card design  My conclusion is that it's too hard to improve on the optimisations already done, both in the kernel and by the delayed write strategy.

For peace of mind I will move my phpfina directory away from the SD card, either to RAM or a USB stick.  Then the only writes will be for software updates, config changes and the 'rememberme' table in mysql.

Backups of the feeds in phpfina will be:

1. emoncms.org
2. daily (cron'd) copy to a Linux machine

I'll use rsync to make a mirror of the phpfina directory on my Linux machine.  This needs no extra software on the Pi, just a key in /home/pi/.ssh/authorized_keys.