Archived Forum

Don't forget SD cards to fail so you are going to want to backup regularly - perhaps to a cloud host?

Well, I'm just transferring my live db to my localhost for testing .. I've been running since December or so .. I have 25 live feeds, a mixture of power and temperature monitoring .. 

root@raspberrypi:/media/external_hdd# du -hs emoncms/
861M emoncms/

37 tables .. 11,429,162 records .... 

25 feeds is a little more than I would I need so lets round you figure down to say 500MB every 6 months so that would be around 3 to 4 GB I would need. I'm thinking an 8 GB card will do the job.

I have also been toying with the idea of getting the raspberry to receive the data from the sensor unit and then send it to my home server which has loads of storage space, backup, raid etc. This would then avoid the need to backup the raspberry. Does anyone know if this has been done?

I presume it will be very similar to sending the data to emonCMS only your using your own server rather than theirs.

25 feeds is nothing. I have 5 TinyTx each reporting 3 feeds (Temp, humidity and battery). The rest is a single Current Cost monitor with three IAM's posting in. However you decide to interpret the data can add extra feeds.

Personally, the data logging is far heavier than my experience with RRDB .. with that I had per second logging with a known database size for 10 years of logging. It is, however, a PITA when you want to move from one system to another (well .. easy enough .. but more of a PITA than copying). Am pretty sure my db sizes were more in the region of 25MB per feed for a 10 year rolling/per second store.

I've been wondering about doing a reverse of my Current Cost MQTT -> EmonCMS extension .. to get all data from EmonCMS to MQTT ... in fact .. the MQTT->EmonCMS could then be heavily improved. (https://github.com/elyobelyob/mqtt)

For long term logging, I think EmonCMS has some db issues. My issues so far I believe to be due to tablespace scans and other reasons.

So, not sure whether EmonCMS or RRDB is best .. my best idea so far would be to write into MQTT and do both, or more .. e.g.  Cosm or remote EmonCMS etc would all be easy enough. MQTT would allow you to add/remove more easily.

Raised earlier is the problem with Pi and SD cards. Be wary .. I had tons of issues with db on Pi. I now use an external HDD for MySQL. I have had no corruption problems since.

1GB is a crazy amount of data for the period I have logged so far.

Hi MadHatter. I have to agree with Stuart and Elyobelyob. I bought a 32Gb SD card for my Pi to ensure there was enough space. After several weeks, I started having problems with MySQL. The lock file seemed to be getting stuck, causing MySQL to halt. I then rebuilt on a different brand 8Gb SD card. After about five weeks I got the same problem, then the system would not boot at all.

I would strongly recommend that you do not buy a large SD card. From what I can tell from my research on the internet, SD cards are not built for lots of writes. They can be read from frequently, but many writes are likely to cause a card failure akin to a bad sector on a hard disk. I am starting to trial booting the kernel from a 16Mb SD card and then running the root file-system from a USB flash drive. Apparently people running WordPress on R-Pi's have had more success this way. Even more reliable would be an external hard disk.

There is a lack of information around about how poor SD cards are when used as a hard disk. I think this is something the Raspberry Pi foundation should actively warn people about.

I'd be interested if anyone has had positive experiences running MySQL from an SD card on a Raspberry Pi.

Hi

Regret to advise that I have had a terminal failure of an SD card running emoncms after about 12 weeks. I now back up daily to a usb memory stick. I have an external usb hard drive on order and I hope I can install emoncms on that rather than the SD card.

Being a bit Linux challenged (but learning fast) I shall be looking for step by step instructions on how to set up linux and emoncms to run on the hard drive.

Regards

Ian

Hi Ian. Raspberry Pi Linux distributions such as Raspian or Arch Linux have two partitions in their images. One is a FAT partition that contains all of the boot files. You can just copy these files onto a standard SD card. They are less than 50Mb in total. You can then write the image as you would do to your SD card to the USB drive. The second partition is the EXT4 root file-system.

All you need to do then is edit the 'cmdline.txt' file on the boot partition (on the SD card) and change the 'root' parameter to something like '/dev/sdb2' depending on how you partition your new hard disk. Your 'cmdline.txt' file should look something like this:

dwc_otg.lpm_enable=0 console=tty1 root=/dev/sdb2 rootfstype=ext4 elevator=deadline rootwait

This guidance should probably be added to the installation procedure for emonCMS to prevent other people making the mistake of running their root file-system from the SD card.

Many thanks.

Will give it a go as soon as the hard disk arrives. I may be back with questions!

Regards

Ian

Thanks for the replies everyone. I think there are two options for me:

1 send the data from the raspberry to my server

2 Save the data on a USB drive plugged into the raspberry.

I think your R-Pi should be your server. The chap who runs this site; www.dingleberrypi.com seems to have had a positive experience running his WordPress server from a USB flash disk. I can't vouch for this approach yet, but this is the structure I'm betting on. Fingers crossed!

Yeh I will try the raspberry as the server first with the data being stored on a USB drive and maybe get it backing up to my main server.

The raspberry arrived today so should hopefully get playing with it soon although I still need to order the bits from here.

WARNING - Writing to often to the SD Card when data logging will corrupt it.

I have another Pi running pywws (weather station) and a number of us a running into SD card corruptions - the cause seems to be the number of writes to the SD card - so be careful with data logging, pywws was doing a significant number of writes especially with live logging, and all sorts of weird errors were occurring. - A bit of analysis and it's been diagnosed as hitting the the number or writes / rewrites to the SD card - so beware.

Hope that helps.

I've read elsewhere that SD cards are a bad idea where frequent writes are concerned. As Barry says, only read from the SD, do all the writes to a genuine spinning hard disk.

Interesting might explain a couple of failures I've had. In cases where the raspi local emoncms install is reporting directly to emoncms.org are there ways of reducing writes to SD to increase longevity? Removing feeds from the local install would reduce mysql writes but would this make a difference or is it all about temp usage?

Hi Matt. A friend of mine recommends not using a journaling file-system to reduce the number of writes. Try using an ext2 partition instead of ext3 or ext4. I have little faith in flash media anymore, the pile of defunct SD cards and USB disks nearby is a testament to this.