Archived Forum

Hi

I can confirm that the PI SD card eventually fails if used as the database. I have had 2 fail. I now have a USB hard disk and used berryboot to move the PI Linux installation onto the hard disk. Time will tell if this is more reliable.

Regards

Ian

Sorry to repeat myself, but the accumulated wisdom elsewhere is to boot from the SD card and run on and do all writing to a genuine spinning hard disc.

Ian, you're right. I think a spinning USB hard disk should be more reliable in time, or so I've heard. But this means higher cost, higher power consumption, and more space needed.

Now, if the Pi is just used as a repeater, like a Nanode RF, there should not be so many write accesses, and I don't see where the problem is in terms of reliability. There may be too much logging but this is something that can be customized, and, well, the Pi is supposed to be running a linux on an SD card, I hope this can handle a reasonable amount of logging.

(One may argue that it is more expensive than a Nanode, but it has other advantages: versatility, possibility to connect to and operate on via ssh,...).

Hi Ian and Jérôme,

in my system, i combine two devices at the same time to run it. At the beginning used only the SD card and the system fails when time passed. So i combine a SD card (small capacity, it´s not necessary a big one. 128 Mb is more than enough) only for boot purposes (RasPi) and then a pendrive (32 GB) to store data. This combination runs perfect for me with no problem and completely stable.  

Best regards,

Nacho

Isn't there just the same issue with a pendrive !?

https://en.wikipedia.org/wiki/Pendrive#Disadvantages

https://en.wikipedia.org/wiki/Flash_memory#Memory_wear

By the way, I read this:

Some developers have produced special versions of operating systems (such as Linux in Live USB) or commonplace applications (such as Mozilla Firefox) designed to run from flash drives. These are typically optimized for size and configured to place temporary or intermediate files in the computer's main RAM rather than store them temporarily on the flash drive.

I don't know what is done on raspian regarding this. (Haven't searched for it specifically.)

Hi Jérôme,

may be, but nowadays has been the only method i find to run it stable. At the moment I have no problems with this configuration. May be in the future. The only thing i could tell you is that it´s work fine for me. Very stable with that configuration at the moment.

It´s very important to me to have my data base in good conditions and for now that's what happens. In the past my data (specifically the SD card) was corrupted after a certain time but not now ... I will tell you if the problems return.

Is important for me simplify system installation without using a usb hard drive and for now, the pendrive works perfectly.

This is my pendrive in use:

Thanks for the wikipedia links by the way ... I´ll read them carefully.

Regards,

Nacho

Hi Jérôme

I just realized that you are suggesting the PI as a replacement for the NandodeRF to be used as a base station forwarding all data to a remote server. I think that is an excellent idea for those prefering to use a PI.

We would then have the possibility of using a PI used in 3 different modes.

1 - As an emoncms server network connected to a base station. (This is how I have been using it prior to the latest emoncms software with a nanodeRF used as a base station)

2 - Fitted with the RFM12PI used as a direct radio connected server which seems to be what is currently being promoted. (Which I think could be a mistake unless the data is written to a hard disk) .

3 - Fitted with the RFM12PI used as a dumb base station forwarding data to a remote server as you are suggesting.

Regards

Ian

Interesting subject,

also check this topic, where they discuss the option to mount the SD as R/O

raspberrypi.org/phpBB3/viewtopic.php?p=217757#p217757

I went through a couple SD cards that didn't work well with emonCms, but have been running for about four months without problems, logging data continuously.  I do expect eventual SD card failure.  So my solution has been to run an automated backup of the mysql database, using automysqlbackup as I read about here on OpenEnergMonitor a couple months ago.  Every day the emoncms data is automatically backed up to my main computer whenever I happen to turn it on.

For now I am using a NanodeRf as an emonBase, forwarding data for emonCms logging on my RasPi.  I do have an Rfm2Pi running on my RasPi, but I am using it for other purposes (controlling my house heat pump).

I did set up (on a second RasPi) a USB hard drive.  It works, but it is loud and draws about 8 watts.  That drive is an older external drive that I was no longer using.  If I had a small, inexpensive, low power USB hard drive, the external drive could be a good solution.  In my quick search, it seems there is no large market out there for hard drives like that.

I have an SSD (solid state drive) in my desktop computer.  SSD warranties are approaching those of HDDs.  But isn't the technology of the SSD memory more-or-less the same as SD card memory?  So why is the SSD more reliable than the SD card?  Or is it, I am a bit confused about that.  A small SSD (say 32GB), if it actually were reliable, would seem good for the RasPi.  Cost should be mostly proportional to size, so it would be relatively inexpensive, though the trend is toward larger drives, which might make small ones unavailable and/or more expensive.

A quick check at Newegg shows a 32GB SATA SSD for $50 with three year warranty.  Add an SATA-USB converter for say $10, and that would be a solution.  Though maybe there are reasons this wouldn't work with the RasPi, I don't know.  And again, just because they say 3 years, I don't know why this would be more reliable than an SD card.

-Ed

My quick solution was to create a MySQL Master to Slave replica to have a backup. Master is the database in the RasPi instance and Slave is a database in a Windows system with Wamp server (and of course MySQL instance) in it.

I know this is not the best solution but a had a lot of problems in forwarding data from RasPi to the laptop instance. Every thing in the Raspi emoncms parameters is correct (green color) but no data in the laptop instance with the forward method.

My emoncms data is backed up automatically hourly but for me, replication is the best method that i could find. When i try to restore a backup, this is a slow process for me. It takes me a few hours to restore the entire database.

Regards,

Nacho

The Raspberry Pi is the first small Linux computer I have ever seen that boots and runs from SDHC or Compact Flash.

All small Linux computers I've seen prior to this boot from SDHC or Compact Flash, and then run from a ramdisk with the SDHC or CF unmounted and powered down.

To preserve data when the computer is powered off, closedown scripts either back up the written files to a Network File system, or scribble them back to the SDHC.

I have only had my Raspberry Pi a few days, but this is the first mod I will make to my system.

Anyone interested in this should look at the buildroot project http://www.buildroot.org/ where I understand Raspberry Pi is now an official build target.

Mike

My SD card failed earlier this week (after just 5 weeks), and rather than buy another 16Gb card (to accommodate my backup image), I've split the SD card partitions to boot from a 120mB SD card (old TomTom card!), and moved the 'root' to a external USB HDD partition.

I want to use one of the other partitions in the USB HDD to act as a network backup device for my other computers (not quite sure how to do that yet!!).
If I can do that, then the cost of buying and running the external HDD is more justifiable.

Paul

I've now got a version of Linux running completely in RAM. There is no SDHC activity after the system finishes booting. The system has been running now for several days, and here are my disk stats

$ cat /proc/diskstats | grep mmc
179       0 mmcblk0 182 3 1480 210 0 0 0 0 0 200 210
179       1 mmcblk0p1 17 3 160 50 0 0 0 0 0 50 50
179       2 mmcblk0p2 80 0 640 70 0 0 0 0 0 70 70

They have not changed since I first logged on.

I guess I could upload the kernel image here because the filesystem is an initramfs, so you'd only need the kernel.img file, but it's not easy for anyone to change the filesystem without the complete build environment. You can of course change it when you log in, but of course you loose any changes when the system reboots.

As I have it running, there's no X environment, no emoncms, the system just listens to the RFM12pi and collects data, passing emoncms .json requests out to the www.

Hi Mike.

Thanks for your inputs.

I guess I could upload the kernel image here because the filesystem is an initramfs, so you'd only need the kernel.img file, but it's not easy for anyone to change the filesystem without the complete build environment.

Perhaps can you just list the steps to accomplish this. Is this a linux you built more or less from scratch ? A raspian ? Did you recompile a kernel with different options ?

As I have it running, there's no X environment, no emoncms, the system just listens to the RFM12pi and collects data, passing emoncms .json requests out to the www.

What gateway do you use to listen to the RFM and send the data ? Currently, I think emoncms is needed because the parameters for the radio are entered through the Raspberry Pi module GUI. Did you hardcode the parameters in an existing gateway script (php or python) or did you make your own ?

I've been using the buildroot environment for creating embedded systems for several years. Details can be found here http://buildroot.uclibc.org/. Buildroot is a complete build system. You tell it what the target system is, what programs you want included, the format of your filesystem etc, and then sit back for an hour or so while it downloads all the code and builds it for you.

There is a default configuration for the Raspberry Pi. I used that. I notice it has pulled in a load of stuff I'll never use, like programs to process the optional camera, but they are not large so I've left them in the image.

As for the RFM gateway, I wrote my own in C. I'll attach the code. It is nothing clever. I used minicom to program up the RFM radio interactively, then locked it. I've not seen any corruption of the frequency or node numbers that the php gateway comments seem to imply happens, though my gateway code (monitor) does occassionally get stuff it's not expecting, i.e.

May 13 08:24:06 pi user.info syslog: monitor: Uknown packet Available commands:
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   123 x      - Toggle configuration change protection, 1=Unlocked
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   <nn> i     - set node ID (standard node ids are 1..26)
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   <n> b      - set MHz band (4 = 433, 8 = 868, 9 = 915)
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   <nnn> g    - set network group (RFM12 only allows 212, 0 = any)
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   <n> c      - set collect mode (advanced, normally 0)
May 13 08:24:06 pi user.info syslog: monitor: Uknown packet   ...,<nn> a - send data packet to node <nn>, with ack
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet   ...,<nn> s - send data packet to node <nn>, no ack
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet   <n> l      - turn activity LED on DIG8 on or off
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet Current configuration:
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet 79 i15 g210 @ 433 MHz  Lock: 0
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet > 0s
May 13 08:24:07 pi user.info syslog: monitor: Uknown packet  -> 1 b

Code for monitor - my RFM gateway attached. Node numbers are currently hardwired, as are the names of various shell scripts I run to do the processing. I had intended to set all that stuff interactively with command line options but have not got around to it yet.

If you want to try buildroot, download it into a directory called buildroot, cd to it. type "make menuconfig", exit out of the menu. Then untar the buildroot_configs.tar file. That will unroll all the config files I have used. Type "make" and sit back. You will get a complete copy of my system built from scratch.

Mike

After my first SD card failure (the original pre-loaded has lasted only 3 weeks), I have moved the root partition to a USB pendrive, and left boot partition on a small 128 Mb SD card.

Nacho, is it yours still running?

I cross my fingers hoping that my pendrive have implemented the wear leveling and so. Next step will be replace the USB by a (small and cheap) USB HDD that I am still looking for.

In order to avoid massive writing to the Sd card or USB pendrive, is it possible to configure the Raspberry as only a gateway and send all the data directly to emoncms.org server?

In order to avoid massive writing to the Sd card or USB pendrive, is it possible to configure the Raspberry as only a gateway and send all the data directly to emoncms.org server?

I've been working on a standalone gateway, and I'd be happy to get your feedback :

https://github.com/Jerome-github/oem_gateway

I did experiment with a USB pen drive, mine lasted only two weeks. I guess the larger capacity, the better wear leveling would be, mine was just a basic 4GB USB drive. My final approach was to do a R/O root file system and have the Raspberry Pi as forwarder only.

I've had a 4gb thumb drive (integral) working for nearly three months now without error. I thought it would make sense to upgrade to a larger capacity drive after reading about wear problems, so I purchased 2 32gb mini usb thumb drives (again Integral) - one for the Pi and one spare. The first failed after about 4 hours of use and is now unmountable on my Debian laptop. The second hit problems after a few hours use and after checking, was found to be heavily corrupted. I guess some are more robust than others, even down to different types from the same manufacturer (the failed drives were purchased direct from Amazon.uk)

Hi Martin,

Sounds greats - downloaded oemgateway but can't login to change pi settings e.g. ssh using normal default pi and raspberry login. What is default raspberry pi login for oemgateway ready to go.

Thanks

Peter

Do you mean you downloaded emoncms as a ready-to-go SD card ?

emongateway designates (to me) the python gateway I coded, to be used on a Pi with a linux installation, it has no login/pwd by itself.

If you intend to use the Pi as a repeater, you don't need the ready-to-go SD card, rather a raspian or other distro (I'm using raspian) on which you just have to install the gateway.

Hi Jerome, I found this file which I assumed was complete ready to go for  oem gateway "oem_gateway24sep2013.img.zip" at http://emoncms.org/site/docs/raspberrypigateway

I loaded it onto cleaned SDHC card and booted my Pi. Seemed to boot ok. It offers login and also displays gateway traffic.

I just tried ssh and that offered login, woeked when I tried login root & username root. Now I'm confused, do I work with above or is there a different procedure I should have followed to get oemgateway.

This comes up on log in -

Industrial Perennial Environment - Release 1
by NutCom Services, Inc. - http://nutcom.hu/ipe/

Thanks Peter

Wow. This indeed is the python gateway I'm referring to. I just didn't know a ready-to-go card had been released that used it (which I'm glad. Means it might be useful to someone.). Hence the confusion.

According to the doc page you provided, root/root is the way to go. Are you saying it worked when you did this ? Then everything's fine. Change root password and you're done.

(I would have created a normal (non root) user account, but perhaps the rationale here is that the system being read-only, it is not meant to be used for usual tasks anyway.)

I'm not sure. Are you still stuck or is everything settled ?

Hi, No not stuck just wondering was I on right path. I'll try configure to suit my setup tonight. Also try a 3G solution I've been working on using "umtskeeper". Excellent

Thanks Peter

Hi Jerome and Glyn, great work on the oemgateway image.  I am having a small problem with the SD image.  When I run -

root@oemgateway:~# ps -ef | grep python
root      1221  1206  2 14:30 pts/0    00:00:48 python /root/oem_gateway/oemgateway.py --config-file /boot/oemgateway.conf
root      1242  1226  0 15:06 pts/1    00:00:00 grep python

the system returns the above.  If I run the kill command for 1221 it tells me there is no such process running.  Also no data is being sent to emoncms unless I run the command

python /root/oem_gateway/oemgateway.py --config-file /boot/oemgateway.conf

and then it spews out the expected data on the monitor and emoncms begins to update.  When I CTRL-C out of that, emoncms quits updating.  Essentially the system does not start or continue the process to run the program when the system boots.

Any ideas about what could be wrong?

Regards...

Hi.

I can't figure out how i experienced two SD failure in two days using emonCMS linux images, while i have a little raspberrypi server running mysql + php since one year without any problem. I think to investigate deeper what kind of failure SD had. One difference with two platforms is that my working set i used ext4 fs with "noatime" option.  May failure due to rewriting many times the same block? I'm not expert of filesystems but i realized it is very strange a so fast failure. If i could write any cycle a different location in my 8GB data structure it would take a long time return in the same cell before filling the whole data space available. Could we consider to use a special filesystem for SD memory cards?

I found an interesting discussion in http://raspberrypi.stackexchange.com/questions/169/how-can-i-extend-the-life-of-my-sd-card

Thanks

Giuseppe.