Archived Forum

Hi

I don't think you will have any success without a powered USB Hub. I tried to get a hard disk running without a hub and ran into problems. The Pi kept hanging. I had another Pi with a powered USB hub that also powered the Pi off a very short gold USB lead. This worked perfectly without a hard disk. I then added the hard drive and the Pi started hanging. I had to end up with 2 power supplies. One for the hub and one for the Pi. I was trying to use a single power supply as my setup runs off a 24V solar system and I wanted to use a single 24V to 5V  DC to DC adaptor. Having searched online it seems many others have found that for reliable operation you have to use a powered hub on a supply  independent to the Pi.

The one thing I intend to try is running the hub and Pi of a single DC DC adaptor each with their own individual connection as the DC DC adaptors I use handle up to 2.5amps.

Regards

Ian

Interesting to hear your experiences Matt. It's a good question you have raised. I know this would defeat the point of using a Pi but it would be possible for the Pi to relay the data straight to emoncms.org or another emoncms server without also logging locally. This will require an edit to the Raspberry Pi emoncms module. 

Yes, this is the approach I was suggesting here : http://openenergymonitor.org/emon/node/2292

This might make the Pi less interesting for a large scale usage, if a Nanode is a simple solution.

But it still has pros : possibility to use it for other tasks as well, remote access through ssh to work on it, even reprogram the RFM2Pi since v2.

I could try to propose a patch, but I think it would make sense to rethink the logic of the module and its GUI (see the other thread mentioned above), so I wanted to have other opinions before diving into it. Besides, it is quite possible that I'd need a bit of help with the php. If Trystan is interested, I'd like to have a design discussion before going down to the code. Perhaps github is a more advised place for these discussions.

BTW, I heard about Olinuxino, which could be a cheaper alternative to the Pi with equivalent features. Not sure what it implies for the RFM2Pi. After all, it is just serial communication.

Thanks Ian, that's really useful information, so any spinning disk model needs to factor in powered usb hub cost and socket availability. It certainly makes the system more intrusive in the living room.

Jérôme this is a very interesting proposal, I'm also doing the data collection and analysis on a remote server, though personally I like the local server 'backup' of feeds for when the internet connection goes down (alarmingly regularly in some cases). I agree even without this there are some advantages of the pi over the nanode. Olinuxino is an interesting prospect if only for its greater openness.

Just spoke to Trystan about this, it seems they've not had problems with SD card corruption using a lower cost Sandisk than the one I'm currently using - which is positive. He also noted that v5 of emoncms executes far fewer mysql queries which may help. One suggestion he made was that perhaps all logging for apache, mysql etc should be turned off by default with documentation on how to enable them for troubleshooting.

With wear leveling on an oversized (8GB) card should be lasting a good deal longer - if we're looking at 2 years+ as people on the raspberry forum suggest, that wouldn't be a problem. In terms of Sandisk SDs only the 'Extreme Pro' specifically mentions wear leveling, there have been questions on the Sandisk forum about which models support wear leveling and specific requests for documentation but they have not received replies.

I understand the swap file is not used greatly under raspbian, but I'd like to start looking at that and perhaps moving to a swap partition on usb flash - as it's faster, cheaper, better documented (in terms of static wear leveling capabilities) and could wear out without immediately bringing the system down.

Thanks for you replies, I recognise that my use case - multiple geographically disparate installs without local support - is a bit marginal, so I really appreciate you taking the time.

As an update to this I have just modified my DC DC converter and wired in a chopped off micro usb to power the Pi. So I have 2 connections. One is plugged into the PI and the second is plugged into the USB hub. I have a wireless Wifi, a pen drive and a hard drive all on the hub. It has all booted up without any problems and is running as I type this. I will report back if I have any long term problems. It suggests there is a market for a Wallwart  with 2 leads, 1 a plug and 1 a microUSB.

Or maybe a small junction box with multiple connections.

Regards

Ian

Ian, I bought one of these which works quite well.

Paul

Hello, As Matt and Jerome mentioned above I think the best way to start is to disable all the logs as default, apache access log, apache error log, mysql, pi gateway script log, this should reduce writes as much as possible and preserve precious disk space, then as you mentioned Jerome have the option for remote forwarding only without any local inputs being written to, although I think this happens already if no inputs are configured (apart from the initial write that creates the input). We may be able to reduce further writes with continued optimization of input processing to reduce number of mysql queries which will be good for performance in general.

There is also the complicating factor of what happens when a pi is powered up and down without a full halt, restart, could the corrupted cards that are going after a short time be due partly to this? We corrupted a card here today by accidently shorting the pi 2 or 3 times. The SD card did work again though after a fresh image was put on it...

Jerome: I thought Id remove the default logging from the php gateway script tomorrow, I wonder whats the best way to configure it to turn logging on if needed? shall we work on this and then update the documentation with steps for turning off all logs as default and enabling only for troubleshooting purposes. Id be happy then do an an updated image with these changes.

Trystan

Hi Trystan

My 2 SD failures (after several months in each case) occurred on a Pi that has never been powered down as far as I know. Running from a battery solar system and the controller logs have no power shutdowns recorded. 

Regards

Ian

Hi Trystan,

Data corruption due to power loss is an interesting point, I was looking at that very issue yesterday when assessing whether to turn off Journaling on the ext4 file system - which would reduce writes but make the data far more vulnerable in the case of power loss. I was also reading that types of static wear leveling can make the data more vulnerable on power loss but I don't think these apply in our case, very readable explanation of some of the issues and tested cards here.

I can't be assured that pis haven't had the power pulled, but on 2 of the cards re-imaging the cards gives disk errors and on a third the SD fails again after a few hours, which makes me think it's a hardware issue rather than just dataloss. None of this is more than anecdotal though I'm afraid...

Mx

Made some progress on disabling all logs as default, just creating a new ready to go image for download at the moment.

So for existing installations:

Turning off apache logs

I've documented all the steps in the raspberry pi build from scratch guide here http://emoncms.org/site/docs/raspberrypibuild

There are 3 different files that need to be edited to turn off all apache logging:

1) In apache2.conf:

sudo nano /etc/apache2/apache2.conf

Replace the ErrorLog line:

ErrorLog ${APACHE_LOG_DIR}/error.log

with:

ErrorLog /dev/null

Comment out the line:

# LogLevel warn

2) In /etc/apache2/conf.d/other-vhosts-access-log

sudo nano /etc/apache2/conf.d/other-vhosts-access-log

Comment out:

# CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

3) In /etc/apache2/sites-enabled/000-default

$ sudo nano /etc/apache2/sites-enabled/000-default

Comment out:

# ErrorLog ${APACHE_LOG_DIR}/error.log
# LogLevel warn
# CustomLog ${APACHE_LOG_DIR}/access.log combined

Restart apache to complete:

sudo /etc/init.d/apache2 restart

Turning off rfm12piphp logging:

I've updated the bash script to take an argument that turns logging on and off so the first step is to download the latest raspberrypi module from github

cd /var/www/emoncms/Modules/raspberrypi
git pull origin master

to update exisiting module installed via git.

Next re-copy the rfm12piphp bash script to /etc/init.d:

sudo cp /var/www/emoncms/Modules/raspberrypi/rfm12piphp /etc/init.d/
sudo chmod 755 /etc/init.d/rfm12piphp
sudo update-rc.d rfm12piphp defaults

To complete restart the rfm12piphp script:

sudo service rfm12piphp restart

Check that the logging is all off

If you call any of the following they should either report a missing file or if the file already existed before turning logging off, no new entries should appear in the terminal window:

tail -f  /var/log/rfm12piphp.log

tail -f  /var/log/apache2/access.log

tail -f  /var/log/apache2/error.log

tail -f  /var/log/apache2/other_vhosts_access.log

You may want to delete any exisiting logs to save on disk space:

sudo rm /var/log/rfm12piphp.log

Both the apache access log and the rfm12pi log are frequently updated logs so turning these logs of should reduce disk writes significantly and also freeup disk space. 

Jerome, would you be able to implement this feature in the python gateway and document how to update and turn off the logs?

Il update when the ready-to-go image is up.

Trystan

Hi Trystan

Which directory do you need to be in when you do the git pull origin master  command?

Regards

Ian

a yes:

cd /var/www/emoncms/Modules/raspberrypi

Il copy that in above

> Jerome, would you be able to implement this feature in the python gateway and document how to update and turn off the logs?

In the init file, there's this line:

DAEMON_ARGS="--logfile /var/log/rfm2pigateway/rfm2pigateway.log"

If replaced by

DAEMON_ARGS=""

the logs go to standard output. I suppose this is /dev/null if launched as a service. We could check that.

Besides, the .py gateway is designed with more granularity in the logging. I was planning to add a multiple-choice selector in the GUI to allow for the choice of the logging level : DEBUG, WARNING, INFO, CRITICAL, NO, etc.

I haven't done it yet because it may be confusing since the php gateway does not have this feature. We could consider all messages in the php interface are of level DEBUG and modify the php script accordingly...

Maintaining both scripts not only means  more work but sometimes it can slow down the pace, as each modification will have to be done for both and were the logic differs, it can be painful.

The other reason it is not done yet is that I'd like to propose a new design for the Raspberry Pi module, with

- a part for the radio config and emonGLCD sending and other general settings

- a part for the emoncms target servers (any number of servers, local or remote treated the same way, local being just a particular case)

- a window for the log to be displayed, as you suggested

I think this is worth doing in a dev branch.

I should try to produce a pseudo-screenshot of what I'm thinking of and we could discuss this before I start coding.

I'm being out of topic. Where should we discuss this ?

Hello Jerome, I've opened an issue in the raspberrypi github repo to discuss further https://github.com/emoncms/raspberrypi/issues/30

We have tried a lot of different SD cards incl. class 10 SanDisk on a non gateway Raspi based emonCMS. We want to have local dashboards so it will not be sufficient for us to use the gateway approach. After app. 3 month all SD cards where dead exept one:

Transcend Ultimate-Speed SDHC Class 10 UHS-1 8GB

After changing the polyfuse I am able to run a 3.5" 500mb usb hard disc without a need for extra PSU.

Petrik: thats interesting, could you share some pictures of what you did?

Changed polyfuse to 3A version, if I remember correctly the original is 700mA - polyfuse resistance increases when current limit is reached which in practise means that voltage is dropping too low for disc to operate. With 3A polyfuse disc seems to be operating as the polyfuse current limit is not reached.

Bought from ebay link below, these are a bit bigger than original - but could be fitted.

http://www.ebay.com/itm/350680862214

Here is a piccie...