Archived Forum

Hi Glyn,

I agree this would be a good step, it's something I have wanted to look at for awhile but never seem to get to. Although logging does seem ok most of the time, I have seen instances where a fault condition has filled the log partition in ram and a quite serious consequence of that is it can stop the cron tasks running. I haven't looked into whether it can be avoided or even exactly why it occurs yet. But it does mean a simple cron based rotate or housekeep may be vulnerable unless unaffected by a full log. I suspect cron stops if it cannot output so a simple redirection to /dev/null maybe all that's needed.

My first thought was "logrotate" too and had also wondered if it would be possible to "rotate" the logs out of ram on to the sd card at regular interval so as to not lose all logs at reboot without impacting the SD card life too heavily, if the logs are kept small by rotation the current log file may also be able to be persisted at shutdown using rsync.

It maybe worth increasing the size too, 30mb isn't much, emonhub alone can use up to a max of 10mb as it's in-built log rotation has a file size of 5mb each. 

Paul

How long do you think would be e decent time to keep logs for? I think it's good having debug turned on my default since users and quickly look at the log in Emoncms to check all is working however this degug log does not need to be kept longer than a few days. What do you think?

What's needed is some sort of service to delete logs to keep the size of /var/log below (80-90%?) of being full. Could logrotate perform this task? I will investigate. 

I think size is a better measure than time, at "WARNING" loglevel emonhub could run forever with less than 6 lines if no problems are encountered, where as any runaway issue could completely fill the 30mb in no time at all. if the logs are managed by size we can better estimate the total posible size for the ram partition, and if the file can be rotated out of ram to the sdcard the max amount of ram required is halved. 

Doing it by time alone could mean "too short" and useful data is overwritten too frequently and "too long" leaves the door open to maxing out the logs too easily. rsync could allow "back-ups" daily or hourly even. So if we estimate a total file size of ~50mb, we should set 30mb ram aside, rotated with upto 30mb on the sdcard plus an additional "rsync" back-up of upto 30mb on the sdcard. OR at least 60mb of ram with zero persistence.

Paul

What's needed is some sort of service to delete logs to keep the size of /var/log below (80-90%?) of being full. Could logrotate perform this task?

Hi Glyn,

Yes. Logrotate can do its job based on log size as well as age.

From the page linked below:

You can specify a file size that logrotate will check when determining whether or not to perform a rotation by using the “size” command. The format of the command tells logrotate what units you’re using to specify the size:

size 100k
size 100M
size 100G

The first example would rotate the log if it gets larger than 100 kilobytes, the second if it’s larger than 100 megabytes, and the third if it’s over 100 gigabytes. I don’t recommend using a limit of 100G, mind you, the example just got a little out of hand there.

The size command takes priority over and replaces a rotation interval if both are set.

www.rackspace.com/knowledge_center/article/understanding-logrotate-utility

Thanks for the link Bill, 

A size time combo would be ideal and the "postrotate" function to call a script at rotatation could switch to RW mode move the older file out of ram to sdcard and set RO again.

Using rsync could be combined with the ntp-backup so that all the logs are "rsync'd" every hour and at shutdown whilst the time is backed up with minimal extra code or disc writes.

Paul

Thanks for your help. I think I've got this running nicely today.

I've configured logrotate to check all files in /var/log hourly and rotate is any log file > 1Mb. Four rotations are kept file.log.1 etc. After four rotates (size taken 4 x 1Mb = 4Mb) the log are deleted. I've also increased the max size of /var/log in tmpfs to 50Mb. Not all of this is allocated at once, apparently it's dynamically allocated in the RAM as it's needed up to a max of 50Mb. Pi2 has 1GB so we should be fine :-) 

The config for logrotate is as follows: 

# check size of log files hourly 
hourly
# default is daily move cron script to make this happen hourly:
# sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

# keep 4 rotations  worth of backlogs (size x 4)
rotate 4

# Max size of log file (rotate log file when this size is reached)
size 1M

# Truncate  the  original log file in place after creating a copy
copytruncate

# append rotated logs with date 
dateext
dateformat -%Y%m%d

# log owners
su root

# Include all logs including 1 level deep subdirectorys
/var/log/* {} 
/var/log/*/*{}

logrotate.conf has been added to ~/emonpi git repo and symlinked to /etc/logrotate.conf for easy updating if needed.

https://github.com/openenergymonitor/emonpi/blob/master/logrotate.conf

If you want to install on your setup:

sudo apt-get install logrotate
sudo rm /etc/logrotate.conf
sudo ln -s /home/pi/emonpi/logrotate.conf /etc/logrotate.conf
suco chown root /home/pi/emonpi/logrotate.conf 
sudo rm /etc/cron.daily/logrotate​ /etc/cron.hourly/logrotate
sudo touch /etc/cron.daily/logrotate​
​sudo ln -s /home/pi/emonpi/logrotate /etc/cron.hourly/logrotate​

When setting up set -d to debug (verbose logs and don't edit) or -v (verbose logs AND makes changes to disk) in /home/pi/emonpi/logrotate simlinked cron file

Hi Glyn,

Something to be aware of regarding logrotate.

Taken from:
http://unix.stackexchange.com/questions/29574/how-can-i-set-up-logrotate-to-rotate-logs-hourly

Depending on your OS. Some (all?) Linux distributions have a directory /etc/cron.hourly where you can put cron jobs to be executed every hour.

Others have a directory /etc/cron.d/. There you can put cron-jobs that are to be executed as any special user with the usual cron-settings of a crontab entry (and you have to specify the username).

If you use either of these instead of the standard log rotatation script in /etc/cron.daily/ you should copy that script there and cp /dev/null to the original position. Else it will be reactivated by a logrotate patch-update.

Thanks Bill, good find.

So given that I've moved logrotate from cron.daily to cron.hourly, what you're saying is I should put an empty logrotate file back in cron.daily to ensure it's not reactivated by a logrotate patch update?  

Spot on.

touch /path/to/filename will give you a zero length file.
 

Paul (pb66) mentioned above about the possibility of writing the logs to the SD card at shutdown. Don't know if you are considering this or not Glyn, but if so you may want to take a look at Ramlog and how they achieve it.

Description
Ramlog act as a system daemon. On startup it creates ramdisk, it copies files from /var/log into ramdisk and mounts ramdisk as /var/log. All logs after that will be updated on ramdisk. Logs on harddrive are kept in folder /var/log.hdd which is updated when ramlog is restarted or stopped. On shutdown it saves log files back to harddisk so logs are consistent. Ramlog 2.x is using tmpfs by default, ramfs and kernel ramdisk are suppored as well. Program rsync is used for log synchronization. 

I briefly tried Ramlog whilst editing the low-write guide and it works really well

Paul

Hi Paul,

I suggested Ramlog back in June of this year, and was more or less blown off.
http://openenergymonitor.org/emon/node/10773#comment-31091

Hope you have better luck than I had!

As you mentioned, it works well, and is free.

Bill

Thanks for the link ramlog looks like a tidy package. However, IMHO I can't see benefit to have the logs saved between boots. I actually like starting with the fresh logs on first boot, occasionally I cycle a reboot just to achieve a nice empty /var/log!

If a problem which we are trying to debug by looking at the logs can't be replicated between boots then it's probably not a problem! 

Agree, on a serious busy web server having log preservation is valuable but many emonPi's will just be used just as an Emoncms forwarding gateways. Having verbose logs is very nice for the user to see via Emoncms viewer and good for us if we have to debug user problems however the logged data quickly looses its value over time.  

If a problem which we are trying to debug by looking at the logs can't be replicated between boots then it's probably not a problem! 

I came to the same conclusion...
Do you think that we need to include logrotate in the emonbase low-write too?

Paul

The same pre-built SD card image will work for both emonPi and emonBase 

I was meaning the self build guide

Paul

Ah yes, good idea. Instructions are above and in the emonPi install script: https://github.com/openenergymonitor/emonpi/blob/master/install#L124.

Config file may be tweaked as I'm still testing but I think the install procedure is good. 

Thanks.

Not sure if you're talking about os logs or emoncms logs.

Anyway, in Lib/Emonlogger.php there is a log file limit restriction hardcoded to 250mb.

Both! Good to heat there is a limit on emonlogger. However the 50Mb tmpfs /var/log limit on the emonPi would be hit long before the hardcoaded restriction. This discussion only relates to running emoncms on Raspberrry Pi with a limited size /var/log mounted in ram as tmpfs.  

Would it be a big job to move the emoncms.log file to /var/log ?
If it were moved to that location, it would enable emoncms logging to remain running, despite being in low-write mode as it would be writing to tmpfs (if mounted). Also, logrotate could then do the emoncms.log housekeeping, rotating and keeping the log size within an allocation.

Paul

Is the emoncms.log not already in /var/log? It is on my emonPi 

No, as a standard installation, its;

/var/www/emoncms/emoncms.log

Are you referring to the emonhub log?

Paul

The "$log_filename" has been set to /var/log/emoncms.log in the default.emonpi.settings.php file but in the "default.settings.php" file it uses the installed directory name and "/emoncms.log". It would be good to see a common location of /var/log/dirname/emoncms.log. The extra directory name based folder level is good housekeeping both for devs with more than one instance and for keeping rotated logs in one place.

Paul 

I agree Paul, I suppose that by changing the log location from /emoncms/emoncms.log in settings would have implications for the log viewer on page Administration > View log

(not at home right now so can't check)

Paul

I don't have a local emoncms right now so I can't check either but "in theory" if it works for emonPi with just the setting changed I would expect It to be good .

Paul

Just tried moving the emoncms.log to to /var/log and yes it sort of works!

Well it works when you manually create the log file and set it's permissions, but of course because it's held in RAM, it doesn't survive a reboot, and users then need to re-create the log file and again set permissions.

touch /var/log/emoncms.log
chmod 666 /var/log/emoncms.log

I need to look how this is dealt with in emonPi, when I get more time.

Paul
 

This is exactly the issue I had with emonhub.log when it was first written. The RO wiping the logs did create a problem as the /var/log permissions can prevent a emon+++.log being created automatically and the folder doesn't get created.

In emonhub the solution was to add the emonhub subfolder creation to the init script, so that at every restart it checks for the /var/log/emonhub folder and creates it if it's not there and set the folders permission for emonhub.py to be able to automatically create the logs and rotate as required.

Since the log and subfolder deletion only occurs to the read-only file system the same can be done by the feedwriter script in emoncms as only the "low-write" mode will usually be used with a RO file system.

For consistency all the install guides should just tell the user to create the folder and add permissions as usual, this will remain in place for normal read-write installs and be replaced if required by the feedwriter init script if RO.

Only if the full version (no feedwriter) is used on a read-only OS will there be an issue. But even that would be covered if a complete ramlog/logrotation scheme was used.

Paul

emonPi (the new image) sets up the logs on boot, see rc.local: https://github.com/openenergymonitor/emonpi/blob/master/rc.local

Thanks Glyn. I'm almost there with this now. And will let you have some proposed changes in the next few days to consider.

Paul

emonpi default is /var/www/emoncms.log 

https://github.com/emoncms/emoncms/blob/stable/default.emonpi.settings.p...

Found this thread after realising that my base station using the pre-built SD card (oldish one from last July I think?) periodically needs rebooting and that the reason was the /var/log had filled up.

Most of the logging was around firewall stuff, multicast igmp packets that were OK on my network but were logged as blocked by UFW. Fair enough.

I looked and it seems that we already have logrotate installed, but the default configuration breaks because of the permission on /var/log being too permissive. If you change the permissions to the expected values (755) then the standard logrotate configuration seems to work. Wouldn't it be better to keep this, and add new config files under /etc/logrotate.d for the emon-specific configs?

(BTW I think generally I fall into the camp of preferring a read-write flash image and reducing writes than having a lot of special-casing for read-only, but I'm also trying to stay close to the 'stock' config for better support, etc.)