Archived Forum

Free up some RAM, for free....

Append "gpu_mem=16" to /boot/config.txt, this caps the RAM available to the GPU (since eMonPi is running headless the GPU should not be in use) and gives the OS some RAM back for free :)

Before:

root@emonpi:~# free -m

             total       used       free     shared    buffers     cached

Mem:           925        435        489         22          2        175

-/+ buffers/cache:        257        668

Swap:            0          0          0

After:

root@emonpi:~# free -m

             total       used       free     shared    buffers     cached

Mem:           973        424        549         17         19        166

-/+ buffers/cache:        238        735

Swap:            0          0          0

Firewalls, we all love to hate them, one way or another; but wouldn't it be great if you wanted to be able to reach your eMonPi while on the move, of course it would but that means setting up dynamic DNS and them poking about with your firewall, what fun.

Well here's some tricks to solve 1/2 of that; upnp for your pi...

There are a couple of components to making this a real sucsess, a binary that actually communicates to the router via uPNP, an init script to call the process with the arguments, and a cron entry to keep calling it every few mins.

Init script / cron details attached, the binaries can be installed with "apt-get install miniupnpc"

Happy uPNPing....

All in one update script;

Short and sweet, says it all really...

**EDIT: removed the rpi-firmware update since this relyably killed my pi image, I'll dig deeper to work out why...

**EDIT2: after spending a little time on this, I can see that the "commit=" argument used for mounting the data partition on the SD card is actually the issue, removing that makes it possible to upgrade the firmware to 4.4

I'll be modifying the update script to account for that and posting a new version in a short while.

**EDIT3: New script worked up, it works but needs some real world testing - ALL I check for is that the system upgrades, so maybe release an RC3 on the new firmware before a full release?

As part of one of the PHP updates you are asked if you want to keep the local version of some config - yes do that, dont replace it; Also I dont currently force acceptance of the kernel upgrade, so you will be prompted for that. I may be able to clean up / automate both of these if you want the update script used here to be included on production builds, but at the moment I'm only aiming to give you the building blocks to intigrate into the web based update method.

The problem with the new firmware appears to be syntax related; "commit=120" on the line for the data volume in /etc/fstab is the issue. Reading from here: http://www.softpanorama.org/Internals/Unix_filesystems/linux_ext2_ext3.shtml it looks like you are trying to use an EXT3 feature on an EXT2 filesystem, so removing the argument from /etc/fstab should have 0 effect (other than allowing the new 4.4 kernel to boot fully).

SSHD gets broken as part of the update too (specifically on Pi3 using on board wifi only) so you will need to append this to the bottom of your /etc/ssh/sshd_config:

IPQoS cs0 cs0

See here for more information (long thread): https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=138631

**update script now does this for you...**

Final side note - updating the rpi-firmware will leave behind additional folders in /lib/modules, you will need to clean up the old ones (after a rebboot) to save sdcard space.

One caveat. UPnP has some security risks associated with it.
Here's an article that explains some of them.

More info:

www.howtogeek.com/122487/htg-explains-is-upnp-a-security-risk/ and

www.root6.com/support-2/tips/upnp-it-was-always-a-bad-idea/
Excerpt from this article:
it turns out that some routers have UPnP enabled on their Internet-facing port!
That’s right; you send the correctly formed UDP-discovery packet at the Internet side of the router and along with being able to open ports you can query the router for lots of details about itself, allowing you to better tailor your attack for other known vulnerabilities of that specific model. HD Moore (of Metasploit fame) has had a cluster of machines probing the public Internet to see how many public-facing IP addresses had UPnP enabled and it turns out around 2% of hosts respond to WAN-borne UDP discovery packets. He repeated his scan weekly for six months and those eighty-one million routers remained reasonably constant.

In case your wondering if your router is vulnerable you can find a list of affected models here, and you’d be surprised how many big-name manufacturers are there.

Here's a quick test to check your if your router is vulnerable to UPnP exploits:
https://www.grc.com/x/ne.dll?bh0bkyd2

Bill,

I totally agree, it does have some security concerns, as does opening up access directly to your pi from the internet, but if the options were not here, nobody would have the choice.

but if the options were not here, nobody would have the choice.

True. I just wanted to point out the security issues for those who might not be aware of them.

Hi Andy,

I built the current emonPi image. 

Thanks for sharing your modifications. 

Memory Tweaks

Setting gpu_mem=16 sounds like a good idea, are there any other side effects to this that you know of other then GUI performance which is obviously not an issue for us. I think I might have read somewhere about stability issues. 

Upnp

Nice trick, although I think I'll leave this out from the stock emonPi image to avoid any potential security issues. 

Firmware Update Issue

I would very much like to make the standard emonPi image compatible with the latest ras-pi firmware. Please could you give some more details on the changes you made to make this possible. After these changes to the way partitions are mounted would the new changes be backward compatible e.g. users running older firmware 

Log rotate and tmpfs /var/log

This is an issue I wonder if you could help shed some light on: Even though logrotate seems to be working I've noticed that /var/log which is mounted as a tmpfs in the RAM fills up after a month or two of operation. Have you noticed this? This is one issue we need to fix before we launch the new image (currently in RC2) 

Glyn,

Memory Tweak - not had any issues with this on my own system, mine runs a radio repeater for weeks / months at a time, stability hasn't been an issue here, it does not run a GUI of any kind so the extra ram is handy; I've run like that on Pi A (the original one), B, B+, 2B and now the 3 - 3 is brand new of course so no real feedback on that.

uPNP - You might consider adding that but disabled by default - installing the demon is perfectly safe form a security point of view - the security concerns are more about how resilient your Apache server setup is and how secure the users router is. As has been pointed out - it does need to come with a health warning.

Firmware update - found / fixed - I'll be posting about that shortly, its a simple fix and my update script should take care of it. (more to follow on that). EDIT: See above, update script updated and information added, firmware update works but breaks SSH over wifi - fix now included in the update script.

LogRotate - I'll take a look at your setup, my eMonPi is all of a day old at this point - so I'd obviously not run into that.

Other advice - delete the ssh keys from the base image before you prep it for download - so every eMonPi RC2 download has the same keys in it at the moment, not a hot idea. EDIT: Still working on this because the base image doesnt re-create the keys when they are missing - not a big deal, I'll write an init script to take care of it.

Andy

Syslog / LogRotate

So one early issue, the syslog is filling up with garbage, due to a bug in the base raspbian system;

Edit the /etc/rsyslog.conf and comment out the last 4 lines that use xconsole, the log balooning should get taken care of (see below). There is a little more work / testing to be done on logrotate to see what it is / isnt doing, but that will take a couple of days for me to check it out.

#daemon.*;mail.*;\

#       news.err;\

#       *.=debug;*.=info;\

#       *.=notice;*.=warn       |/dev/xconsole

LogRotation;

Once you fix the syslog issue (above) that should reduce the crap being logged to /var/log/messages, it was probably that file / var/log/messages that has been filling up the /var/log tmpfs volume; moving on...

/etc/cron.daily/logrotate was found to be empty, it should look like the one attahced.

/etc/logrotate.conf - small changes here to capture ALL logs in the /var/log/ directory (including the ever expanding messages output) and to make sure the rotation happens daily - change daily if you want weekly / monthly etc. (again see attached).

/etc/fstab required an extra line for the filesystem where logrotate wants to keep its status, since logs are non-persistent, this can live on tempfs too.

There is a Pi3 unique breakage introduced when you fully update the firmware and base OS, just what you really want in your life.

SSHD gets broken, but ONLY when using the on board WiFi adapter and NO ethernet, this was a bear to find, but still here is the fix;

Edit your /etc/ssh/sshd_config and append the following to the end of the config;

IPQoS cs0 cs0

Restart SSHD (or reboot the eMonPi) and SSH is now forever cured.

0 side effects to any other process, and will have no apparent effects on other pi models / wifi cards etc.

EDIT: Update script (re-attached above) now fixes this as part of the update.

Memory Tweak - not had any issues with this on my own system

Same here. I run all of my Pis (4 of them) with this same tweak and have noticed no ill effects. I've seen uptimes of over 270 days. (reboot caused by mains power loss)

No worries about this one, Glyn.

SSH Keys - Auto Generation (the idea here is that we form a service that always starts to create the keys if they are missing. That way the new images can be built and setup for download without any keys in them, and each new install will have new SSH keys.

1. install the files attached in the locations that I hope are obvious form their names

2. chmod them to 500 (read / execute by root)

3. systemctl daemon-reload (tell the system the files have changed)

4. systemctl enable emonpi-sshkeygen.service (enable the service)

Now you can remove the SSH keys and they will be re-created at boot time.

Glyn,

I'd mentioned this a few months ago, but at that time you decided to go with the stock log rotation.

Ramlog writes the log files to ram, saves them to disk at shutdown and can be configured to save the logs at a user selectable interval. It's a bash script, and is free. (as in beer) Sounds like it might be able to solve your tmpfs memory issues.

Fom the author's web page:

Ramlog act as a system daemon. On startup it creates ramdisk, it copies files from /var/log into ramdisk and mounts ramdisk as /var/log. All logs after that will be updated on ramdisk. Logs on harddrive are kept in folder /var/log.hdd which is updated when ramlog is restarted or stopped. On shutdown it saves log files back to harddisk so logs are consistent. Ramlog 2.x is using tmpfs by default, ramfs and kernel ramdisk are suppored as well. Program rsync is used for log synchronization.

Ramlog is intended to be used on netbooks or notebooks with SSD or HDD drives.

In case ramlog is restarted (by default it is one time per day), directory /var/log.hdd is synchronized with /var/log using rsync. Frequency of the automatic log saves can be controller via cron, by default, the ramlog file is placed into /etc/cron.daily

www.tremende.com/ramlog/

Hi Andy,

Thanks so much for you help. Yes, it was the /var/log/messages that was filling up the partition. I think log rotate was working to rotate the emonpi log files (at least on my version of the image) e.g. emonhub.conf etc. However /var/log/messages was not being rotated and this caused this issue.

I have made the change you suggested to logroate and it now seems to be working well and inclusing all logs. Obviously it will take a couple of months to be able to tell for sure. 

I have added regenerating SSH keys to the emonpi factory reset script
https://github.com/openenergymonitor/emonpi/commit/f5ea0de800e9c463905c6...

I will include the fstab, SSHD fix and memory tweak in the next image. I will have the image up for download tomorrow. 

You suggested changing logrotate from:

#!/bin/sh

/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status 
/etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log

to

#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

What's wrong with doing as I was previously doing: moving logrotate.status and logrotate.log to /var/log to ensure it would work on a RO FS. Would doing this negate the need to mount /var/lib/logrotate as tmpfs

What's wrong with doing as I was previously doing: moving logrotate.status and logrotate.log to /var/log to ensure it would work on a RO FS. Would doing this negate the need to mount /var/lib/logrotate as tmpfs

Nothing - I hadnt realised that was what you were doing, you can trim the changes for logrotate to the very bottom of the logrotate.conf file - change the bottom line to remove the .log parts of the catch all, this is why it didnt rotate "messages".

For anyone following this thread using full wildcard is a bad idea:

The manpage for logrotate says

   Please  use wildcards with caution.  If you specify *, logrotate
will rotate all files,
   including previously rotated ones.  A way around this is to use the
olddir directive or a more exact wildcard (such as *.log).

I found this post which explains things well:
http://hudolejev.blogspot.co.uk/2012/06/wildcards-in-logrotate-configura...

The current logrotate.conf symlinked on latest emonpi image has been changed to

/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages {}