If your using emoncms.org to store your data, you have probably noticed it has been quite slow recently and not very stable.
Here's a picture of the server load (1 minute) as measured with the command uptime over the last few months
Although its been growing for some time above server loads that would be considered healthy its really in the last 2 weeks that its gone through the roof. We're now measuring average sever loads of around 80. When we moved the server which is now on a virtual machine to a dedicated server which had 4GB ram and 4 cores the dedicated server reached a load of about 20 and became completely unresponsive so it was put back on the virtual server.
Emoncms is hosted on BigV http://bigv.io/ and a few people have suggested that even thorough the virtual machines spec is lower than the dedicated server, the virtual machine may be being given bursts of capacity above its stated spec, so thankyou bigv!
Its not clear yet why the load has grown in the way it has in the last few weeks, the rate of increase of active feeds has actually slowed due to the slow downs.
I've been working on trying to solve this problem with Ynyr Edwards a good friend of mine and a software developer with many years of experience. Ynyr joined Glyn and I and is helping with both software dev and running the shop as Glyn mentioned in the blog post: http://openenergymonitor.blogspot.com/2013/10/new-office-space-and-new-units.html.
So far we've started to try moving some of the feed and input meta data information to redis which is in memory rather than on disk database, the branch of emoncms with redis in it is available here: https://github.com/emoncms/emoncms/tree/redismetadata
We havent seen a great improvement with redis yet, I think it appears to have moved the load from waiting io to the processor but not sure.
Over the next day or so there will be some downtime and strange behavior on the server as we try and better understand and solve the problem.
A few things on the list to try are:
- upgrade the vm to 4GB ram and 2 cores
- redis based input queue so that apache instances can close faster, connections are not left waiting.
- split the load out to multiple vm's, maybe completely separate instances of emoncms.
- try nginx web server
- optimize apache? does anyone have any experience with this.
I will keep this post updated and would be happy to hear suggestions if anyone has any.
Re: Emoncms.org Status [Solved]
Ok that was surprising. I've just changed the apache2.conf setting of maxclients from 150 to 100 and then 50 and the load as dropped right down to below 1. Maybe it was just set too high and was swapping. Im now going to increase it until the performance starts to degrade and work out if a lower value is causing requests to queue and timeout.
Re: Emoncms.org Status [Solved]
With MaxClients of 80, load is ~16, free -m [+- buffers 473].
vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
8 0 52680 345064 29168 196552 1 1 42 3174 40 44 70 18 1 11
5 5 52680 346368 29172 196556 0 0 1 1457 448 9388 71 29 0 0
19 0 52680 343132 29216 197100 0 0 191 3964 539 5209 51 11 0 38
Re: Emoncms.org Status [Solved]
Ok! progress!
With 1GB of ram and 1 core emoncms could only handle 50-80 simultaneous apache2 clients before running out of either CPU or memory. Reducing the MaxClients helped the load but meant that a lot of connections are being queued and so the response time of emoncms.org becomes really slow, especially if the connection is not kept alive and you cant keep all the connections alive as an open connection reduces the number available for other requests. (If I understand correctly) Testing with ab showed request times of 4s.
Upgrading the bigv vm to 2 cores and 4gb of ram and then upping the maxclients to 150 again meant that the apache queuing is much reduced. There are now 88 apache2 clients and the response times are an amazing 10ms right now. Thats almost as fast as emoncms running localy here on my own laptop which has 4 cores and 8gb of ram (60-100ms for a timestore graph load vs 44ms on my local machine)
Re: Emoncms.org Status [Solved]
Whatever you've done is working well on my setup. I noticed yesterday morning that emoncms.org seemed to have been turbocharged. It seems just as fast as my local timestore instance now despite my emoncms.org account still being on phptimeseries.