While doing a server update and upgrade today at 18:40 (uk) to emoncms.org, I needed to reboot the server, unfortunately the server hasnt come back online. I've contact the technical support at the server hosting and they are looking into it. I will update with progress as soon as I can.
»
Re: Emoncms.org server outage [SOLVED]
The server company are running hardware diagnostic tests. Im waiting for the conclusion
Re: Emoncms.org server outage [SOLVED]
Ok, the machine has been upgraded with a DRAC enterprise card which gives me virtual KVM access to the server, its getting to the boot os selection screen where you can either boot normally into ubuntu which fails or boot into recovery mode which gets a bit further before stopping at a warning that the RAID is degraded, I've sent another ticket to the server company and will update as this progresses.
Re: Emoncms.org server outage [SOLVED]
Progress, after booting in degraded mode, I now have normal ssh access to the server and it all looks fine. Now I need to work out why one of the disks was degraded either fix it or move the whole server to a new one.
Re: Emoncms.org server outage [SOLVED]
Hi Tristan,
I am running emoncms on a Raspberry Pi, and forwarding to emoncms.org
Last night, I noticed that emoncms on the Raspberry Pi was sometimes reporting that the RFM12 to Pi interface was not running, and the inputs were not updating - sometimes for several tens of minutes.
I eventually tracked down the rfm12piphp.log and realised that it was full of errors trying to open the socket to emoncms.org (and I then found this thread on the forum!).
Is there anyway in which emoncms could be made more responsive in this scenario, as I lost a lot of data during the downtime?
Re: Emoncms.org server outage [SOLVED]
Same here emgeeaitch, it's not very often that emoncms.org goes down, but whenever it does, my system also grinds to a halt too.
I lost about 3hrs of local data, before I returned home and removed the API key.
Re: Emoncms.org server outage [SOLVED]
Trysyan,
Sorry to hear that you are having server problems. Emoncms is great!
One thought, it appears if you are using the RaspberryPI with emoncms it stops polling for data from the Arduinos if the *main* Emoncms server is down.
Would it be possible in a later RaspberryPI software revision to get the RaspberryPI (running emoncms) to continue to poll and log locally if the main emon server is down?
Best
Andy
Re: Emoncms.org server outage [SOLVED]
Yes I think we need to add a timeout to the rfm12piphp script, so that it doesnt keep waiting on the connection, Im not sure if this is an issue with the python gateway however?
To be safe, I've ordered and am now configuring a new server that has hardware rather than software raid and a few other improvements, double the disk capacity. I hope to get this up in the next few hours.
Re: Emoncms.org server outage [SOLVED]
The problem is the timeout connection in oemgatewaybuffer.py :
result = urllib2.urlopen(url_string, timeout=60);
either you change it with shorter value i.e 10 seconds or it should be done in a separated thread but it involves different management of the buffer.
Ciao.
Re: Emoncms.org server outage [SOLVED]
Yes a separate thread would be a good way to do it, Il add an issue to the new emonhub development repo to discuss it.
Server migration is going well, 45GB o 63GB has been copied, there's a lot of small files which makes it relatively slow.
Re: Emoncms.org server outage [SOLVED]
Getting very close now, just doing final testing to ensure everything is working as it should before I reconfigure the dns records to point to the new server
Re: Emoncms.org server outage [SOLVED]
Im happy to say emoncms.org is back online! sorry for the interruption
Re: Emoncms.org server outage [SOLVED]
and thankyou everyone for your patience
Re: Emoncms.org server outage [SOLVED]
Thanks for your efforts Trystan you could probably do with a kip now :)
Re: Emoncms.org server outage [SOLVED]
Thanks for all the time and effort you have put into restoring what is after all a free service. I'm sure you've probably got better things to be doing on a Bank Holiday weekend.
Andy.
Re: Emoncms.org server outage [SOLVED]
Servers are like children: they always feel sick during the night or in the week ,
;-)
Re: Emoncms.org server outage [SOLVED]
.....or when Glyn goes climbing mountains!
Paul