Archived Forum

It's not just you. I've emailed G&T.

I can confirm its down, Im looking in to it.

An update:

The server (ubuntu virtual machine) went in to read-only mode  after what looks like a disk IO error, this is the kind of error logged to dmesg

[169443.693957] Buffer I/O error on device vda2, logical block 25230031
[169443.693957] lost page write due to I/O error on vda2

Ending with:

[169476.068862] EXT3-fs (vda2): error: remounting filesystem read-only

I can log in fine and the data appears to be there , I also have a backup of all feed data that's just over 24hours old

The advice I've been given is to reboot and let the server run a filesystem check.

I want to be extra safe so before I do that and while I still have read access to the main emoncms.org server Im going to start up a second server and copy over all the emoncms data and check that its ok.

Luckily data transfer speeds appear to be much faster than usual and I've just successfully transferred all the mysql meta data tables (users, input, feeds, dashboards etc) and timestore data to the new server, Im now going to transfer all the mysql data.

Il then check that the second server works as expected and then run the filesystem check on the first and hopefully we should be back up.

Will update again shortly.

Trystan

are the backing disks for the vm on iscsi if so beat the vsp with a large stick

I'm making good progress, We now have a copy of emoncms.org running on the second server, Still waiting on the mysql data to transfer but the timestore data and mysql meta data tables are working after a repair to the feeds table which did not close properly.

Good stuff its getting up again - felt this morning a bit disabled without being able to see how the heating system is operating. Emoncms is a great tool in a house with several heatpumps to monitor that everything is working - at this time of the year the outdoor units may get frozen due to sleet and rain, something that can be immediately seen on the emoncms graphs.

Everything is up and running now on the new server, which I'm also hoping will be a bit faster. Data all looks intact. Just doing final checks.

Could you let us know when it should be accessible from internet - at least right now browser does not find a web server at www.emoncms.org

Still down for me too??

Ok I think we're good, emoncms.org is up!!

It is a different ip address this may cause problems 213.138.100.101, Raspberrypi's should be ok, and nanode's should reset after 10 failed attempts and get the new address.

might have spoken too soon there

Its always better to be aware than not to know whats happening - so its never too soon.

Let the force be with you in resolving this issue

Thanks Petrik, it appears to be back now. 600 feeds back online out of 3000 so far and rising.

639 feeds back online, your feeds are still off though Petrik.

I see mine are back online, I had to hardcode the /etc/hosts file to the new IP as the oem_gateway was still responding to the old address. noce this is that it was buffering the data and started buy updating the missing data, unfortunately it only kept the last hour of data and as emoncms was down for 21+ hours we have missing data. What I also see is that if there is missing data emoncms does a tend rather than just making zero's

But good work Trystan,

Tnx for your effort but, we have to use the IP?

in case it say to point to /emoncms where i can see my dashboard but my feeds seems inactive.

Probably this is why i have the raspi as gateway and it point to emoncms.org and not the ip and i have no path in the config. should i change it in this way?

[[emoncms_remote]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = emoncms.org ---------> 213.138.100.101
        apikey = 1axxxxxxxxxxxxxxxxxxxxxxxxxx475a2
        protocol = http://
        active = True
        path = ------> /emoncms

tnx again

http://www.emoncms.org does not seem to point to a HTTP server does not at least from here seem to be on line.

anyhow http://emoncms.org does seem to work

It may be dns caching http://www.ubuntugeek.com/howto-clearflush-dns-cache-in-ubuntu.html

If its easy to change and it works then go for it, but Id hold off for a little while. The new server is being a bit flaky right now. Im going to restore the other server so if this new one does not work out and this ip address issue is a big problem then we can switch back, or Il set up forwarding from the other server, The hosting company can also temporarily swap the ip addresses, which may be an option.

2300 of the 3000 previously active feeds are now online, server loads are higher than I hoped on the new server, lets see if they settle in the next few hours.

2700 of 3000 previously active feeds online.

Any remaining monitors posting to the old ip address should now be forwarded to the new address. I've installed a tool called rinetd on the old server to forward the data. The old server was recovered successfully with everything intact.

Hey Trystan,

something is not right with my feeds, they are reporting way to high, specifically the kWh and kWhd feeds, please can you take a look and even delete them if you can't fine the issues . For the 18th & 19th

Shot

Never mind, I have deleted and recreated my feeds, lost all historic data...... oh well.

Is it possible that something is still not right with emoncms.org server. Its very low to access and I often get an error: server not found when trying to access my dashboards ?

Yes I think your right, the load has been all over the place over the last few days.

One of the difficulties at the moment is that its running on a virtual machine so its hard to separate emoncms.org performance issues from load caused by other virtual machines running on the same host.

I think its time to try a dedicated server which should give us the confidence to say the load on the system is of our own doing and therefore help with development to lower it. I'm going to try the bytemark dedicated server FS11 package here: here http://www.bytemark.co.uk/hosting/dedicated.

I will try to combine the move with full conversion over to timestore so it may be the end of the week or start of next before its up.

We will also have 4 times the ram, cpu cores and 5x disk space which should help, for about 2x the current price.

Have not looked into this in detail but to me it looks like HTTP server does not get enough processor time in current setup, almost like running out of open files or other alike resources. With a dedicated server its way easier to pinpoint the reasons as  you say. 

Hey Trystan,

I have also notice an issue with emoncms, I lost power at my house to day for around 5 hours, before this everything was fine and all feeds were reporting correctly. Two things happen when emoncms does not receive reading from the base unit. 

1. It assumes and average usage base on last reading and the next reading it receives. Now for consumption this might be true however to solar production in my case it can't be. I have a grid-tied setup and if I lose power I don't produce anything.

2. and my kWhd feeds generate stupid readings, I should generally never use more that 15-20kWhd, yet right now it's reporting that I have already used 96kWhd which is impossible. 

The issue will all of this basically mean any kind of stats recording is useless and inaccurate. The suggestion I have on resolving this, is if emoncms does not receive a input from the base it should mark it as a 0 until it get's that update, the base unit should be resilient enough that if it can't connect to the emoncms is should buffer the data until such a point as it can.

I am using the RPi and oem_gateway to report the emoncms.oeg system. 

Ignore me,

I am being a real git, I assume kWh was the kWh for the last hour, where it's actually accumulative ?

Seems the system is down again ?

Down for me too.

Thanks, Im looking in to it.

I'm currently preparing the new dedicated server so it wont be long until we have that online as a replacement. But I will hopefully get the existing one online in the mean time.

Ok emoncms.org is back online. Im not sure why it went down, there where a lot of division by zero errors and request URI too long errors from about 7:30 this morning GMT. In particular line 808 of process_model.php

https://github.com/emoncms/emoncms/blob/master/Modules/input/process_mod...

But I would have thought that would be a pretty lightweight error and not enough to bring a server down, it may have been the result of a slow down rather than the cause.