Emoncms.org service outage 25th Feb

There an ongoing issue on the emoncms.org server which Im trying to resolve. I carried out a reboot after security upgrades and it did not come back online issue free. Reloading everything to redis is taking time and so the throughput is slow but its rising. I will update as this progresses.

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

Data being receieved on the server is currently queued, you may notice  feeds saying updated 30 mins ago but the data is coming through slowly. At the moment the queue is growing faster than the processors are able to process.

Needing to reload feed last value data to redis may be causing the slower throughput but its not clear.

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

The queues are now going down and should all be updating as normal in a couple of hours.

It does however, look like some kWh/d and kWh feeds have reset to zero. I'd be interested to hear if anyone can see any other issues.

Bbaass_TMH's picture

Re: Emoncms.org service outage 25th Feb

This explains everything, and confirms that I am in fact NOT going crazy.

We installed our system at a client's distribution board today, between 4 and 2 hours ago.

I was getting very frustated with trying to figure out LAN and IP settings on the EmonBase, trying to get it to send data to emoncms.org.

Umski's picture

Re: Emoncms.org service outage 25th Feb

My data started updating again at just after 3pm and seems to have been okay until about 40 minutes ago. Am missing anything between 11.30ish 3.00pm ish so not sure if it will catch up or not. I've not lost any feeds by the looks of it so not too big a deal - to be fair it's been working absolutely fine for 6+ months :)

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

My apologies Bbaass I can understand how frustrating that must have been!
Umski, you should now see the feeds that stopped 40mins ago catching up.

nrgbod's picture

Re: Emoncms.org service outage 25th Feb

My kWh data has all reset to zero. I've been keeping running totals for the last couple of years of things like kWh to electric car charging from my solar panels also how many kWh of own produced electric has been used by the house and how many kWh exported. Shame I'll have to start again from scratch, is there no way of retrieving any of this data. Just yesterdays or this mornings totals would be enough to restart the running totals. 

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

@nrgbod yes its retrievable, I will update with a solution soon

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

A little good news is that emoncms.org now supports HTTPS, preparing for which however caused the outage today. I will update in more detail on the https support in another post soon.

It looks like the majority of the feeds are now updating as normal, the queues are all processing well. Sorry again for the outage today.

nrgbod's picture

Re: Emoncms.org service outage 25th Feb

Thanks Trystan.

I looked at my kWh feeds this morning and realised that the historical data is still present but just reduced to zero yesterday. This is not as bad as I thought as I can use the edit realtime Visualisation to re-align the feeds with what they where yesterday morning. The only problem with doing this is that by adding say 3800 kWh to a kWh Accumulator feed results in a 3800 kWhd being recorded for the day which plays havoc when trying to view daily graphs due to the auto-ranging . Is there any way you could modify the kWhd accumulator processor on emoncms.org to ignore daily figures over say 100 kWhd or have the limit user selectable, or would this break the functionality of the edit realtime visualisation?

pbertra's picture

Re: Emoncms.org service outage 25th Feb

It seems like the servers are a bit slow again: feeds are updating but with a increasing delay (2 minutes one hour ago, 4 minutes now).

Do you also notice this on your feeds?

Bbaass_TMH's picture

Re: Emoncms.org service outage 25th Feb

Not experiencing any problems here.

nrgbod's picture

Re: Emoncms.org service outage 25th Feb

Yes, mine started getting behind around mid-day (Gmt). It's now about 6 minutes behind.

pbertra's picture

Re: Emoncms.org service outage 25th Feb

See the feeds attached. They should update every 20 seconds

TrystanLea's picture

Re: Emoncms.org service outage 25th Feb

Ok, I can see the issue and I'm trying to correct it. There are 4 input queue processors, each allocated to a block of a certain number of users. Queue2 has been processing slower than it was, getting filled up over the last couple of hours.  I have attempted to balance them better, and can see queue 2 going down now. It may take a few minutes for the feeds to catch up

andrea.rimo's picture

Re: Emoncms.org service outage 25th Feb

Hi,

I just noticed that 1 of my feeds (71314:ENEL_kwh) has reset to zero on the 2nd of February. The others have not (solar and house consumption).

Is it something I can fix or do I need help from you?

thanks

A

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.