Archived Forum

Some of the datafiles look a bit suspect too.

Using the documented descriptions of the meta files I've viewed the stored data and I'd appreciate some comments on what would be expected to be stored for the data where the input was fed with timestamped data at one minute intervals as above (Sorry but stamps here are GMT so add one hour for BST)

1 minute FINA - looks OK
31/03/2016 23:00:00 0
31/03/2016 23:01:00 1
31/03/2016 23:02:00 2

1 minute FIWA_0 - looks OK
31/03/2016 23:00:00 0
31/03/2016 23:01:00 1
31/03/2016 23:02:00 2

1 minute FIWA_1
31/03/2016 23:00:00 4.5 would have expected 0 as there is no previous data
31/03/2016 23:10:00 14.5 would have expected average of 1,2,3,4,5,6,7,8,9,10=5.5 or value at 23:10=10
31/03/2016 23:20:00 24.5                 " = 15.5 or value at 23:20=20

1 minute FIWA_2
31/03/2016 23:00:00 29.5 would have expected 0 as there is no previous data
01/04/2016 00:00:00 129.5 should be average approximating to 50 or value at 00:00=100
01/04/2016 01:00:00 229.5 should be average approximating to 150 or value at 01:00=200

5 minute FINA - looks OK
31/03/2016 23:00:00 0
31/03/2016 23:01:00 1
31/03/2016 23:02:00 2

5 minute FIWA_0 - looks OK
31/03/2016 23:00:00 0
31/03/2016 23:05:00 5
31/03/2016 23:10:00 10
31/03/2016 23:15:00 15

5 minute FIWA_1
31/03/2016 23:00:00 2.5 would have expected 0 as there is no previous data
31/03/2016 23:10:00 12.5 would have expected 5 or value at 23:10=10
31/03/2016 23:20:00 22.5 would have expected 15 or value at 23:10=20

5 minute FIWA_2
31/03/2016 23:00:00 27.5 would have expected 0 as there is no previous data
01/04/2016 00:00:00 127.5 would have expected 50 or value at 00:00=100
01/04/2016 01:00:00 227.5 would have expected 150 or value at 00:00=200
 

60 minute FINA
31/03/2016 23:00:00 would have expected 0 as there is no previous data
01/04/2016 00:00:00 159 would have expected 50-55 or value at 00:00=100
01/04/2016 01:00:00 259 would have expected 150-155 or value at 01:00=200

60 minute FIWA_0 - LOOKS OK!
31/03/2016 23:00:00 0
01/04/2016 00:00:00 100
01/04/2016 01:00:00 200

Thanks

Mike
 

Hello Mike

The way the emoncms data request works is that it returns the nearest data to the request times set by the request start time and interval.

Let's say we set the visualisation start time to 1460217206,000 (04/09/2016 @ 3:53pm + 26seconds (UTC)) and the interval to every 10 seconds, then emoncms will return the closest data it can, but allocated according to the request parameters. The data returned will be timestamped:

1460217206
1460217216
1460217226
1460217236
...

If the feed was started at say 3:53pm and 0 seconds with data arriving at the server every 10 seconds, then for PHPFina the actual data locations are

1460217180
1460217190
1460217200
1460217210
1460217220
1460217230

The nearest datapoint to request timestamp 1460217216 is 1460217220, which will appear to shift the data 4 seconds backwards on the multigraph.

The solution is to align the request interval and start time with the underlying feeds, so that if the request interval is 10s, the start_time is rounded to the nearest 10s.

The request would then have a start time of 1460217210 and return the datapoint allocated to that 10 second block in the phpfina file.

I had already implemented this for the default graph view on emoncms, but not for multigraph. I've now added this code to the multigraph visualisation both on emoncms.org and the master branch. A cache refresh might be needed to load the new javascript.

There is still an issue if the request interval is less than the feed interval, which perhaps could be improved upon. Let's say the request interval is 10s and limit interval is set to 1, so that the data returned in the case of a 60s feed will be returned at an interval of 60s rather than the requested 10s. With limit interval set to 0, it will return datapoints at 10s, each one corresponding to the nearest 60s period.

The start time is still rounded to 10s, even with limit interval set to 1 on a 60s feed. The data returned could therefore have timestamps that are out compared to the data file position by 30s.

In addition, a node posting to the 60s feed has its timeslot allocated based on math function 'floor' after division by the data interval. So it could be out by 59 seconds:

datapoint_position_in_file = floor((post_time - feed_start_time) / feed_interval)

where feed_start_time has been rounded using floor according to the feed_interval in the same way.

It's possible to use the graph visualisation (there's a post about it here https://openenergymonitor.org/emon/node/12280)  to fix the request interval and set your own start_time and end_time in order to return the data in a more controlled way, if the timing is more important.

Hope that helps explain a bit of how it works.

Hi Trystan

Thanks for the very quick response.

I'm trying to get my head around the translation from the feed data to the data viewed on a graph/multigraph.

I've had a poke around the engine php and it seems the feed data is always aligned to the feed storage interval, so a 10 second feed will always be at 00, 10, 20 seconds etc and a 60 second feed always on the 00 seconds.
I believe fina and fiwa_0 data is stored this way. (I'll come to fiwa_1 etc later).
This is great!

I can see the need to provide a subset of the base feed data where the request to graph is for a long time period.

I can't see the need to change the time stamps of the feed data in a small dataset (graph a short time period).
I tried the new graph module on emoncms.org for a 5 minute feed and it indeed is much better in that the time stamp does not seem to move as you scroll left-right. But the time stamps were moved to be mid way between the base data stamps. In this 5 minute data the points graphed were at 09:02:25, 09:07:25 etc rather than 09:00:00, 09:05:00 etc. I can't see the point of this.

Coming back to the FIWA base data, it seems that each layer contains averages of the layer "beneath" it.
For 1 minute data the _0 layer has data every 1 minutes (aligned to the 00 seconds) and the _1 layer has 10 minute data aligned to the hour. This too is great.
BUT the averages seem to be shifted to the front of the averaging period.

Say, for a data set from 00:00:00, the layer_1 data for 00:00:00 is the average of the layer_0 data from 00:00:00 to 00:10:00. This is strange as one would expect the average to be stored at the end of the period (ie the data from 00:00:00 to 00:10:00 should be stored at 00:10:00. It sort of says when retrieving the 00:00:00 layer_1 data "This value at 00:00:00 WILL BE the average for the NEXT 10 minutes" rather than "this was the average over the previous 10 minutes"

Is this deliberate?

FYI my need for accurate stamps is to be able to see what caused what when our community hydro turbine trips or throws a wobbler. We have a mix of 5 minute and 1 minute feed data with inputs every minute.

Thanks for a fantastic product

Mike