Hi,
I am getting my consumption and production data from 2 totally different meters and I am uploading both every 10 seconds to emonCMS. The data upload works fine, however, I just noticed when I was doing simple deductions of the time series of one meter from the other, that the timestamps are not aligned / synchronised. This apparently means that the data of one feed is up to one minute ahead or behind of the other, which means that I get garbage if I do calculations of the feeds.
Or, it could be that the one meter actually sends delayed data to my gateway forwarder. In this case I would like to be able to build in a delay somewhere in the data chain to compensate for that. Where could I best do that?
The 2 timestamps of the two buffers at the time of upload are however nearly identical. e.g. 1402340609.
Does anyone know how the timestamps work, what do they tell me, how are timestamps stored in emonCMS (are they created when data is received or does emonCMS use the timestamps hat I send along with the data)? How could I manage the lag that seems to be ocurring? Furthermore, how do you convert them to actual time in Excel format?
Any answers are much appreciated.
Frank
Re: Alignment / sychronisation of feeds with the timestamps
Hello Frank,
Where are you doing the subtraction? in input processing? or afterwards using exported csv data?
What feed engines are the feeds in question stored in? PHPFina, PHPFiwa, PHPTimestore or PHPTimeSeries?
How are you sending data to emoncms?
Emoncms can either use a timestamp you give it or if it doesnt find a timestamp it will apply a timetamp at the time the request is received.
To try and summarise how the data is stored:
PHPTimeSeries is a variable interval feed engine. Every time you post a datapoint it stores the datapoint in the data file. Each datapoint takes up 9 bytes in the data file, the first byte is a flag (used to flag deleted data), the next 4 bytes store the timestamp as an unsigned integer and the 4 bytes after that store the data value as a float.
PHPTimeSeries data is usually stored in the /var/lib/phptimeseries folder and has .MYD file extension (as PHPTimeSeries was originally designed to access datafiles produced by MYSQL directly)
PHPFina, PHPFiwa and PHPTimestore are all 'fixed interval' feed engines. They start with the observation that most of the kind of data we deal with is recorded regularly at a 5, 10, 30s etc and so you dont need to store the timestamp for each datapoint in your datafile. You can instead record the start time of your feed and the interval in a meta data file and then just record 4 byte float values in your data file. The timestamp for any given datapoint can be worked out by start time + the position in the file of the datapoint multiplied by the interval.
This 'fixed interval' approach provides significant performance benefits as searching for a datapoint in a file is very simple, its also more compact, it can take up less than half the diskspace of the variable interval approach.
The source code for the engines can be found here in the emoncms repo here: https://github.com/emoncms/emoncms/tree/master/Modules/feed/engine
Re: Alignment / sychronisation of feeds with the timestamps
Hi Trystan,
thanks for your response. I am doing the calculation in input processing and the times series in question are stored within PHPFIWA.
I suppose that what happens to the "calculated" timestamps of the fixed interval feed engines is that my uploaders(buffers) sometimes miss/skip an upload, which every now and again does happen in my case for one of my meters. see example data below.
While I understand the benefits of the fixed intervals, should I possibly consider using the variable feed engine for my case in order to allow for alignment of calculations of data with gaps? Or should I better do some pre-processing of the data in my RPi in the gateway forwarder?
Are the fixed interval feed engines just looking for the nearest fixed timestamp in the database and then store it under that timestamp? I suppose the input processing also just works with the latest data, disregarding that fact that it may be time-shifted.
this is example data of the 2 timeseries (timestamp1, value 1 timestamp2 value 2) downloaded in csv:
1401736830 41 1401736830 0
1401736840 40 1401736840 0
1401736850 39 1401736850 0
1401736860 39 1401736860 0
1401736870 38 1401736880 0
1401736880 38 1401736900 0
1401736890 38 1401736910 0
1401736910 38 1401736920 0