Archived Forum

I know this must be possible, because my feeds in emoncms.org were originally TIMESTORE, and now they are PHPTimestore.  Can someone help please?

From what I understand, the PHPTimestore is a PHP port of the SQL TIMESTORE engine and the resulting data would be the same, Maybe you could make a backup copy and try the PHPTimestore to PHPFina on the TIMESTORE data.

See Emoncms Time Series engine history for in-depth info.

Paul

Isn't TIMESTORE a program written in C, by Mike Stirling, (later ported to PHP by Trystan) and PHPTimeseries the SQL compatible storage engine?

www.mike-stirling.com/redmine/projects/timestore

That's where I'm a little hazy, because I thought (rightly or wrongly) TIMESTORE by mike-sterling WAS the timeseries engine that Trystan then ported to PHP as PHPTimeseries.

The blog is titled "Timestore TimeSeries Database"  says "It uses a NoSQL approach"

The "variable interval" documentation states

"Note: The 9 byte datapoint structure used in PHPTimeSeries is the same as the structure used by MYSQLTimeSeries engine, PHPTimeSeries was designed to be able to read the mysql time series data files directly. PHPTimeSeries does not yet implement the use of the deleted datapoint flag byte. If I where writing a new variable interval feed engine I would be tempted to drop the use of the deleted datapoint flag, An erronous datapoint could be corrected rather than deleted."

and Trystan also says 

"I poured through the timestore code (thanks to Mike for making timestore open source) and started to understand how timestore worked, Im not a good c programmer so to try and check if I understood it I started writing a port in php. "

And in this forum post he refers to PHPFina, PHPFiwa and PHPTimestore as the "the new engines", a link to which appears as the first line of this excerpt from the Emoncms Time Series engine history

"18th Feb 2014: Emoncms v8, New feed engines, PHPFiwa, PHPFina
Documentation: Variable interval time series
Documentation: Fixed interval time series
Documentation: Fixed interval with averaging time series

This brings us to the present day with the 3 main emoncms feed engines:

PHPFiwa which is based on timestore with a few changes.
PHPFina which takes the same fixed interval approach but without the averaging feature.
PHPTimeSeries a variable interval engine.

These 3 engines provide between them an implementation that can fit most applications.

PHPFiwa is great for power and temperature data. Averaging is useful in these cases and the data is highly regular without time shifts.
PHPFina is great for accumulating data where averaging provides no benefit.
PHPTimeSeries is great for irregular data such as state changes, but its also good for daily data while regular for most of the year may need to have an hour time shift due to daylight saving time."

So whilst I can find no hard evidence I had assumed PHPTimestore and PHPTimeseries were the same and never really questioned it to be honest. This is my own conclusion so any clarification is welcome.

Paul

Hi Paul,

    That's where I'm a little hazy, because I thought (rightly or wrongly) TIMESTORE by mike-sterling WAS the timeseries engine that Trystan then ported to PHP as PHPTimeseries.

It looks as if Trystan called his PHP port of TIMESTORE, PHPTimestore. It's referenced in the default.settings.php file that's in emoncms. Here's an excerpt from v8.4.0:

        'phpfiwa'=>array(
            'datadir'=>'/var/lib/phpfiwa/'
        ),
        'phpfina'=>array(
            'datadir'=>'/var/lib/phpfina/'
        ),
        'phptimeseries'=>array(
            'datadir'=>'/var/lib/phptimeseries/'
        ),
        'phptimestore'=>array(
            'datadir'=>'/var/lib/phptimestore/'
        )

As can be seen, both PHPTimeseries and PHPTimestore are referenced there.

If memory serves, I recall reading an article that said something to the effect of "Although PHPTimeseries is faster than MySQL, it isn't as fast as Timestore, but, PHPTimeseries data is compatible with, and can be read by, MySQL."

Trystan mentions both Mike's program, and his port, in this post: http://openenergymonitor.org/emon/node/3868 (here's a clip of some of it)

PHPTimestore

Posting 30000 takes 6.5s ~ 0.000217 seconds per post (4615 posts/s)
Posting 25000 takes 5.1 ~ 0.000204 seconds per post(4902 posts/s)

Timestore

Although a single post can take 0.000275 seconds (3636 posts/s), testing again today I could not get it to run continuously at a stable post rate at anything much faster than 310 posts per second 0.00323 seconds per post. The emoncms.org server has also reached a similar post rate and its seeing the same unstable slow down effect.

Looks like some of the docs need updating ;-)

Regards,

Bill

You could well be right, although I was awarwe of the different locations I assumed one just proceded the other and was retained for compatibility, although I'm sure that's still the reason PHPtimestore was still in the settings I am no longer sure (if I ever was)  which engine(s) are which. 

The thread you link to mentions PHPtimeseries twice, the second mention states

"If a particular use case required a longer post interval the phptimeseries engine could be used which is a variable interval engine in that you can post at any interval. A variable interval engine is also useful for when the application only requires an entry sporadically i.e in response to a state change such as counting the number of times the internet goes offline on a basestation." 

while discussing the fixed intervals of the "php port of timestore". still not absolute however, and http://openenergymonitor.org/emon/node/4334 states

Timestore is one of many optional feed engines, however timestore started to give some problems see here: http://openenergymonitor.org/emon/node/3868 and so we are now moving over to two new native php feed engines which have better performance: phpfiwa and phpfina.

which also links timestore to phpfina and phpfiwa, so maybe PHPTimeseries is just a variable interval cousin of timestore ???

Anyway, having totally hijacked the thread, I have also noticed engeeaitch's original question was "how do I get from TIMESTORE to PHPTimestore", which I believe we are agreed PHPTimestore is a PHP port of TIMESTORE whatever it is, regardless of what PHPtimeseries maybe too. 

So engeeaitch, I think it would be worth running the "PHPTimestore to PHPFina" script on a copy of the TIMESTORE data, what do you think Bill?

One thing we can absolutely agree on is the need for clarification :-)

Paul

Ahha! just saw this post by Trystan

"I have also added another feed engine called phptimeseries which is much faster than mysql but slower than timestore and without averaging, it also uses the same datafile format as mysql so its just a matter of copying the mysql datafiles. To use phptimeseries you need privileges to create a folder and assign www-data access to it."

Ahha! just saw this post by Trystan

"I have also added another feed engine called phptimeseries which is much faster than mysql but slower than timestore and without averaging, it also uses the same datafile format as mysql so its just a matter of copying the mysql datafiles.

Looks like the ol' memory cells are still workin' after all! (at least somewhat) I just couldn't remember WHERE I'd read it.

Your suggestion of running the "PHPTimestore to PHPFina" script on a copy of the TIMESTORE data sounds like it should do the job.

Thanks for the suggestions - I'll give this a go when I have a bit of time.