Archived Forum

I was about to post my own topic, but this seems to be the same problem that I have in my local emoncms installation. In my case it is clearly due to the summer time change at 4:00 am. The time from 3:00 to 4:00 is repeated two times (as it was the case, I don't know what could be done about this) and the second time the kWh feeds are dropping rapidly, as you can see in the screenshot attached.

Hi All,

I have experienced exactly the same by this morning and i am wondering if there is a way to remove those negative peaks from the data, as they are completely ruined my consumption summary values. My system is also running on emoncms.org

I searched through the forum and found only one post, about some plans of the improvement on how to handle the DST changes (http://openenergymonitor.org/emon/node/4008).
So according to this, the DST change caused problems in the past also.

I am quite new here to remember those problems, and also those problems could be different ones, but i did not find any older posts about DST related problems,so i would be very grateful if someone could help me on this.

Thank you
Regards
Attila

Just to confirm I have observed this too on my emoncms.org kwhd feeds

The problem is that we are not talking about a instanteneous peak that can be removed. If that was the case there is a script here on the forums called remove_spike.php that can be used to fix this (on local installations at least).

The problem here is that for a whole hour the the kwh were dropping and all values since then are wrong. I am more happy it was an once in a year bug and not something else, but I really don't believe there is a way right now to fix the feeds and restore the normal values.

I really love emoncms and I understand the need to move the data storage engine from mysql to the current system. But the problem is we are severely limited to accessing and manipulating the raw data. We need a better system of editing, or at least a system to export the data and import them again, after corrections.
Software and hardware bugs, will always happen especially in an evolving project like this. It's really a pity for data to be permanently corrupted by them. It also prevents us for experimenting more in fear of corrupting the data.

any idea how you re-edit the data ? 

I'd like to fix it as the graphs are messed up.

if it were in a database it would be rather trivial with a simple add this to that after that date. But with the time data I don't know where it is / how to do it.

As its "only" the calculated kwh & kwhd values that are affected is there anyway to re-generate that data?

If we could re-generate that the issue would be resolved no?

Thanks for heads up on this. I can see the error in the kwhd feeds. I need to look at what would be involved to get the realtime and bargraph editor working again, now that the storage engine development is stable, it would be a good time to look at it again. Il update on how I get on.

For anyone interested in how the storage engines work I've written some documentation on it here: https://github.com/openenergymonitor/documentation/tree/master/BuildingB...

Could this have anything to do with summer time ending? We 'put back' our clocks on Saturday evening..... That's when my kwhd visualisation went negative, though the power readings were correct.

My negative spikes started at Sat, 25 Oct 2014 23:00:00 GMT (unix time of 1414278000) and in one case it was a completely new bar ; i.e. the normal positive values we produced correctly. 

Eamonn

I think Robpeel had decided that! For my two penn'orth, there's only one way to do this: convert to or get the times in UTC before storing in the database, and convert back to local when labelling the graph. Anything else is bound to screw up when DST starts/ends.

Hi Trystan

Any update on getting the kwh feeds fixed? At the moment all my graphs are weird due to the very strange numbers.

Using the editdaily view it looks like there are 2 readings for the day of the time change. 

When does the calculation of the "daily" data run? I would guess at 1am or around then - hence the double numbers. I would imagine in spring it will be ok as there will only be one 1am!

Thanks

A little progress, thankyou for waiting:

editrealtime and editdaily accessible from Vis then 'select visualisation' are now re-enabled on the emoncms.org server. However the support for editing is currently dependent on the feed engine:

- Editing of historical data is supported by PHPFina, PHPTimeSeries, MYSQL & PHPTimeStore

- New data points older than the latest data point can be added in PHPFina but not in PHPTimeSeries.

- PHPFiwa does not yet support editing of historical data.

I've disabled the 'delete data in window' button and 'scale data in window' feature as both where only supported by the timestore engine. It would be good to reintroduce them in the future with more work to the feed engines.

At the moment its not possible to delete the double reading - only to resize it/change the value. Il try and look further into making it possible to delete the reading.

Hi Trystan,

How do you re-generate that data in its entirety?

The edits are available on the locally hosted install but i don't know what the "correct" numbers would be. Also all the future records are incorrect - one of the graphs now has its upwards trajectory "reset" (the kwh data) ie the data goes :

Time - reading.

00:50 - 509

01:00 - 509

01:10 - 506

01:50 - 495

01:00 - 368   <<<<  BST -> GMT 

01:10 - 368 

07:00 - 370 

why the data was going downwards during that time I don't know. its an accumalator ...

@octagon are you using the wh_accumulator or just the plain accumulator input processor? The plain accumulator can result in descending trends if the value that is being accumulated is negative. I dont think what your seeing is related to the BST->GMT change, can you see this happen at any other point in your data? Could the negative input be generated outside of emoncms somehow?

Do you have a power feed that you could use to regenerate the kwh data? What feed engine is the power feed?

it goes : 

input -> log->kwh->kwhd->hist

its recording the power used on one circuit that has no generation on it at all !

this negative thing has never happened before.

Its the data feed from the emontxv3 don't know how it would make negative data?

its a PHPTIMESERIES data feed 

How do i get it to regenerate the data from the log into the kwh / kwhd & hist ? 

Thanks

There isn't an in-built mechanism for regenerating feeds at the moment so we would need to write a script to do this. is your power feed PHPTIMESERIES? I would have thought it would be PHPFiwa or PHPFina, or older timestore? phptimestore?

Here's an example of how to read from a phptimeseries feed directly using php:
https://github.com/emoncms/usefulscripts/blob/master/enginereaders/phpti...

<?php
// Directory of phptimeseries feeds, see: settings.php
$dir = "/var/lib/phptimeseries";
// Feed id to read:
$feedid = 1;
//----------------------------------------------------
$fh = fopen($dir."feed_$feedid.MYD", 'rb');
$filesize = filesize($dir."feed_$feedid.MYD");
$npoints = floor($filesize / 9.0);

for ($i=0; $i<$npoints; $i++)
{
    // Read next datapoint
    $d = fread($fh,9);
    // Itime = unsigned integer (I) assign to 'time'
    // fvalue = float (f) assign to 'value'
    $array = unpack("x/Itime/fvalue",$d);
    $time = $array['time'];
    $value = $array['value'];
    print $time." ".$value."\n";
}

The next step is to run each datapoint through the math's that calculates accumulating kwh and then write the resultant datapoints to a feed.

This is how to do the same in python:

import struct, os

# Directory of phptimeseries feeds, see: settings.php
datadir = "/var/lib/phptimeseries/"
# Feed id to read:
feedid = 1
#----------------------------------------------------
    
fh = open(datadir+"feed_"+str(feedid)+".MYD", 'rb')
bytesize = os.stat(datadir+"feed_"+str(feedid)+".MYD").st_size
npoints = int(bytesize/9.0)
    
for i in range(npoints):

    array = struct.unpack('<cIf',fh.read(9))
   
    time = array[1]
    value = array[2]

    print str(time)+" "+str(value);

Heres a complete script for converting phptimeseries power data to phpfina kwh data:

https://github.com/emoncms/usefulscripts/blob/master/process/phptimeseri...

The phpfina feed will need to have been created in emoncms already with the interval set. The source and target feed id's and feed engine locations as well as installed emoncms location needs to be set in the script. Then the script can be run using sudo with:

sudo php phptimeseries_power_to_phpfina_kwh.php

After completing this, its probably best to reset redis with

$ redis-cli

flushall

That should  do it. I havent been able to test it as I dont have a phptimeseries power feed here yet althought I might create another script so that I can generate one for testing

I've tested it and it works a treat! Might be useful for anyone who has a power feed and kwh per day data but would also like to generate accumulating kwh data to use with the new myelectric dashboard or the new bargraph visualisation.

phpfina power to phpfina kwh data:
https://github.com/emoncms/usefulscripts/blob/master/process/phpfina_pow...

phpfiwa power to phpfina kwh data:
https://github.com/emoncms/usefulscripts/blob/master/process/phpfiwa_pow...

An important thing to note is that its important to double check that the target phpfina feed is empty otherwise it will overwrite the target phpfina feed and so could result in data loss.

hi 

all the calculated kwh & kwhd are all PHPTIMESERIES only the histograms are PHPfwia 

Could you include into the emoncms distro a "recalc" button? 

Some of my inputs have multiple process steps to get to the correct numbers. and so re-doing them (kwh & kwhd & histo) isn't going to be easy.

I'm on version 8.3.5 - dunno how out of date that is...

@octagon emoncms isnt unfortunately set up in a way to make this kind of batch conversion process easy to do at the moment  Having a recalculate button would be amazing I agree but to implement its really a complex bit of development.

I've further improved on the conversion scripts above they now run 4x faster and I've merged the three different scripts into one:

https://github.com/emoncms/usefulscripts/blob/master/process/power_to_kw...

To use this script download the usefulscripts to the machine emoncms is running on and open power_to_kwh.php in an editor.

If your emoncms installation is located in a location other than /var/www/emoncms then change the line: chdir("/var/www/emoncms"); to point to your emoncms location.

Under the settings section set your source power feed feed id and set the engine manually. Create a new PHPFina feed in emoncms with a suitable time interval then set the $target setting to this newly created feed:

$source = 27;
$source_engine = Engine::PHPFINA; // or: Engine::PHPFINA, Engine::PHPTIMESERIES
$target = 50484; // must be a PHPFINA feed

Save the script and exit.

Run with:

$ sudo php power_to_kwh.php

amusingly (?) these lines 

    require "Lib/PHPFina.php";
    require "Lib/PHPFiwa.php";
    require "Lib/PHPTimeSeries.php";

don't work. as they don't exist in my install. 

Should i change the path to Modules/feed/engine/.......

Or just download yours?

Thanks

@octagon, you will need to download the full 'usefulscripts' repository/folder with all those files in there. They will all then be in the right location and it should run ok.

@octagon I noticed you mention your histograms use the PHPFiwa engine above, they should be using MYSQL, if they are using PHPFiwa then they may have got created incorrectly and probably dont have any data in them?

Hi Trystan,

I don't have local emoncms installed, but I am using emoncms.org to store my data. I do have 2 feeds affected with this problem. Both are kwh/d and are PHPTIMESERIES. (Below an example on the invalid values.)

Is there a way to fix these for non-locally installed emoncms as well somehow?

DataType: DAILY
Engine: PHPTIMESERIES
ID:55280

Data
1414191600,31.18
1414278000,-469.44
1414281600,-440.37
1414368000,19.90

DataType: DAILY
Engine: PHPTIMESERIES
ID:55284

Data

1414191600,25.28
1414278000,-440.05
1414281600,-418.20
1414368000,12.94

Many thanks
Kind Regards,
Attila

Hi Trystan

I've got a type2 PV system. I'm recording 2 data points to allow me to track the combination of consumption, generation, flow to the grid and flow from the grid and the combined to/from the grid. This way I can see what i'm exporting/importing/generating/consuming. so the set of log->kwh->kwhd->histogram happens 

All my histograms a PHPFIWA.

​All my data feeds are from a REALTIME::TIMESTORE data log. 

Can you convert from them ?

Hello @octagon, I've added the ability to process Timestore feeds via the PHPTimestore reader. That should allow you to at least recalculate the kwh data which will mean you can use the bargraph as configured at the bottom of this blog post to show daily kwh value: http://openenergymonitor.blogspot.co.uk/2014/08/monitoring-solarpv-heatp...

Il create a processor for kwh/d data next. Im still confused as to why your historgram data is labeled as PHPFiwa, can you share a screen shot of what you see there, and the historgram data?

@Attila Can you edit the values manually via Vis > editdaily to get them in the expected region? Realistically Im still quite some way from having an automated system that could recalculate on emoncms.org.

screenshots of my feed list and the generation histogram. 

I'll re-generate the kwh data later !

Hi Trystan,

Many-many thanks for the hint!  Basically by changing those two values, did the trick, and the graphs and values are now in synch with the real consumption.

Thank you again!

Regards,
Attila

Hi Trystan,

I need the output to be a PHPTIMESERIES as they do the variable interval with no averaging.

or should i be using the fixed interval with no averaging (FINA) ?

What should make me choose one or the other? I had gone for the variable interval as some of my inputs only give 5 min feeds.

Thanks!

Hello @octagon, fixed interval no averaging is the best to use for accumulating kwh data, phptimeseries is needed for daily data as the midnight time changes at summer time hour changes but kwh data is at a much more regular time interval ie 10s so phpfina is more suitable. The added benefit is that phpfina is much faster to search to kwh per day data generated from kwh data is extracted and calculated much faster than if the feed was stored in phptimeseries data.

Hi Trystan, 

The recalculated FINA data seems to have fixed the BST/GMT issue. 

To extract the correct data types for each scenario would you suggest then

REALTIME::TIMESTORE  for the "raw" data

FINA for the kwh data

MYSQL for the histogram data

What for the kwhd and any others?

Any reason for not using mysql for them all?

Thanks!

Great good to hear you've got a kwh feed recalculated, brilliant!

I would recommend REALTIME PHPFIWA or PHPFIWA for "raw"/power data. PHPFINA for kwh data, PHPTIMESERIES for daily data and MYSQL for histogram data. There are good reasons for all these feed engines, I wrote a history of emoncms feed engine development here https://github.com/openenergymonitor/documentation/blob/master/BuildingB...

Im going to extend this feed processor to allow conversion of any engine to any other engine, so you could convert your timestore feeds to phpfiwa.

Hi Trystan

If you could finish those converters i would imagine a few others will be very interested 

obviously with the kwhD and histogram ones. Then i can recalculate all my data. Think it will be best done with emoncms not running. sadly each feed will take about an hour!

FYI telling the system to save every 1000 records shaved a bit of time - only 3 mins but every helps.

Could you update the emonCMS with more text as to which you should choose when adding a feed. When i saw "With averaging" i thought oh no it will remove the peaks of the data, rather than Oh goody it will add a second or third data set comprising of the averaged data *in* addition to the raw data.

Thanks

If i move my feed to use a new FINA data set for the KWH data - can i recalculate into that data set in real time - ie both the conversion and "now" use the same target data set? or would that make things explode?

Is it a question of telling the RPi running my copy of EmonCMS to ignore daylight saving and just stick to Zulu or GMT+0 to avoid this situation in the future?

I only started logging more complicated stuff since the last hour change, but if I can avoid it happening to/impacting 6 months of data in the future by telling it to avoid time changes then that should make things simpler.

z

Having fought this problem a long time before the Internet (in relation to logging the running hours of motors!), I believe there's only one solution, and that is that the database must only keep time as UTC. Incoming times must be converted to UTC, and outgoing times must be labelled as Local^note according to the offset in force at the time, which in turn implies that a record of when DST is in force needs to be referenced. Of course, I recognise that this means that 'days' and larger units do not start at 00:00 database time, which complicates data retrieval. And that's why the change to or from DST never touches or crosses midnight, so that there is at least no confusion over the day.

Zag, I think your proposal will make things consistent. Whether it will make everything simpler is a totally different question.

(Note) By 'labelled', I mean the graph or histogram is plotted using UTC but the time axis labels are local. Therefore in the UK around midnight on 25/26th October, the hours would be shown correctly as ... 23:00 0:00 01:00 01:00 02:00 03:00 ...