Archived Forum

Hello Charly86

I wrote a script for this just a couple of hours ago for use with phpfiwa feeds. It works through the phpfiwa data files searching for values that are larger or smaller than the limits, when it find one it sets it to NAN which means its ignored in the graph.

If you copy the following into a file called say: removespikes.php. Then run with sudo php removespikes.php it should do what you want. I would recommend making a backup before you run it just in case.

<?php

$dir = "/var/lib/phpfiwa/";

$id = 0;

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id."_$n.dat"))
    {
        $npoints = floor(filesize($dir.$id."_$n.dat") / 4.0);
        $fh = fopen($dir.$id."_$n.dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];
           
            if (!is_nan($val))
            {
                if ($val>1000000)
                {
                    echo $val."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<-1000000)
                {
                    echo $val."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }
            }   
        }
       
        fclose($fh);
    }
}

Tristan,

thanks for your quick reply, just tweaked for the values I needed to remove, this is working fine on the first time.

Great job, I think this script could be placed into useful scripts folder, because it is really a excellent one, I will try to change it to have set of parameters to be as kind as "universal" spike remover.

I will consider also to have this script to try to analyze spikes (looking values before and after a point) as soon as I will have some time.

Thanks again for your great job

Charles

Hi,

Very interesting script until the edit-realtime feature is not back.

It would be great to be able to run this script with our own feeds stored on emoncms.org.

One should also be able to spikes in histgraph.

Eric

Ok guys, I changed the script to be able to get values from command line, assuming you named the script remove_spike.php the launch command is now

sudo php remove_spike.php -i FeedId -n MinValue -x MaxValue

for example to remove value above 5.5V and below 1V (battery feed in V for example) on feed 22 the syntax is

sudo php remove_spike.php -i 22 -n 1 -x 5.5

Here is the new script

<?php

$dir = "/var/lib/phpfiwa/";

function usage($scriptname)
{
  print $scriptname ." usage: \n";
  print $scriptname ." -i feedid -n minvalue -x maxvalue\n";
}

// arguments parameters
$opts = getopt("i:n:x:");

// Handle command line arguments
foreach (array_keys($opts) as $opt)
  switch ($opt)
  {
    // feed id
    case 'i': $id = $opts['i'];  break;
    // min value
    case 'n': $min = $opts['n']; break;
    // max value
    case 'x': $max = $opts['x']; break;
  }

// check all parameters are good
if (isset($id) && isset($max) && isset($min))
{
  print "Feed ID   : ".$id  ."\n";
  print "Min value : ".$min ."\n";
  print "Max value : ".$max ."\n";
}
// Error display usage
else
{
  usage($argv[0]);
  exit(1);
}

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id."_$n.dat"))
    {
        $npoints = floor(filesize($dir.$id."_$n.dat") / 4.0);
        $fh = fopen($dir.$id."_$n.dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];

            if (!is_nan($val))
            {
                if ($val>$max)
                {
                    echo $val." > ".$max ."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<$min)
                {
                    echo $val." < ".$min . "\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

            }
        }

        fclose($fh);
    }
}

Hi

Is there a version of this script to correct phptimeseries which have to be used for daily data?

I have ended up with a negative entry due to an emontx reboot.

http://79.78.250.151/emoncms/ianeagland&id=4

Regards

Ian

That's a handy little script to remove data glitches, such as the errors made whilst setting up emoncms inputs/feeds, and works without issue.

Charles, your further development of Trystan's script makes it a very flexible tool, and I agree, it would be a great addition to the useful scripts Git folder.

Paul

Edit - I've wrtten it up and submitted a git pull for consideration

Thanks a lot Paul, just merged it

https://github.com/emoncms/usefulscripts/blob/master/removespike/remove_...

I've attempted to edit the script to use with phpfina but although it appears to work the values come back again, I'm guessing there's some other storage location and/or process going on which reinserts the data, my modified version below

<?php

$dir = "/var/lib/phpfina/";

function usage($scriptname)
{
  print $scriptname ." usage: \n";
  print $scriptname ." -i feedid -n minvalue -x maxvalue\n";
}

// arguments parameters
$opts = getopt("i:n:x:");

// Handle command line arguments
foreach (array_keys($opts) as $opt)
  switch ($opt)
  {
    // feed id
    case 'i': $id = $opts['i'];  break;
    // min value
    case 'n': $min = $opts['n']; break;
    // max value
    case 'x': $max = $opts['x']; break;
  }

// check all parameters are good
if (isset($id) && isset($max) && isset($min))
{
  print "Feed ID   : ".$id  ."\n";
  print "Min value : ".$min ."\n";
  print "Max value : ".$max ."\n";
}
// Error display usage
else
{
  usage($argv[0]);
  exit(1);
}

for ($n=0; $n<4; $n++)
{
    if (file_exists($dir.$id.".dat"))
    {
        $npoints = floor(filesize($dir.$id.".dat") / 4.0);
        $fh = fopen($dir.$id.".dat","c+");

        for ($i=0; $i<$npoints; $i++)
        {
            $tmp = unpack("f",fread($fh,4));
            $val = $tmp[1];

            if (!is_nan($val))
            {
                if ($val>$max)
                {
                    echo $val." > ".$max ."\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

                if ($val<$min)
                {
                    echo $val." < ".$min . "\n";
                    fseek($fh,$i*4);
                    fwrite($fh,pack("f",NAN));
                }

            }
        }

        fclose($fh);
    }
}

Hi all,

How can I deal with a spike on hosted emoncms.org?

I just got a huge spike in my temperature/humidity data, that is squishing the good data due to the chart auto-scaling. Alternatively, if only there was a way to manually set the chart vertical scale, then I can leave the glitches alone. This is the first huge spike after charting about 15 units of EmonTH's for about half a year.

Stuck in same situation.

I have the same issue... spikes on the battery voltage reading are messing up the scale of the graph making it impossible to see small variation (the spike is 600V for 2 AA battery it's just flattening the whole plot)

I tried the EditRealTime but it doesn't work. there are posts that it has been disabled because of a bug.

Any news on this?

I have a similar issue with EditRealTime.  I tried to delete a spike from July 15, 2015 at 2:54:50 PM CDT and it did not work.  When I looked at the graph, the EditRealTime ended up changing a value on March 18, 2016 at 10:21:59 PM CDT.  This is the time I was doing the editing.  (Now I have two spikes to fix)

emonPi image emonSD-20Feb16 and low-write 9.4 | 2016.03.15