Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Fri, 14/12/2012 - 13:34

In the function EnergyMonitor::calcVI, the following lines of code are used to remove the DC-offset from the raw current and voltage readings:

//-----------------------------------------------------------------------------
// B) Apply digital high pass filters to remove 2.5V DC offset (centred on 0V).
//-----------------------------------------------------------------------------
filteredV = 0.996*(lastFilteredV+sampleV-lastSampleV);
filteredI = 0.996*(lastFilteredI+sampleI-lastSampleI);

I understand why we must remove the DC offset.

What I don't understand is why we bother using a high-pass filter to do this? Why not remove the DC offset by just subtracting a suitable value from the sampled value? It seems to me that this would be computationally cheaper, and possibly more accurate. For example, when the system starts from cold, measure the raw voltage waveform for 1 second and calculate the average value for these samples. Then, to remove the DC offset, just subtract this average value from the measured value.

I have tried doing this (and tried comparing it to the value calculated by the high-pass filter). The answers are close but not identical. Here's a small snippet of that data... the first column is calculated by subtracting 506 from each sample, the second column is the high-pass filter output (I let the filter settle for about 1 minute), the third column is the difference...

-446.97 -448.27 1.30
 -62.97  -64.01 1.04
 376.03  373.49 2.54
 373.03  369.01 4.02
-125.97 -129.47 3.50
-440.97 -442.69 1.72
-136.97 -138.14 1.17
 347.03  344.48 2.55

Does the DC-offset drift over time?

Re: Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Fri, 14/12/2012 - 16:18.

I've been collecting some data to try to find if DC offset drifts over time. I'm repeatedly sampling for 1 second and finding the average. I've only been recording data for 10 minutes but the average stays pretty stubbornly at 506.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by calypso_rae on Fri, 14/12/2012 - 15:34.

Hi Jack,

My Mk2 PV Router code works in exactly the way that you describe. Because there is one voltage reference, it made sense to determine the DC offset just once and then subtract it from each of the raw sample streams. I found that by using a standard low-pass filter (as per Wikipedia), and only updating it once per mains cycle, the performance was excellent.

Although this arrangement does work, and many 'free' tankfuls of hot water have been produced this summer, I have since decided that this is not the best approach. Because the standard V and I sensors are both floating, their secondary coils are continuously adjusting their DC-position according to the waveforms that they are seeing. This effect may not be very great, or have much overall effect in practice, but I now feel fairly sure that an independent HPF per input stream is actually the best way to go.

The processing time for these HPFs can be greatly reduced by using integer maths, as I posted yesterday.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Fri, 14/12/2012 - 15:44.

Hiya,

Thanks loads for your reply! I've been reading your integer maths threads with interest! I might well implement that code in my project right now ;)

Thank you for confirming that these sensors do drift; as that pretty much kills the idea of taking a single offset at the beginning.

Perhaps the best approach is to "prime" the HPF using an offset calculated by taking the average and then use the HPF on its own from there on out.

Thanks again,

Jack

Re: Why use a high-pass filter to remove DC-offset?

Submitted by calypso_rae on Fri, 14/12/2012 - 16:09.

Perhaps the best approach is to "prime" the HPF using an offset calculated by taking the average and then use the HPF on its own from there on out.

The HPF doesn't really need to be primed. After a couple of seconds, everything settles down just fine. That's what the 0.996 factor is doing, filtering out the DC at 0.4% per loop, or thereabouts.

When I was using a modified LPF to determine the DC offset, there was a problem in that it could lock up. As published, a standard LPF is fine. The workaround that I found was to link it to a HPF which could be guaranteed not to lock up. A simpler fix would have been to constrain the output of my LPF from straying more than 100 ADC levels from the mid-point. This method works just as well.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Fri, 14/12/2012 - 16:24.

The HPF doesn't really need to be primed. After a couple of seconds, everything settles down just fine

Yes, sorry, I meant that the HPF could be primed to allow even those first first seconds to produce usable readings.

(I'm planning to use the EmonTX hardware to gather a dataset for doing some computer science research into smart meter disaggregation. This dataset will probably form part of an academic publication; and the dataset will be released. So I need the data to be as clean as possible. Having momentary glitches while the HPF warms up isn't OK ;)

Re: Why use a high-pass filter to remove DC-offset?

Submitted by calypso_rae on Fri, 14/12/2012 - 17:49.

Without knowing precisely when your voltage sensor will start up within the mains cycle, I don't see how you could know anything about the magnitude or direction of the initial DC offset. Both input sensors are physical devices which will need some time to settle. They have their own agenda, and timescales!

Rather than tweaking your data to make it look OK more quickly, why not just suppress it for a few seconds until the filters have settled? My code uses a boolean flag, beyondStartUpPhase, for this very purpose.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Fri, 14/12/2012 - 18:04.

Without knowing precisely when your voltage sensor will start up within the mains cycle, I don't see how you could know anything about the magnitude or direction of the initial DC offset.

By sampling rapidly for a second and then taking the mean value of those samples.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by calypso_rae on Fri, 14/12/2012 - 20:22.

If you did manage to quantify the DC offset shortly after start-up, to what use could this knowledge be put? The system will be rapidly moving towards its point of minimum energy, which only it - and Michael Faraday - knows.

Re: Why use a high-pass filter to remove DC-offset?

Submitted by jack_kelly on Mon, 17/12/2012 - 15:34.

If you did manage to quantify the DC offset shortly after start-up, to what use could this knowledge be put?

I'm using it to "prime" the HPF so the HPF almost immediately produces the "right" answer instead of taking several seconds (or, in some cases, much longer) to settle.

(As a related point - and I know this won't be relevant for you calypso_rae because you're using your own code - but it might be worth mentioning that the HPF in the EmonLib starts from cold with uninitialised variables (hence those variables will be filled with random values which will cause the HPF to sometimes take a long time to settle). I submitted a github issue and a patch).

Archived Forum