Archived Forum

I'm not convinced that the point of equidistant crossing IS the correct zero.

Consider this thought example: When looking for the zero point we want the area under the graph to be the same for each part of the cycle (don't we?). Now, go back to your hand drawn picture of a sine wave and trim the top of it off, so that the positive lobes are clipped to just above your zero line. 

Your algorithm will keep zero in the same place, but there will be drastically more area under the zero line than above it.

Can you see what I mean, so shall I add pictures?

I'm not saying that what've you've done isn't better than what we already have, I'm just questioning whether the point where the samples are equidistant is really the correct place for zero.

P.

I'm just questioning whether the point where the samples are equidistant is really the correct place for zero.

Yes, this is the most crucial question about the LTAD algorithm (crap name, I know... but I don't feel comfortable calling it "my" algo as I'm sure it's not original)!  Unfortunately I simply don't know enough about power electronics to know the answer.  All I know is that the LTAD algorithm appears to do a decent job in empirical testing.  And my guess is that the voltage waveform will always have equal duration negative and positive lobes so the current waveform should too, although the current waveform might be out of phase with the voltage waveform.

When looking for the zero point we want the area under the graph to be the same for each part of the cycle (don't we?)... Can you see what I mean, so shall I add pictures?

I think I see what you mean but I'm a bit confused... aren't you describing a situation similar (but less extreme) to the hair dryer on "low"? i.e. the hair dryer chops off the entire "negative lobe" but we do in fact want the zero-offset to respect this "missing lobe" and not try to compensate for it.

I suppose that what we really need is to sample neutral. On the 'other' side of the transformer it's easy - when Live is above neutral then we're in the +ve part of the waveform, and when it's below neutral we're in the -ve half. But the transformer (and indeed the DC bias) adds a floating element to things.

Maybe going back to a couple of high-value resistors rather than using a transformer is the way forward, despite the risks. No distortion. No added phase lag. No saturation problems. But no galvanic isolation, either...

P.

Maybe going back to a couple of high-value resistors rather than using a transformer is the way forward

My laptop still has PTSD from its last run-in with no galvanic isolation ;)

Jack, I too have long been troubled by the idea of the DC offset being 'wrong' when the current is not a true sinusoid.  For my Mk2 PV Router, the 'zero' DC point was determined just once, using a LPF on the voltage stream.  This value was then subtracted from both sets of raw samples. 

Having realised that the 'zero point' of the CT's output would drift when the waveform is no longer sinusoidal, I decided to reinstate the standard arrangement of twin HPFs, one on each sample stream.  By that means, each stream would have the best chance of finding its own level. 

In the case of your (wife's) hair-drier, or my (very own!) hot-air gun, the current taken by these appliances at half power is indeed extreme.  In each case, the CT's output will shift until the waveform is sitting as symmetrically as it can around the 2.5V reference point.  A simple sketch of the voltage and current profiles shows that the power calculation is quite complex.

For most of the active half of each mains cycle, the instantaneous power values will be low because the current waveform is offset.  Similarly, for the inactive half of each cycle, some power will be seen when none is actually being drawn.  Close to each zero-crossing point, there will be a short period when the voltage and current have opposite polarites, so the contributions from such sample pairs (instP) will reduce rather than increase the overall sum (sumP).

A good way to study this kind of topics is with a spreadsheet.  But an easier way is to use the code that I recently posted for my Mk2a PV Router.  Its TALLYMODE mode is ideal for this kind of investigation :D

Using limits of 0W and 2000W, I did a 10-second run on each power setting.  Each set of data was transferred to a spreadsheet and displayed as a scatterplot, as attached.  The full power graph is straightforward enough, with similar amounts of power in both halves of the mains cycle, but the half-power one looks very odd.  Each half of the cycle contains power, with one half having around 30% more power than the other. 

The real point of this investigation, however, is to ascertain whether the standard power/energy calculations that we make using our non-invasive sensors are significantly different than the actual power that's being consumed by the load, which is what our utility meters will be measuring.  The graphs that I'm seeing suggest that our calculated values will actually provide a surprisingly good match.

Simulations with sinusoidal waveforms have shown that a DC offset has very little effect until the lop-sidedness becomes really extreme.  By this I mean once the offset exceeds the peak voltage!   Whichever direction of offset is added relative to the true mid-point, the calculated current always increases.  But because the true mid-point is at a minimum of the energy curve, small offsets in either direction cause very little increase. 

Similarly, the overall error that is caused by (the CT's representation of) our wacky half-wave appliances being able to drift to their own 'incorrect' DC point, and thereby show power in both halves of the cycle, is actually pretty minimal.  Micropence, some may say!

Thanks loads for the detailed reply!

 

Simulations with sinusoidal waveforms have shown that a DC offset has very little effect until the lop-sidedness becomes really extreme.   

Yes, I expect that's right (and you can see evidence for this statement in the table of appliances I measured: emonTX's standard HPF does a great job for all appliances except the hair dryer).

I should've mentioned my motivation: my ultimate aim is to gather a data set for doing non-intrusive load monitoring (NILM) computer science research.  Having accurate readings for real versus reactive power is quite important for this research.  So I was quite frightened by the really wacky "apparent" power value the emonTX gave for the hair dryer and wanted to find if there was anything to be done in software.  I think I will stick with my algorithm, especially if I can adapt it to use the integer arithmetic wizardry you've pioneered ;)

One way to compensate for the drift - if it were necessary or helpful to do so - would be to note the moment when the voltage waveform crosses its zero point.  This could be done fairly easily by interpolating between the nearest samples on either side.  Then, transfer this temporal information to the current stream and calculate the equivalent value of the raw current stream at that same moment.  Having applied the appropriate phaseCal correction, the two streams are effectively being sampled at the same instants.

Whenever the voltage is crossing its zero point, it seems fair to assume that the current will be zero too.  So, just note the raw value of current at that moment, and subtract it from all subsequent raw current samples.  In practice, some kind of accumulator would be maintained that holds the calculated 'zero point' for the current stream.  At every z/c point of the voltage waveform, the accumulator  would be updated according to the actual value of current at that point.  By this means, the 'correct' zero point (aka mains Neutral) could be ascertained for calculation purposes despite the current being absent for half of the cycle.

Given that a solution of this kind should be technically feasible, it's tempting to come up with some super-efficient algorithm that performs this function.  But if the benefit would really be very little, then maybe there are more worthwhile things to be done.

Here's another spreadsheet which shows the effect of lop-sidedness on calculated power.  I can't now remember why I did this one, but maybe it will be of some use to someone!

One way to compensate for the drift - if it were necessary or helpful to do so - would be to note the moment when the voltage waveform crosses its zero point.

But is that even possible? We don't have an accurate zero reference on the secondary side of the transformer, especially once we've applied the DC bias. We could just assume that zero is the middle of the waveform, but V can be almost as wonky as I.

P.

"Whenever the voltage is crossing its zero point, it seems fair to assume that the current will be zero too"

Not true when there is a reactive component in the load. Connect a pure capacitor or a pure inductor and the current zero aligns exactly with the voltage peak.

"but V can be almost as wonky as I."

But mercifully it usually isn't.

I really like this idea Jack, a bit of genuine original thought - well done!

It will clearly work for a pure sine wave, the half-wave case, or any symmetrical, monotonic waveform that has a slope at the zero crossings.

Obviously it wouldn't work for the extreme case of a square wave so it's just a matter of whether there's any real-world waveforms that would confuse it.

There's also the difficulty of accurately measuring the half periods with limited sampling (I haven't looked at your code yet to see how you've done this).

Regardless to any problems though this is a clever idea and was worth your restless night even as a theoretical solution to the half-wave problem. 

I really like this idea Jack, a bit of genuine original thought - well done!

Really glad you like it ;)

There's also the difficulty of accurately measuring the half periods with limited sampling (I haven't looked at your code yet to see how you've done this).

Absolutely.  At the moment, the code does no interpolation.  The code really is just a messy proof-of-concept at the moment (I wanted to test the basic idea before tweaking it).  There's a heck of a lot of polishing to do.

On the plus side, this "LTAD" algorithm gives me a good excuse to try to push the sample rate up as high as possible (because the algorithm requires accurate zero-crossing detection) using all the great ideas developed by you and Robin and others (integer maths, IRQ-driven non-blocking sampling, circular buffers etc).  Not sure if I'll get time to do this before Christmas though...

What about using high quality ADC's to do this calculations, so the main CPU has some extra processing power?
I mean IC's like:
http://www.analog.com/en/analog-to-digital-converters/energy-measurement/products/index.html
http://www.microchip.com/ParamChartSearch/chart.aspx?branchID=11029&mid=11&lang=en&pageId=79

I like this last one, it's made for accurate 3-phase metering (6 ADC's):
http://www.microchip.com/wwwproducts/Devices.aspx?dDocName=en554558

 What do you guys think about this?

 EDIT: Or this one http://www.ti.com/tool/msp430-energy-library

What about using high quality ADC's to do this calculation...

24-bit ADC resolution certainly sounds tasty.  (ADC resolution is bugging me too).

so the main CPU has some extra processing power?

I'm not sure exactly what you mean?  The ATMEGA's ADC doesn't really sap any processing power (at least, not if you use it in non-blocking mode instead of using the analogRead() Arduino function).

I'm not sure exactly what you mean?  The ATMEGA's ADC doesn't really sap any processing power (at least, not if you use it in non-blocking mode instead of using the analogRead() Arduino function).

Aha, didn't know about that non-blocking mode. The few times I played around with the ADC's was with AnalogRead :-).

Aha, didn't know about that non-blocking mode

I should clarify a bit: I perhaps slightly mis-represented the situation by calling it a "non-blocking mode".  Basically, analogRead does three things: it tells the ADC to start conversion, it then busy-waits in a while loop waiting for the ADC to finish, and it then reads the ADC's output.  If you use the low-level commands to tell the ADC to start in your own code (instead of calling analogRead()) then you can do useful work while waiting for the ADC to finish (but you'll need to keep checking if the conversion has completed).

...or better still use the ADC interrupt so you don't have to keep checking

I like that 3-phase metering chip BTW fluppie007.

or better still use the ADC interrupt so you don't have to keep checking

The ADC fires an IRQ when it finishes?  Brilliant!  This just keeps getting better ;)  So we really can keep the ADC running flat-out (as long as we can process data fast enough)

RE: One way to compensate for the drift - if it were necessary or helpful to do so - would be to note the moment when the voltage waveform crosses its zero point.

PO: But is that even possible? We don't have an accurate zero reference on the secondary side of the transformer, especially once we've applied the DC bias. We could just assume that zero is the middle of the waveform, but V can be almost as wonky as I.

If a HP filter is used on the voltage stream, then the DC mid-point is being repeatedly being pulled this way and that.  If you compare the filtered and raw samples, there's quite a difference around the zero-crossing points.  When the raw samples cross the 512 point, the filtered values are significantly different on the +ve and -ve going directions, maybe +/- 30 steps.  This effect is caused by the 0.996 factor in the algorithm.  Although this looks rather worrying in micro terms, the macro effect of the HPF is essentially just to advance the phase of the waveform slightly.  By having a similar HPF on each sample stream, their effects balance out nicely.

For the scheme I'm suggesting, it would be better to use a LP filter on the voltage waveform only.  By updating this only one per mains cycle, as per my Mk2 design, the contributions from individual voltage samples are mostly pre-cancelled before the LP filter is updated.  By this means, ripple on the output is virtually eliminated. 

We now have a stable DC point, expressed in ADC samples, which corresponds to the mid-point of the voltage waveform.  This point will never change much.  The profile of the voltage waveform is surely an order of magnitude or two more predictable than that of the current.  For our practical purposes, whenever the voltage is at this point, the current will always be close to zero.   Correct use of the phaseCal algorithm helps to achieve this situation, by drawing the Power Factor value towards unity.

Each second, this 'zero' level will be crossed 100 times by the voltage stream.  On each occasion, the samples on either side of this point can be examined.  By interpolation at some appropriate resolution, the zero-crossing point can then be accurately determined.  When using integer maths, it may be sufficient to split each sample interval into sixteenths, then the calculation would take minimal time. 

Having ascertained that the voltage waveform crossed the zc point, say 3/16 of the way between samples N and N+1, we can then determine the equivalent raw sample value for the current at that same instant, i.e. 3/16 of the way between its sample N and N+1.   This calculated value can then be regarded as the zero point for the current stream at that moment.  Each individual result could be used to update a suitably damped longer-term value thereby allowing the true zero point for the raw current waveform to be tracked.

By subtracting the calculated zero point from each current sample, and then using these values for our calculations, power would correctly only appear in the active part of each mains cycle.  Although this would be a nice way to go from a purist's point of view, I doubt whether there would be any real benefit on the micro-pence scale.  It's probably better to keep our general calculations simple and fast, thereby leaving more time for other activities to be included within the processor's busy schedule.

The ADC fires an IRQ when it finishes?

Yes Jack, I use it in my code on this thread if you want an example..

http://openenergymonitor.org/emon/node/1535

Robin, Robert has already explained to you that voltage and current aren't in phase for reactive loads.

Having ascertained that the voltage waveform crossed the zc point, say 3/16 of the way between samples N and N+1, we can then determine the equivalent raw sample value for the current at that same instant, i.e. 3/16 of the way between its sample N and N+1.   This calculated value can then be regarded as the zero point for the current stream at that moment.

This sounds tempting but, as Robert pointed out above, surely this would only work for purely resistive loads (where voltage and current are perfectly in-phase)?  Loads with some reactive component appear to be the norm rather than the exception (even my 75W incandescent lamp has a PF of 0.95 for some odd reason).

I like that 3-phase metering chip BTW fluppie007.

Then you can check out the MCP3903 eval. board manual: http://ww1.microchip.com/downloads/en/DeviceDoc/51994A.pdf
I see that Farnell has the chip and evaluation board.

Robin, Robert has already explained to you that voltage and current aren't in phase for reactive loads.

Yes, Martin, I do appreciate that.  Our dump loads, however, are generally resistive ;)

Over the past few months, much effort has been spent in analysing the OEM's PHASECAL mechanism whereby any phase-shift between the measured voltage and current streams can be accommodated.  Our measurement systems need to be able to record a resistive load as just that: a resistive load, where the power factor is unity.  I have done a lot work on this, much of which can be found in the PHASECAL thread that I started in July.   A tool that I've posted for automatically determining the correct PHASECAL value for any rig is mentioned in the Mk2a code that I've just released.

The point about the CT's output, surely, is that it is inherently floating.  If a half-wave appliance is measured, the CT's output will drift until the DC balance-point is regained.  This skews our measurements of power in a way that does not appear to be retrievable.  RW said as much a few days ago.

I'm simply pointing out that this obstacle is not actually quite as fundamental as it may appear.  Yes, the CT's output will drift, but the temporal alignment between the V and I waveforms for resistive loads can be usefully employed to correct for this.

I hope to produce a proof-of-concept build which will show how this idea behaves in practice.  By this means, all the power from Jack's hair-drier, or my heat-gun, should be seen to be in just one half of the waveform.  That's how the utility meter sees it, and it would surely be advantageous if our rigs could perform their power calculations in a similar manner.

The point about the CT's output, surely, is that it is inherently floating.  If a half-wave appliance is measured, the CT's output will drift until the DC balance-point is regained.  This skews our measurements of power in a way that does not appear to be retrievable. 

That's the entire point of the LTAD* algorithm!  It attempts to retrieve the DC offset no matter if the load is a half-wave appliance, or full-wave resistive or partially reactive load (without resorting to dangerous connections to the mains wiring).  And, so far, real-world testing appears to suggest that the LTAD algorithm does what it sets out to do (take a look at the data presented at the top of this thread if you haven't already).

*(feel free to come up with a better name!)

Let's take a step back.... 

Ideally we need to sample neutral, so that we know where the zero point is. But we can't because we're using a transformer which is floating.

A resistor divider could also be used, but there's an inherent danger because that doesn't provide galvanic isolation, should something nasty appear on the mains, or should a malfunction occur.

HOWEVER.... what about if we just tied the mains neutral to one side of the secondary. Or even earth (which ought to be pretty much the same as neutral). Or even a sodding great stake in the ground, if need be. Wouldn't that then allow is to have our proper zero reference point, without compromising safety?

P. (still think resistors are the best option, for all kinds of reasons!)

Paul, connecting to neutral is still dangerous. The neutral is considered a LIVE wire and must be treated as such. Even though most of the time it is close to earth potential, that does not have to be so and it is then that the danger arises. My Institution has been in existence since 1871 and over that time we've accumulated a fair bit of knowledge about electricity and electrical safety. We don't want to miss your contributions because you've killed yourself.

If you earth one side of the secondary, what good does that do you? Think back to how transformers work: Secondary emf is proportional to the rate of change of flux. You've lost your d.c. coupling. End of story.

calypso-rae wrote: "This skews our measurements of power in a way that does not appear to be retrievable"

That's exactly what this thread is about. Jack has proposed a novel solution to the problem that appears to work. Did you not appreciate that?

Your dump loads may be nearly purely resistive but we are talking about whole-house loads here and they can be very reactive.

This has nothing to do with PHASECAL, that's purely to compensate for the phase error caused by the transformers. We don't want to remove the phase difference caused by reactive loads.

Paul - the use of a resistive divider has been proposed for voltage measurement, I don't think anyone has suggested it for current?. I suppose you could measure the voltage drop in the neutral cable to determine the current via a high value resistor (for safety, not as a divider).

The problems with using a shunt are (a) safety and (b) power loss.  To keep the voltage drop acceptable, you need amplification, which needs to be d.c. coupled (or you're back where you started), and you still have the dynamic range problems. If you have a 100 A shunt giving you (say) 75 mV, you potentially have 7.5 W of power being lost, and exactly the same dynamic range problems as with the c.t.   At maximum current, you need an amplifier with a minimum gain of about 13 times. Assuming you want switched gain to improve the low level performance, you need a maximum gain of about 800 to give you 16-bit ADC performance, that is not easy whilst retaining d.c. stability.

OK guys, here's a sketch which works very nicely in the manner that I've been trying to convey.  There are also two graphs, each showing the performance of my Bosch hot air gun at half-power. 

One run was taken with my new Mk2a code.  That sketch is super-fast, uses integer maths, but suffers from the inherent feature that it can't detect when the DC offset from the current sensor drifts.  Consequently, although current is only flowing during one half of the mains cycle, power is calculated to exist in both lobes.

The second run was taken with my new proof-of-concept code, called temporalAlignment.ino.  This works just as I've been saying recently, the key part of the code being:

 // The DC-offset for the current sensor is updated every half-cycle,
// immediately after every zero-crossing point:
//
// first, locate the moment when the voltage was zero
float T_fraction = lastSampleVminusDC / (float)(lastSampleVminusDC - sampleVminusDC);

 //  then, calculate the raw current value at the moment when the voltage was zero
float I_zero = lastSampleI + (T_fraction * (sampleI - lastSampleI));

 // and finally, update the DC offset value for current
float oldOffset_I = DCoffset_I;
DCoffset_I = oldOffset_I + (0.01 * (I_zero - oldOffset_I));    

In the second graph, virtually all of the power is now seen to be in one half of the cycle, just as it should be.  Bingo!

----

MartinR: That's exactly what this thread is about. Jack has proposed a novel solution to the problem that appears to work. Did you not appreciate that?

Martin, I trust you will appreciate that there is often more that one way to tackle a problem.  Here is an alternative solution that will hopefully meet with Jack's approval, and maybe even yours. 

Robin

It's not that we don't understand what you're proposing Robin, it's a pretty simple concept. I even suggested it before you did in the "smaller current CT?" thread yesterday, but only as an interesting idea since it won't work with reactive loads and therefore isn't a solution.

I don't think I'm the one that's having trouble appreciating that there is more than one way to tackle a problem - that made me chuckle!

It's not that we don't understand what you're proposing Robin, it's a pretty simple concept. I even suggested it before you did in the "smaller current CT?" thread yesterday, but only as an interesting idea since it won't work with reactive loads and therefore isn't a solution.

I'm always pleased to hear that people can understand something that I propose.  In this case, I've gone a bit further than just proposing something.  I've delivered a working solution to the underlying problem that this thread appears to be about.

I'd not spotted the "small current thread" until you brought it to my attention.  There does appear to be to some crossover of ideas between these two threads.  

DC has to be removed from each stream of raw samples by some means or other.  Although twin HPFs do the job OK, my preference has always been to subtract an appropriate DC value.  What my latest build does is to derive the optimum DC offset value for each waveform independently.  By this means, our calculations of power are no longer affected by any DC drift that's introduced by the CT.  I can now feel confident that half-speed use of my hot-air gun will not adversely affect the performance of my rig for diverting surplus PV.

PHASECAL provides a convenient way of realigning the voltage and current waveforms if they are found to be out of step.  Our sampling process, and the effect of our non-invasive sensors, will always introduce some degree of phase shift, so it's helpful to be able to remove this.  When measuring a resistive load, the Power Factor will then be unity.  For a non-invasive energy monitor, is this not the best arrangement to aim for?

[Not aimed at anyone in particular]

Folks, 

C'mon, play nicely! 

Bouncing ideas of each other is great. Getting snarky with each other isn't.

Posting up new code and ideas is great. Trashing other peoples' ideas just because they are different to yours isn't.

Lots of people here have made excellent contributions. And projects like this work best when the main input is from a team, rather than an individual.

And for those of us that don't contribute as much, it's much nicer to see people working together, rather than playing intellectual battleships.

There - I've said it!

P.

I agree, it is always good to have multiple options (with the possible exception being types of shampoo in the supermarket.  Then it's definitely good to have a small set of options rather than the trillion which seem to exist at our local supermarket!  "Ah, I see "shampoo" on the shopping list. Shampoo?!  WHICH shampoo?"  Anyway, I digress....)

What would be really, really nice is to have a common, shared testing dataset of, say, 10 different types of load.  This dataset could be produced by setting an emonTX to save as many raw analogRead() I and V samples as will fit into 2k of ram and then spit these out over the serial port (or, even better, use an SD card shield to save several minutes' worth of data in one go).

Then we could compare performance of different algorithms against that single shared appliance dataset (and we'd also have to agree on a common set of performance metrics)

Having said all this, I doubt I'll get time to do this any time soon.  Already way behind schedule...

What would be really, really nice is to have a common, shared testing dataset of, say, 10 different types of load.

One of these loads would, presumably, be one that is purely resistive.  In that case, you would want your system to see V and I as being perfectly in phase with a Power Factor of unity.  Your measurement system could then provide a reliable means for measuring less your conventional loads.

It is the provision of an accurate, non-invasive, energy measurement system which is of prime interest to me.  Then I can feel confident that my PV Router will monitor and divert power as effectively as it can.  Unless I'm arc-welding or whatever at the same time, my system should always divert any surplus power properly. (Not that there would be much in that case!)

This dataset could be produced by setting an emonTX to save as many raw analogRead() I and V samples as will fit into 2k of ram and then spit these out over the serial port 

Some weeks ago, I was working on a version of my RawSamplesTool which operated in just this way.  Rather than displaying the V & I waveforms in low-res form using "ASCII Art", the tool would store all raw samples for up to three mains cycles and then send them out to the Serial port.  If this is of interest to anyone, just let me know and I'll make sure it works.

Here is the tool that I mentioned earlier.  After waiting for a few seconds, it records all the raw data for 3 mains cycles and then sends it to the serial port. 

The graph of the current taken by my half-cycle hot air gun shows how quickly the CT moves to adjust its 'zero' position.  This effect is not so evident with a full wave load, but it must still be there.  I think its macro effect is just to advance the phase.

PaulOckenden wrote: "Trashing other peoples' ideas just because they are different to yours isn't."

I realise you didn't name names Paul but it's clear you include me in that comment and to be honest I'm a bit offended. I don't believe I, or anyone else here, has done anything childish like that. In this particular case the "idea" was mine too so it makes no sense anyway. Also pointing out problems with an idea is completely different from trashing it for no valid reason.

I'm not sure why Robin and I seem to rub up against one another. From my point of view it seems that if I point out a problem with something Robin has said he becomes very defensive and it becomes a long drawn out process to get the point over and maybe I do become a bit short of patience. I certainly don't intend or take any offense over it and I hope Robin doesn't either.

Anyway, getting back to the issue, and perhaps to get my point over with a picture instead of a 1000 words, here's an example of a load which I think will give the wrong result with the "0 volts - 0 current" idea but will be ok with LTAD algorithm..

It's a desk fan (hence the very low current) but it has a clear phase shift between voltage (yellow) and current (blue) of 2.6ms. As you can see the current at 0 voltage is closer to the peak current than it is to 0.

it's clear you include me in that comment and to be honest I'm a bit offended.

It wasn't aimed at any one person. Or any one side. 

P.

"The graph of the current taken by my half-cycle hot air gun shows how quickly the CT moves to adjust its 'zero' position.  This effect is not so evident with a full wave load, but it must still be there.  I think its macro effect is just to advance the phase."

Indeed. I did not test the frequency response of the YHDC c.t. below 50 Hz, but it will be behaving more or less the same as any high pass filter, with the corresponding amplitude and phase response.

It wasn't aimed at any one person. Or any one side.

Well that doesn't exactly exclude me does it!

Gents,

It's Christmas, the season of goodwill to all men. Can we keep this argument to the technical issues, please?

MartinR:

Your load is clearly one of those nasty little motors that are renowned for having an abysmal power factor - 0.68, about what I'd expect. Fortunately, most of the time the effect will be swamped by other loads, but it will still be there. But I think you're wrong about it giving the wrong answer. If I've understood Robin's code correctly, it will (using your picture as the example), give a positive spike on the negative-going crossing, and a negative spike on the positive-going. But the values are fed into a software filter so, even if there is some ripple remaining, the resulting value will be well smoothed. No doubt Robin will correct me if I'm wrong.

I think you're right Robert that the positive and negative errors will cancel in the filter if they are equal - unless Jack's wife is drying her hair at the same time :)

incidentally, what is a typical power factor for the aggregate, whole house load? (i know this is a bit like asking how long a piece of string is)

You'll need to ask someone who works or has worked for one of the Electricity Companies to get that.

Time to air my original thought which I kept to myself because as it's so simple there must be something wrong with it.

Put a sensitive Hall effect sensor near the mains cable and sense the magnetic (not electric) field. This way you could measure DC current if required and DC restoration becomes unnecessary.

OK there must be something wrong with it :)

Brian

Here's a couple of spreadsheets to show how I think the power in Martin's desk fan would be measured. 

The first shows the situation when V and I are in perfect alignment.  The voltage line can't be seen because it's hidden by the current.  The power is a perfect sinusoid of double the frequency.  As the cycle progresses, there is a small cyclical error between the sum of the energy slices that we measure and the trig value.  But by the end of the cycle, everything has evened itself out.

In the second sheet, the current is now 45 degrees ahead of the voltage.   This amount of phase shift causes the calculated power to be reduced to just 0.765 of what it was before.  The cumulative difference between the measured and trig powers now looks massive, but I'm not sure what we should read into this.

My understanding is that a supply meter only records "real power".  So the meter would record this fan as only consuming 0.765 of what it's really taking.  At this stage, we need to be clear about what results we are trying to obtain. 

From my own point of view, my interest is solely to mimic the supply meter.  Thankfully, this seems to be just what our monitor would do in this case.  If one wanted to measure the Power Factor of the fan, then some further processing would presumably be needed.  I think this can be done using the standard calculations in emonLib. 

If this fan were to be measured using an Arduino. I think the proof-of-concept code I posted yesterday would work just as well as any other method.  At each moment when the voltage passes through its zero level, the current at that same instant is derived.  Because of the large phase shift between V and I, these points where the current is nominally zero would alternately be some way above and then below the centre line.  A small fraction (1%) of each of these individual 'deltas' is used to update a master DCoffset_I value.  In this case, the 'master' DCoffset_I value would move up and down each half cycle very slightly which would in turn affect the magnitude of the power reading very slightly. 

A useful refinement of the algorithm might be to pre-combine each pair of adjacent 'deltas' so that the master value is updated only once per mains cycle.    This is exactly how the LPF in my Mk2 Router worked.  There would then be no ripple, and no side-effects that a conventional HPF would introduce.

Brian D,

Yes, there is something wrong with it - not in principle, but practice. I was looking into this only yesterday. Hall effect split core current sensors are indeed available commercially, but they appear to require +/-12 V supply, and the price is well into 3 figures (GBP).

I did a quick sum on a readily available Hall effect sensor. The sensitivity is 2.5 mV/G, and if I've got my magnetics right, the field 1 cm from the centre of a straight conductor carrying 1 amp is 0.2 G.  I make that 0.5 mV per amp, and we need about 1 V rms for a 3.3 V p-p emonTx input (more for a 5 V Arduino). That means it requires a drift-free d.c coupled amplifier with a gain of 20 to equal the sensitivity of the standard c.t., and a gain of over 1000 to approach the likely performance of the SMT emonTx.

Clearly, a ferromagnetic core is used in the commercial offering to reduce the amount of gain needed, but from the price I suspect they still find it necessary to employ a chopper-stabilised amplifier to manage the drift.

Robin:

"My understanding is that a supply meter only records "real power".  So the meter would record this fan as only consuming 0.765 of what it's really taking.  At this stage, we need to be clear about what results we are trying to obtain. "

Half-right. The meter does indeed record (i.e. charge for) real power, but the current is 1/0.765 higher than it would be with a unity power factor. The excess, quadrature, current oscillates backwards and forwards returning in each half-cycle the power it extracted in the previous one. So it consumes exactly the same amount of power as would a d.c. motor shifting the same amount of air.

Sorry, this is slightly off-topic, but would it be possible to gain extra energy per £ by devising some way of offsetting the power factor? 

When running our tumble drier, it would seem very nice if the current through its resistive element could be 1/0.765 times greater than our supply meter records.  Free quadrature current sounds like it should dry our clothes to perfection!

There's only one problem with free quadrature current - as it's on the imaginary axis you only get imaginary power!  But it does actually cost you, because although you get no nett energy out of it, it still drops voltage in your cables and that's real energy that you pay for. So it's actually better for you to run as close to unity power factor as you can.

Robert

Thank you for your input. I was expecting a purely technical rejection of the idea so let us deal with the commercial point first.

A 60A device such as the Tamura L01Z050S05 is available for £12.95 from Mouser Electronics.

The required operating Voltage is 5V and the full scale rating of 60A is the same as the CT's I am using on my rig so I am not convinced that we have grounds to reject the idea yet.

Maybe someone will come up with a technical reason why this won't work?

Brian

Brian wrote:

Put a sensitive Hall effect sensor near the mains cable and sense the magnetic (not electric) field. This way you could measure DC current if required and DC restoration becomes unnecessary.

That does sound very intriguing!

A 60A device such as the Tamura L01Z050S05 is available for £12.95 from Mouser Electronics.

Unfortunately that's not a split-core device, which rules it out for my project (and, probably, most people's projects).  Of course, it's not actually that technically difficult to turn off your mains supply, disconnect one end of a meter tail, feed the tail through the Tamura device, reconnect the meter tail and turn the power back on but I suspect most people won't be happy doing that, not least because I'm 99% sure that, strictly speaking, it would be illegal to do that work if you're not part-P registered.

Robin wrote:

Hi Jack

I asked for a technical reason why this won't work. Your personal reasons for rejecting it are no doubt valid but I would genuinely like to know if my proposal will overcome the original problem that you have identified. Your original post did not qualify the acceptability requirements for proposed solutions therefore I still consider this a valid proposal worthy of technical appraisal.

I have noticed that on a subsequent post you have mentioned non-intrusive load monitoring - maybe you should add this to the initial post.

If there are technical reasons why this won't work (and I suspect there are) then the practical implementation issues will not need to be considered.

You have identified the fitting of the Tamura device as requiring the involvement of part-P registered personnel. This is a wise precaution but when I checked the reg's I could not find the statement that makes this mandatory - maybe you can point me to it.

If there are no technical reasons to reject the HAll effect current sensor then it will be worthwhile investigating other products to see if split core versions are available. Right now I don't see why this will not work.

Brian

You have identified the fitting of the Tamura device as requiring the involvement of part-P registered personnel. This is a wise precaution but when I checked the reg's I could not find the statement that makes this mandatory - maybe you can point me to it.

I'm afraid I'm no expert in the regs but I was under the impression that mere mortals aren't allowed to fiddle with the consumer box (as confirmed by this forum thread), and my (quite possibly invalid) assumption is that we're not allowed to fiddle with any wiring upstream of the consumer box.  If I remember correctly, the utility meter is the utility company's property, so they might not like us fiddling with it.  But, like I say, I'm really no expert so maybe it is fine to just disconnect and reconnect a meter tail.  It would be great to get clarification on this issue.

I have noticed that on a subsequent post you have mentioned non-intrusive load monitoring - maybe you should add this to the initial post.

Good idea.  I will do that now.  Also, while we're on the topic, just to further clarify my own aims:  For my specific project, all the hardware has to be easy to install by computer science (not EE!) students because I will be running an MSc group project next term on "Visualistion and Analysis of Domestic Electrical Energy Consumption" where  6 MSc students will install meters  in their houses/flats.

But that's just my own project.  I'm sure plenty of other people will be interested in alternative solutions, even if the hardware is a little harder to install.

Sorry, Jack, I forgot to tick the 'List' box.  That spreadsheet file should be there now.

The various algorithms which measure power are fairly similar.  In all cases, energy is accumulated by summing individual contributions of energy (power/time).

Samples of voltage and current are obtained alternately from the Arduino's ADC.  Because V & I are not sampled at identical moments, that effectively introduces a phase shift between them.  Each of the floating sensors also introduces some phase shift, so there is always likely to be some residual phase difference between corresponding V and I samples.  This effect can be usefully removed by the phaseCal algorithm which acts on the voltage stream to bring it in line with the current.  By this means, a resistive load will be correctly seen to have a power factor of unity.

In all cases, DC has to be removed from both sample streams.  DC is introduced by the Arduino's ADC because the output range is always positive (0-1023), rather than +/-.  There are two easy ways of dealing with this.

The first is to insert a standard HPF into each input stream.  This works fine but has associated side-effects such as introducing a small amount of additional phase-shift, and other distortion.  All of the standard OEM sketches use HPFs.

The second approach, as I used in my Mk2 code, is to ascertain the DC content of the voltage waveform by using a LPF, and then subtract this amount of DC offset from the raw sample stream.  Because my setup only uses one Vref supply (2.5V), I reasoned that there could only be one DC offset.  I therefore subtracted the output of the LPF from both sets of raw samples. 

This arrangement seems to work OK but does not cope well with half-wave appliances where the inherent drift of the CT is not tracked.  Energy is then seen to be in both halves of the waveform rather than just one.  Whether or not that makes any difference in terms of £££s is another matter.  Incidentally, the HPF approach is no different with half-wave loads.

My latest approach is to determine the equivalent 'zero' value of the raw current stream whenever the voltage stream crosses its zero level.   The 'correct' DC offset can then be subtracted from all raw current samples.  This approach allows the true zero point of the current waveform to be tracked under all conditions, and it should always be better than subtracting a 'non-tracked' value.  Whether subtracting a DC offset is a better idea per se than applying a HPF is a moot point.

Personally, I favour subtracting the DCoffset.  There's no requirement to apply a HP filter.  The DC has been introduced by our measurement system; we just need to remove it.

So, to clarify my preferred algorithm for removing the DC.  In both cases, there is a master value which is subtracted from each sample in the relevant stream (DCoffset_V and DCoffset_I).  Each of these are updated once per cycle by adding one hundredth of the accumulated deltas that have been recorded during the previous cycle.

In the case of voltage, each individual delta is the difference between the raw sample value and the DCoffset_V value.  Over the course of a full mains cycle, these values cancel each other out almost completely.  Periodically adding in one hundredth of their accumulated sum will cause minimal ripple to the DCoffset_V value.

In the case of current, there are only two individual deltas per mains cycle.  Once the system has settled, these two contributions will be equal and opposite so their difference will be very small.  Periodically adding in one hundredth of this value will again cause minimal ripple to the DCoffset_I value.

OK, so we now have a stream of instP values which are summed each cycle as sumP.  This then gives realEnergy which is what is added to my Power Router's energy bucket.  Real energy is what the supply meter measures.

To obtain power factor values, additional per-loop calculations need to be done.  This is as set out in the calcVI() routine in emonLib().  Although not required for my basic power router needs, I copied these calculations into a calibration tool that I wrote for phaseCal.   I recall discussing this algorithm with RW at the time, and it transpired that some further adjustment of amplitude was needed. 

I therefore believe the Power Factor calculations in my PhasecalChecker tool, as attached, to be correct.  Robert will hopefully be able to confirm this.
 

Robin, thank you for such a detailed reply.

Do you, by any chance, have any empirical real-world performance data for your new DC-offset-tracking algorithm?  I'm very eager to see how it performs on data from a number of different types of load.  In particular how well it can measure both real power and power factor.  That will help me to decide whether I should use LTAD or your zero-offset-tracking algo in my metering setup.

I took a look at your spreadsheet and it is very useful but I couldn't see an implementation of your DC-offset-tracking algo in the spreadsheet?  (If such a thing is even possible in Excel!)  Perhaps  I missed it?

I took a look at your spreadsheet and it is very useful but I couldn't see an implementation of your DC-offset-tracking algo in the spreadsheet?  (If such a thing is even possible in Excel!)  Perhaps  I missed it?

No, that's because it's only in the code :D

Can you remind me what LTAD stands for?

Can you remind me what LTAD stands for?

It's a shockingly bad acronym, isn't it.  It's "lobe timing asynchrony detector".  It's the algorithm I proposed at the top of this thread for replacing the HPF.  It removes the DC offset created by our measuring hardware and attempts to determine the correct DC offset for our floating sensors, no matter if the load is a half-wave load, a low-PF load or a normal resistive load.  The data performance data at the top of this thread suggests it performs reasonably well, although it's certainly not perfect.

(incidentally, from now until the new year I won't be able to check the OEM forum often, if at all I'm afraid.  Combination of young kids and Christmas and all that.  Oh, and MERRY CHRISTMAS to everyone!  OEM really is an amazing project with an amazing community; thanks so much for everyone's help... ok... see you in the new year...)

Brian D

Tamura L01Z050S05

I didn't spot that because I was considering only split-core devices; as others have pointed out above, there are problems with fitting a solid-core device. Your main fuse, neutral link, any Henley Blocks and meter terminal cover should all be sealed by your electricity company, and breaking the seals is likely to lead to a degree of unpleasantness from them. I would only break those seals if there was immediate danger to person or property (and then I'd pull rank and argue if they tried to be unpleasant) but otherwise they are sacrosanct. I wouldn't want to disconnect the line tail from the main switch given that the only protection is a 100 A fuse, and the neutral wouldn't be much better.

But it does look as if this device would be worthy of serious consideration. Unless you follow it up with a variable gain amplifier, you still have the problem of the resolution of the 10-bit ADC however, limiting the measurement range to > 0.5 A or thereabouts for reasonable accuracy.

"Energy is then seen to be in both halves of the waveform rather than just one.  Whether or not that makes any difference in terms of £££s is another matter."

Yes, it will make a difference to the £££s because the rms value is wrong (it will be reading low - remember the 'squared' bit in rms).

"I recall discussing this algorithm with RW at the time, and it transpired that some further adjustment of amplitude was needed. "

The Phasecal algorithm is explained here in Building Blocks. After applying it, there may need to be a small once-only adjustment to the voltage calibration in order to correct the measured voltage - this is apparent in the illustration. Note it doesn't actually correct for phase, but does a time shift instead. The voltage wave is close enough to a pure sine wave (when these two are effectively the same thing) for this not to be important in practice. If there are significant harmonics present, then the error increases (and this can in fact be seen on the flattened crest of the voltage wave).

The algorithm interpolates or extrapolates using the present and last values. For minimum amplitude distortion, the value of PHASECAL should be kept between 0 and 1 (interpolating) and depending on the phase errors of the sensors used, it may be possible to achieve this by swapping the order of reading voltage and current.
[And yes, I do realise that the 'nominal' value when using the shop components works out at 1.7, and yes, that could be corrected back to 0.7 by swapping the order in emonLib, but that raises backwards-compatibility issues].

Robin's PhasecalChecker.ino sketch won't give you the correct value if you are using the calcVI( ) method in emonLib because the order of reading V and I is reversed. I haven't run it on a test rig with known phase shifts to verify the numbers, but it looks OK.

This is probably a good place to point out that phase angle and power factor are not the same. Phase angle applies only to a pure sine wave. Power factor is the relationship between real power and apparent power. p.f. = cos(phi) is true only for a pure sine wave.

Jack, thanks for the "LTAD" reminder.  (Sorry, I should have spotted that!)

LTAD appears to rely on at least one of the current lobes being fairly intact.  In certain circuits, this won't be the case.  I'm thinking of a lightly-loaded DC regulator that uses a half-wave rectified AC source.  This will pull current at only one of the mains peaks.  In this case, the output from the CT will be relatively flat for most of each 20mS period, so the LTAD algorithm will struggle to find a DC level where the two lobes have equal baselines.  It would presumably end up locating the short baseline of each peak which is not actually the level that you're looking for.

My alternative approach is similar to the extent that a LPF will most likely come up with much the same level as your LTAD monitor would do.  But only for conventional waveforms.  Once the current waveform starts doing strange things, that's when our results could differ. 

My approach is always to monitor the voltage waveform to establish its DC level.  All schemes are probably equally good for this: HPF, LPF, LTAD, whatever.  Having found the "centre-line" of the voltage waveform, I can then locate each individual zero-crossing point and map these across onto the current waveform.  This provides a "temporal alignment" between the two waveforms that other approaches do not.

Your requirement to charaterise appliances by their power signatures sounds very much more demanding than simply identifying and diverting surplus power - which is all that I've ever been interested in doing.  For a start, your V and I waveforms will need to be perfectly aligned for a resistive load otherwise your Power Factor values will be out.  As I understand it, the standard PF calculation does not care which waveform is ahead of the other, but you may wish to record this separately. 

Finding an effective and efficient means of processing all this data in a meaningful way, with limited processing power and memory, will certainly be "interesting".

LTAD appears to rely on at least one of the current lobes being fairly intact.  In certain circuits, this won't be the case.  I'm thinking of a lightly-loaded DC regulator that uses a half-wave rectified AC source. 

You're right.  It is curious that LTAD appears to work correctly for the dimmed lamp; I would have thought that this waveform would have tripped it up and I honestly don't know why LTAD is giving the right answer for the dimmed lamp.  But yet, I agree, when measuring a single appliance, a cheap dimmer or a DC regulator, LTAD should get confused.  But I'll be using my algorithm to measure whole-house aggregate loads so the current waveform should be vaguely recognisable as a sine wave, I assume ;)

Finding an effective and efficient means of processing all this data in a meaningful way, with limited processing power and memory, will certainly be "interesting".

Absolutely right.  Which is another reason why I'm thinking of "cheating" and using an Raspberry Pi with a sound card to replace the emonTX.  With the RPi's processing firepower, we could do something like this to find the DC-offset:

For each cycle, calculate a rough estimate of the DC offset  using Robin's algorithm or LTAD or a HPF or just average the samples.  Also try to find the peak amplitude of the measured I wave and measure the period of one voltage cycle.  Use these parameters to synthesise a true sign wave.  Then interactively try to fit this true sine wave to the measured waveform (using the initial guess as a starting point for the search).  One potentially fitting algorithm would be "least mean squares".  For a given candidate sine wave, and for each point in time, calculate the difference between the "true" sine wave and the measured waveform.  Square this and add it to an accumulator.  Then, when we've been through every time point, divide the accumulator by the number of time points.  This gives us a "cost" for the fit between the true sign wave and the measured waveform.  The lower the "cost", the better the fit.  Keep trying different parameters for the sine wave (trying, where possible, to move in the direction of the differential of the cost function: this is "gradient descent") until we find a good fit.

  Of course, the reconstructed sine wave will never fit the measured waveform perfectly.  But it should be possible to find a single set of parameters where the true sine wave fits the measured waveform as well as possible.  The zero-line of our best-fit true sine wave should be an accurate estimate of the true zero point for the measured sine wave (I think).

Jack,

You need to do some research on recent developments regarding the fast Fourier transform. http://web.mit.edu/newsoffice/2012/faster-fourier-transforms-0118.html

It is curious that LTAD appears to work correctly for the dimmed lamp; I would have thought that this waveform would have tripped it up and I honestly don't know why LTAD is giving the right answer for the dimmed lamp.

A standard dimmer provides a short burst of power at the end of each half cycle, so is inherently symmetrical.  Your LTAD algorithm is looking for a 50:50 situation. 

Although the centre-line may not meet your criterion precisely, anything above or below this line would be way off the mark.  So LTAD will sit there, angrily muttering to itself as it tries each direction in turn.   My 'temporal alignment' algorithm will behave in a similar manner when V and I are well out of phase.

Jack, the phasecal calculations in the tool that I posted yesterday are indeed OK.  There is just one change from the equivalent calculations in emonLib.  Deep breath ...

Power Factor is the ratio of real power / apparent power.   Real power is phase-shifted voltage * current.   Apparent power is Vrms * I rms.  For PF to be calculated correctly, the same magnitude of voltage needs to be used in both halves of the equation. 

The phase-shifted voltage value is obtained by linear interpolation between the most recent pair of voltage samples.  This process has a small effect on the magnitude which needs to be taken into account.  (For PHASECAL values between 0 and 1, the magnitude gets smaller; outside this range, it gets larger.

As written, the emonLib code uses the non-shifted voltage (squared) for calculating Vrms.  To get the right value for PF, you need to use the phase-shifted value (squared) instead.

From calcVI() in emonLib.cpp:

    //-----------------------------------------------------------------------------
    // C) Root-mean-square method voltage
    //----------------------------------------------------------------------------- 
    sqV= filteredV * filteredV;                 //1) square voltage values
    sumV += sqV;                                //2) sum
   
    //-----------------------------------------------------------------------------
    // D) Root-mean-square method current
    //-----------------------------------------------------------------------------  
    sqI = filteredI * filteredI;                //1) square current values
    sumI += sqI;                                //2) sum
   
    //-----------------------------------------------------------------------------
    // E) Phase calibration
    //-----------------------------------------------------------------------------
    phaseShiftedV = lastFilteredV + PHASECAL * (filteredV - lastFilteredV);
   
    //-----------------------------------------------------------------------------
    // F) Instantaneous power calc
    //-----------------------------------------------------------------------------  
    instP = phaseShiftedV * filteredI;          //Instantaneous Power
    sumP +=instP;                               //Sum 

corrected version in my phaseCal tool:

  float  sampleIminusDC = sampleI - DCoffset;
  float  phaseShiftedVminusDC =
                lastSampleVminusDC + PHASECAL * (sampleVminusDC - lastSampleVminusDC);  
  float instP = phaseShiftedVminusDC * sampleIminusDC; //  power contribution for this pair of V&I samples
  sumP +=instP;     // cumulative power values for this mains cycle

  // Additional items to support power factor calculation
  sumP_forPF += instP;
  float sqV = phaseShiftedVminusDC * phaseShiftedVminusDC;
  sumV_forPF += sqV;
  float sqI = sampleIminusDC * sampleIminusDC;
  sumI_forPF += sqI;

Phew!
 

Update:  Just to clarify what's meant here by "phase shifted voltage":  The PHASECAL algorithm is routinely applied to the voltage stream so as to align this waveform with that of the current when a resistive load is being measured.  This mechanism allows an individual rig to be configured correctly so that the measured Power Factor of a resistive load is one. 

If a reactive or inductive load were then to be measured, the current waveform would no longer be in alignment with the voltage and the calculated value of Power Factor would become less than unity. 

Sorry Jack, this may not be of relevance to anyone who needs to measure Power Factor and/or Irms.

If the only thing that's needed is Real Power, it turns out that there is no need to pre-process the raw current samples at all.  Any DC offset is automatically removed by the standard calculation for RP, I'd just not realised it before.

Even better than having a super-quick integer maths HPF is to have no filter at all!