Archived Forum

[Edited]

There's something queer going on there at your end!

With the sketch unmodified, a standard emonTx and Ideal adapter, I get 296 - 297 reported for 243 - 244 V measured. Change the voltage calibration to 227.5 (standard emonTx) and I'm within 2 V.

Hi Trystan,

I am atm using the sketch attached. All the math is included (somewhere :-)).

Still using floats for the 'final' values after averaging is done, but at that point this does not lead to performance problems.

I have done this some months ago and not touched since then. And it was not written in a way to be published. But please ask if you are unsure about anything.

BR, Jörg.

PS: this code is working since some weeks in my home. Three phase CT current sensing combined with one 400V AC trafo for voltage sensing. Data sent to a RasPi, where Emoncms shows the data. Impressive results, I have reached ~+-1% accuracy compared to my ferraris counter (but I am using a 12 bit adc MCP3208). Hmmm, yes, the code is for the MCP3208, but you can easily replace the MCP_Read() by reading the internal ADC.

Hello Robert, a yes the voltage calibration is for an emontx here with a 120k 10k voltage divider combination.

Thanks Jorg for sharing your latest version will take a look.

Just copied and pasted your latest filter Jorg and it works great:

TempL = (long)(sampleV-lastSampleV)<<8;
TempL += filteredV;
filteredV = TempL - (TempL>>8);
TempV = (filteredV+128)>>8;
sumV += (TempV*TempV);

Thanks!

Before I forget to mention it:

At least part of the integer code optimization was done and/or inspired by Robin Emley! 

High pass filter in integer maths? Is your question how or why?

It depends on recognising that 0.996 isn't a magic number, all that is required is a number reasonably close to unity in order to provide an adequately long time constant so that there is little phase and amplitude distortion at the 50 Hz fundamental frequency being measured.

It so happens that 255/256 is a number that fits the bill, and this is easy to do with very efficient low level operations - bit shifts and subtractions - because  n × 256 = n << 8  and n × 255 = n × 256 - n, and then if you take n × 256 - n and right-shift it by 8 bits, you divide by 256.

I suspect if you wanted a longer time constant, you could go for 0.998 ( = 511/512 - a 9-bit shift ). The actual time constant, equivalent to RC, is  (sample period) × α / (1 - α)   [α of course being the coefficient], so it would go from 250 × sample period to 499 × sample period, with a correspondingly longer settling time to reach a stable value.

Thanks Robert that makes a lot of sense, yes it was 'how'.

I think I may have been getting caught out with the type conversion

long testA = (long)300*256;
long testB = ((long)300)<<8; 
 
Serial.print(testA);
Serial.print(" ");
Serial.println(testB);

The importance of putting (long) before 300*256 to get the expected result 76800 instead of 11264. I have often added .0 so 300.0/256.0 which takes it into floating point math but as we're trying to avoid floating point math here was getting caught out.

Ok this is making much more sense now

  int n = 300;
 
  double testA = 0.996 * n;
  long testB = ((((long)n)<<8)-n)>>8; 
 
  Serial.print(testA);
  Serial.print(" ");
  Serial.println(testB);

gives: 

298.80 298

It's great that you are looking at this Trystan. an "offical" sketch that can support continuous monitoring will be very beneficial.

You might also want to look at my PLL library here as that is what I am using now in my system. It doesn't use filters to determine the DC offsets but instead takes advantage of the PLL knowing where the zero crossings are and implements a second loop to adjust the zero crossings to 0 volts.

In my 3-phase system I have also implemented the WHr metering with backup to EEPROM when the power fails and I also have self-calibration with the calibration parameters stored in EEPROM. I created a single-phase version a couple of months ago but never got around to fully testing it.

With a small change:

  long testB = ((((long)n)<<8)-n+128)>>8;

it would match the float result even better :-))

But I understand that you just wanted to get an idea if the results are comparable.

And of course you recognise 128 as being  ½ << 8  (The old trick to round to the nearest integer.)

Trystan,

looking into your code I see one possible problem (which you also mention):

Runtime of your calc() function has to be (quite a bit) shorter than the time between two ADC interrupts. This could be hard to achieve even with integer calculations. On the other hand you do not 'use' the time between most of the ADC conversions.

One possible solution would be to set a flag 'conversion_ready' in the IRQ routine after sampling all channels and do the calculations in the main loop. But then you have to guarantee that this calculation gets done before the next setting of the flag.

BR, Jörg.

Thanks Martin, Jorg, Robert

So for voltage 

n = lastFilteredV + sampleV - lastSampleV;
filteredV = ((n<<8)-n+128)>>8;

appears to work well, the 128 made all the difference bringing the voltage down form ~270V to ~240V

When I apply the above to the current channels the values dont come out right at all, am I right in thinking this is because a small current signal is getting rounded at to few decimal places as is where?

So Im trying to relate:

n = lastFilteredV + sampleV - lastSampleV;
filteredV = ((n<<8)-n+128)>>8;

to your code Jorg:

TempL = (long)(sampleV-lastSampleV)<<8;
TempL += filteredV;
filteredV = TempL - (TempL>>8);
TempV = (filteredV+128)>>8;
sumV += (TempV*TempV);

I wonder if you could explain why filteredV is different to TempV, or why bring out filteredV at the earlier point, Im guessing its to do with getting better accuracy but I'm struggling to decode it with tests here. 

Thanks a lot

I agree with Jörg, it's not a good idea to try to do all your calculations during one interrupt. It would be better to update each filter after it's associated ADC has completed. There will be time to do this with integer maths.

Also, you can't use the same phaseShiftedV value for all the CTs as the sample times will be different.

In EmonLib.ino TrystanLea wrote:

"- Dev Question: MartinR uses a phase locked loop approach to always sample over an integer number of wavelengths and sample in the same place each time. But would a method that deviates slightly each time be better given issues around ADC step size relative to signal, sampling at slightly different places each time could average out errors caused by sampled value being one step up or down from actual value.."

This has been suggested before but I'm not sure I understand the reasoning behind it. The sampling is a fixed place yes, but the voltage and current are varying continuously so there's still a large element of randomness to the sample value relative to the step values.

This has been suggested before but I'm not sure I understand the reasoning behind it.

Martin, I think you are absolutely right there. Varying the sampling point between fullwaves is effectively a sampling with a frequency different from a multiple of 50Hz (or 60Hz). This will almost certainly lead to a 'beat' frequency in the measuring result. So, not a good idea (I think).

(PS: Martins PLL is a really good solution (as far as I can recall it). In most of my professional projects I use a zero crossing signal to measure the mains frequency (or better period) and then try to divide the next  fullwave into the same number of sampling intervals, effectively varying the interval length. This tends to give perfect results without any beat effects. Another useful solution is to average over a 'sufficient' number of fullwaves, the only question is what sufficient really means.)

Just to say in relation to my earlier question on why filteredV is different to TempV, or why bring out filteredV at the earlier point, I think I've got it so no need to go to the effort to explain. 
Il write up my understanding of it.

Will get on to thinking about PLL next, thanks a lot

All the theory I was ever taught had samples at precisely defined points on the wave, so I'm naturally inclined to think this is right, and intuitively it seems right. Because the PLL maintains that relationship (to at least a very good approximation) over each and every cycle, then by locking to mains and with a guaranteed integral number of samples per cycle you immediately dispose of all 'end effects' and the inaccuracy that those bring with them.  There's one possible snag: I haven't thought it through but there might be implications for an anti-alias filter - or the absence thereof. I've not even contemplated testing it with a deliberately distorted wave with loads of 25th harmonic and above, as there shouldn't be much of that on the voltage wave in any case.

Which begs a question for Martin: presumably it would not require much change for your PLL to lock to 60 Hz?  I tried to check the actual lock range but I couldn't generate enough voltage from my sig.gen for it to lock at all, and I didn't want to tack too many wires onto my emonTx.

I wonder if you could explain why filteredV is different to TempV, or why bring out filteredV at the earlier point, Im guessing its to do with getting better accuracy but I'm struggling to decode it with tests here. 

Ups, sorry Trystan , I oversaw your original question.

Just to say in relation to my earlier question on why filteredV is different to TempV, or why bring out filteredV at the earlier point, I think I've got it so no need to go to the effort to explain.

No real effort :-))

As you already said yourself, filteredV is scaled by a factor of 256 in order to maintain a higher accuracy during calculations. FilteredV has to have this high accuracy (or long 'memory' about hundreds of samples taken before) for the high pass filter to work as expected, whereas the accuracy of TempV has to be just 'good enough' for calculation of the momentary voltage squared sample.

The importance of putting (long) before 300*256 to get the expected result

If you want to avoid hazards like that, enable compiler warnings in your build environment.   gcc will then warn you with:

warning: integer overflow in expression

Which begs a question for Martin: presumably it would not require much change for your PLL to lock to 60 Hz?

Shouldn't be a problem. Currently I limit the lock range to about +/-0.5Hz but by changing the allowed timer count range it would be possible to lock to 60Hz.

More of an issue might be how many samples per cycle you could get at 60Hz with 3 CTs. If you're sampling 4 analogue signals (3 current & 1 voltage) at the standard 104µs then the absolute maximum is 40 samples per cycle. Even at 50Hz you could only manage a maximum 48 samples per cycle.

You would probably have to run the ADC at 250KHz, as I do in all my sketches (Arduino default is 125kHz). This drops ADC cycle time down to 52µs.

I've written up a building block page on digital filters, trying to cover everything I learnt yesterday so that it will help others asking similar questions, any comments, improvements are very welcome: http://openenergymonitor.org/emon/buildingblocks/digital-filters-for-offset-removal

So to clarify the concern with not using PLL is that it could lead to a beat frequency in the measuring result.  

If we dont use PLL but still check for zero crosses and count say 20 zero crosses per full measurement, starting and ending as close as the last sample is to the zero cross the concern is that this may not be close enough to the zero cross to avoid beat effects even if its within a single sample width of the zero-cross.

Another approach to add to the mix is the AVR465 code which if I read correctly samples 401 samples which they worked out to be an integer number of both 50Hz and 60Hz. Here's their comments in defines.c:

// Number of samples that will be accumulated for measurements. Since measure-
// ments are based on a fixed time window, the accumulation cycle should be
// such that it fits an integer number of mains cycles. Otherwise, measurement
// results will fluctuate and create short-term variations. Even so, the
// effects can be completely removed by means of averaging the measurement
// results. No harm done in the long run, but may look strange and gives a
// false impression of poor accuracy.
//
// For compliancy with both 50Hz and 60Hz environments, the cycle length
// should be chosen such, that it can fit an integer number of both 50Hz and
// 60Hz signals. For example, cycle lengths of 200ms multiples are fine.
// Some suitable cycle lengths are as follows:
//
//    XTAL |   ADC | ADCCLK |  Sampling | S.R./Ch |  Cycle |    50Hz |    60Hz
//   [MHz] | Presc |   [Hz] | Rate [Hz] |    [Hz] | Length |  Cycles |  Cycles
//   ------+-------+--------+-----------+---------+--------+---------+--------
//   3.580 |   128 |  27965 | 2151.169  |  717.06 |   N*72 | N*5.021 | N*6.025
//   3.689 |   128 |  28800 | 2215.385  |  738.46 |   N*74 | N*5.010 | N*6.012
//   4.000 |   128 |  31250 | 2403.846  |  801.28 |   N*80 | N*4.999 | N*5.999
//    -"-  |   -"- |   -"-  |    -"-    |    -"-  |     80 |  4.9920 |  5.9904
//    -"-  |   -"- |   -"-  |    -"-    |    -"-  |    401 | 25.0225 | 30.0270
//   4.096 |   128 |  32000 | 2461.538  |  820.51 |   N*82 | N*5.000 | N*6.000
//   4.608 |   128 |  36000 | 2769.231  |  923.08 |   N*92 | N*4.983 | N*5.980

#define NMAX 401
#define NORM    1.0/NMAX

Trystan

If we dont use PLL but still check for zero crosses and count say 20 zero crosses per full measurement, starting and ending as close as the last sample is to the zero cross the concern is that this may not be close enough to the zero cross to avoid beat effects even if its within a single sample width of the zero-cross.

I think that summing over 10 or 20 fullwaves of the mains frequency will effectively cancel out most of the discussed 'beat' problem (to a level where it does not really matter any more). But at the same time the sample frequency should be high enough! Doing this at 10 samples per fullwave (extreme example) will have the similar effect as summing over a smaller number of fullwaves.

Adn, summing over a higher number of samples (before doing the divide and square root of that sum), you have to make sure that the sum does not overflow (this is the main reason why the above mentioned TempV is not used with higher resolution).

PS: I read your write up about digital filter. Very nicely done! I especially like the way you did it, showing the improvements which come up after thinking about it in more and more detail.    

A bit of progress:

I've implemented integer mains cycle sampling using the zero-cross detection method. Not ruling out PLL at this stage, just went with what I was nearer to implementing.

I've then used your approach Martin of having total accumulators that are updated with cycle accumulations at the end of each cycle making it possible to do the final calculations outside of the interrupt function in an asynchronous manner so as not to slow the interrupt function down. It also has a nice benefit of being able to control the measurement window in the main loop.

I've also gone for the approach that seems to be generally adopted of not measuring the supply voltage using internal bandgap, interesting to hear that using the bandgap measurement are less accurate, I havent done the calculations. It certainly simplifies code a bit to do it manually and also less of an issue with the continuous sampling approach as its not for running off a battery.

It did turn out that my approach of doing all the analog input calculations (filters etc) at once after sampling all 4 analog inputs took longer than the interrupt period of 104us. So I split the calc() function up so that analog input specific calculations are done as the ADC value becomes available as seems to be the general approach. The calculations then take between 28us to 56us out of 104us.

I've added grid frequency measurement, its amazing (but I guess one would worry if it didnt) how close it matches the value recorded on the www.dynamicdemand.co.uk website. Quite fun to watch.

The code is in the main emontxfirmware repository here: 

https://github.com/openenergymonitor/emonTxFirmware/blob/master/emonTx_CT123_Voltage_Interrupt/emonTx_CT123_Voltage_Interrupt.ino

Next to add the RF part and test temperature sensing.. 

Good work Trystan.

Just a couple of points...

I don’t think it’s a good idea to base your outer calculation loop purely on millis() since the interrupt that updates the total_xx variables may occur while you are using these values. You need to set a flag in the interrupt routine when all the total_xx values have been updated and then check and clear it  in the outer loop. This way you have a safe 20ms to do your calculations.

Presumably you plan to add phase adjustment later?

Any particular reason why you use a bunch of “if (sample_in_register == inPinx)” statements instead of a case statement?

Also wanted to say that you need to be very careful temperature sensing as the DallasTemperature library can lock out interrupts for long periods, that's why I didn't use it. I avoided using JeeLib for RF for the same reason but in this case I didn't actually try it so I don't know for sure that it interferes with the interrupt timing. 

I wonder if you are considering the implications of a variable gain CT input as demonstrated in AVR465.when developing this library.  To my mind this approach has some benefits, but also some potential pitfalls. 

Looking at the AVR465 code it seems that the HPF is applied to the raw sample.  I've not studied the code in detail or tried it , but is seems to me that this will introduce a DC offset each time the gain is changed and a period to settle back as the HPF ultimately removes the offset.

I'm considering implementing a variable gain and read elsewhere that it is part of the future emontx strategy, so I would be interested in how much of a problem this is and is the AVR465 code the right approach?

Will variable gain be implemented in the EmonLib?

Thanks Martin, yes phase adjustment to come. 

John, I'm not planning to add variable gain at this point and haven't really thought about the code implementation in detail yet. Would be great to have it though!

Trystan,

Nice that you're able to get the grid frequency accurately without resorting to PLL.  Have you figured out how many samples per grid cycle? I see your resetting your counters every 5 seconds (5000ms).   Since you're not using Martin's timer interrupt approach to starting the ADC, I was thinking you might be over sampling?

To measure frequency, you're using the voltage zero crossing (filteredV>0).  I am considering your approach for a "current only" sensor.  This would be a design of a current sensor and data logger (with I2C data delivery) using the ATtiny85 for electric vehicle charging station.

The ATtiny85's built-in clock is not very accurate (+-10%), so I was looking using the grid frequency to accurately time intervals instead of using millis().  I realize the grid frequency varies, but it's more accurate than the internal clock I'm working with. The ATtiny85's internal clock also varies with temperature, so grid frequency seems the best approach for my application.  Only 8 MHz, but I'm just looking at capturing data from one sensor to measure current draw and to estimate power consumption in kWh on one type of load (vehicle charging).

The sampling rate when all 3 CTs are enabled will be once every 416µs, or about 48 per mains cycle. This means that the per-cycle frequency measurement will have 416µs of jitter, but when sampled over enough cycles this will be averaged out. The PLL method has the advantage that the frequency measurement is accurate every cycle, but I doubt this gives any advantage for just displaying frequency.

The frequency of the internal clock on the ATiny is user programmable so a really neat solution would be to design a software PLL that adjusts the oscillator frequency to lock it to the mains frequency. I used this technique with a PIC chip for an MSF clock decoder and it worked really well.

Thanks Martin,

I missed that... Trystan's stated 104us "interrupt period"  is the interval between interrupts.  So for 3 CTs enabled, at 50Hz, it works out to 48.08 samples per grid cycle, and at 60Hz, it works out to 40.06 samples per grid cycle.

Good suggestion about using the user programmable feature of the ATtiny's oscillator calibration register!

Just tried adding the jeelib rfm12 code, it seems to work ok, will have to check sample timings in detail to be sure, but nothing is locking up.

Trystan, recently wrote on a Building Blocks page:

Why a low pass filter? The low pass filter was introduced with the Mk2 Energy Router which uses a common bias supply (buffered by an operational amplifier) to supply the offset voltage for both voltage and current channels. The filter only has to be calculated once, and the resulting offset is naturally the same for both channels and therefore can be subtracted from both readings. This saves a significant amount of processing time.

The LPF in the Mk2 PV Router is subtly different than the normal Wiki-style one.  Rather than updating the filter after every sample, which invariably results in ripple, it is only updated once per mains cycle.  Once the filter has settled, there is no ripple at all, hence there is no attenuation of the AC signal.  Because the output value is only required once per mains cycle, this technique works really well.  It's been there right from Day 1:

RAE 15/7/12:  "In the standard EmonLib code, there is a separate high-pass filter (HPF) on each of the voltage and current streams.  I’ve taken a different approach (thanks Robert for the suggestion), and have instead implemented a single LP filter which allows the DC bias to be accurately determined.  By updating the LPF only once per mains cycle (my idea!), its performance is nigh-on perfect, with no attenuation or phase shift.

Too good to believe?  Well, the LPF is not acting alone.  There is also a standard HP filter which acts just on the voltage stream.  The purpose of this secondary filter is to group the voltage samples into cycles so that the LPF can be accurately updated.   Unlike the LPF, the HPF can always be relied upon to start up correctly.  Together, they form a great combination."

Later versions of the Mk2 code avoid the need for the HPF by simple preventing the LPF output value from straying far from the mid-point value.  Providing that an AC signal source is present, the single LPF filter always starts up correctly.  Simple and effective.

For "real power", there is no need to remove DC offset from one of the sample streams.  Any DC component is removed automatically within the standard maths calculation.  There's more about this in Section 6 of my Surplus Power Diversion article.

The point that I made yesterday about how the LPF in my Mk2 PV Router operates is already covered near the end of the Building Blocks article. 

Thanks Robert for pointing this out, and sorry for any confusion.

See my post about EmonLibPro http://openenergymonitor.org/emon/node/2406

It does what you want.

I thought Id pick up from where I left off last time with interrupt based emonlib development (or EmonTX_CT123_Voltage firmware dev - as EmonLib in its current form is still useful for battery operation and easy, simple coding because of the library and discrete in time nature of it).

I have still been in need of an interrupt based direct replacement for EmonTx_CT123_Voltage as I have Robin's Solar PV diverter which because of its short pulsed power draws is not compatible with the current EmonLib, To be a direct replacement to my current setup I need ideally both RF support and DS18B20 support (I currently run a solar hot water controller + solarpv/whole house energy monitor on one emontx)

As I mentioned above it appeared on initial testing that adding in the RF sending code worked. I have uploaded an example with RF sending to github. I have also created an example for temperature sensing with a DS18B20.

RF example:

https://github.com/openenergymonitor/emonTxFirmware/tree/master/InterruptBased/emonTx_CT123_Voltage_Interrupt_RF

RF + DS18B20 example

https://github.com/openenergymonitor/emonTxFirmware/tree/master/InterruptBased/emonTx_CT123_Voltage_Interrupt_RF_temperature

There are a couple of things as MartinR pointed above that need adding or addressing in these examples:

- The first is the use of millis() in the outer calculation loop instead of a flag based system as the variables used may be changed by an interrupt mid calculation, this could mess up the calculation.

- Phase adjustment needs to be added

Testing

But I thought before going on to both of these things I would do some spot testing of how the current implementation compares to EmonLib, how it compares at different power factors and with RF and reading from a DS18B20 temperature sensor.

The testing consisted of:

• comparing 5 different load combinations made up of loads that give a range of power magnitudes and power factors.
• on two identical emontx builds running:
  • Current EmonLib
  • Interrupt based EmonTx_CT123_Voltage
  • Current EmonLib phaseshift set to 1
  • Interrupt based EmonTx_CT123_Voltage + RF
  • Interrupt based EmonTx_CT123_Voltage + RF + DS18B20

Results are in this spreadsheet:

http://openenergymonitor.org/emon/sites/default/files/Interrupt_based_EmonTx_CT123_Voltage_testing.ods

From these few tests the results seem good for powerfactors above 0.5 and most of the CT's even with RF sending and reading from a DS18B20 however there appears to be two issues that I cant immediately explain:

• The powerfactor recorded by the interrupt method is significantly below both the reference meter and non-interrupt emonlib result at powerfactors less than 0.5.

• The current EmonLib did not show a great change in powerfactor accuracy when phaseshift was set to 1 which was unexpected as I remember this having a marked effect when I first added the phaseshift calibration to the code as recommended and implemented in the AVR465 example.

• When running the EmonTx_CT123_Voltage + RF + DS18B20 code, the CT channel sampled immediately following a Voltage sensor sample has a spurious result every 5-10-20 readings where the rms current and apparent power values spike significantly, the other CT channels appear to be ok.

I wouldn't think that adding a flag based system or phase correction will fix the last issue so it must be something else, the only difference between the voltage input and current input is the magnitude of the waveform being sampled, that would suggest some sort of interference caused by the fast running switching of the ADC channels.

Edit: but then again the effect does only become apparent when the DS18B20 temperature sensing is added, which would suggest some timing influence. Could it be that the ADC sampling is being 'squashed together' because the atmega is busy processing all the additional RF and DS18B20 stuff. If this is the case maybe the ADC does not have enough time to settle between readings.