Archived Forum

The NILM project started in the 80's at the MIT by George W. Hart and his colleagues (he called it NALM though "Nonintrusive Appliance Load Monitoring). The first approach, that was patented by them, used hardware to estimate any power step change and managed to locate mainly resistive loads with a clear step change. The approach in Washington is to monitor the noise and harmonics that resonates over the phase when appliances are turned on or off or change their state. This approach requires high frequency monitoring and there might be a good ADC to translate this for the Arduino. A combination of these two approaches is currently being persued at MIT to detect and monitor large loads like climate controlling units in commercial buildings. 

I have written a small moving window analysis of the incoming data for my arduino. The step changes are followed and once they are above a certain threshold they are matched against known measurements. I haven't had time to tidy up the code but I think you'll be able to follow it. Right now it only tracks one entity of power and only one phase but you can easily add the other measurements to get a more sophisticated recognition. If you have any questions or suggestions please let me know. 

PS. Finding appliances today cheaply (without high frequency measurements) is really hard today since many of our appliances consume power very differently depending on how they are used and therefore blend together in measurements. You should be able to find the bigger appliances though with this code (washing maschine heater, toaster, electric kettle... - which is still fun )

 

// Simple moving window or "t-test" statistical analysis 

// of power measurements. 

// Inspired by great projects like the MIT NALM and many others

// Code scribbled by Anders Dalen and edited by Timm Teubner

//=========================================================================

// Appliance disaggregation variables

//=========================================================================

// Incoming values from energy measurements 

// (code not included here)

double measuredPower;

// Constants to compare step change to known values

const int minStateChange = 100; // Minimum 10W to be a state change

const int stateCangeTolerance = 100;

const int knownAppliances[] = {1400, 300, 1150};

const char* knownApplianceNames[] = {"Electric Kettle","Iron","Washing Maschine"};

// Setting up window constants and variables

const byte windowSize = 10; // Has to be even (saves a loop)

double window[windowSize];

int stepCount = 0;

boolean beginFound = false;

double beginLevel = 0;

boolean switchEvent = false;

// Variables for t-test

int switchStart;

const double tValue = 2.8; // Sensitivity level for first step change

double sum1=0, sum2=0, s1=0, s2=0;

double x1Hat, x2Hat;

double t0 = 0;

double diff, denom, change;

void loop()

{    

  // Just for pointing out when start is found in log

  switchStart = 0;

  

  // tTesting  

  for (int i=0; i<windowSize-1; i++) {

    window[i] = window[i+1];

  }

  window[windowSize-1] = measuredPower;

  

  sum1=0, sum2=0;

  

  for (int i=0; i<windowSize/2; i++) {

    sum1 += window[i];

    sum2 += window[i + windowSize/2];

  }

  

  x1Hat = sum1 / (windowSize/2);

  x2Hat = sum2 / (windowSize/2);

  

  // 2. Calculating sum of squared deviation from mean

  s1 = 0, s2 = 0;

  for (int i=0; i < windowSize/2; i++) {

    s1 += (window[i]-x1Hat) * (window[i]-x1Hat);

    s2 += (window[i+ windowSize/2]-x2Hat) * (window[i+ windowSize/2]-x2Hat);  

  }

  

  // 3. Dividing squared deviations to get sample variance

  s1 = s1 / (windowSize/2-1) + 1; // adding +1 in order not to run into too high t-values when there is no standard deviation at all...

  s2 = s2 / (windowSize/2-1) + 1;

  

  // t-Test calculation

  diff = abs(x1Hat - x2Hat);

  denom = sqrt((s1 + s2) / (windowSize/2));

  t0 = diff / denom;

  

  // Finding the beginning of the step change

  if (t0 >= tValue && !beginFound) {

    beginLevel = x1Hat;

    beginFound = true;

    switchStart = 1;

  }

  

  if (t0 >= tValue && beginFound) {

     stepCount += 1;

  }

  

  // Finding the end of the step change and saving the change value

  if (t0 < tValue && beginFound) {

    change = x2Hat - beginLevel;

    switchEvent = true;

    beginFound = false;

  } else {

    change = 0;

  }

  // I haven't tested this part, just put it here as an example of how one might write out a recognized step

  if (change > minStateChange){

    for (int i = 0; i < 3; i++){

      if (change > knownAppliances[i] - stateCangeTolerance || 

        change < knownAppliances[i] + stateCangeTolerance){

        Serial.println(knownApplianceNames[i]);

      }

    }

  }

}  

 cheers!

 Schepparn, that's some pretty cool implementation!

Approximatation of the larger loads is usually "good enough" to track and trend the majority of energy loss. Of course, it won't capture every vampire load but will catch the big players. And even if it's got to be "hand tuned" to certain appliances over a week or two (something big corps. would NEVER sell) it's still something to explore and learn about the process.

Now I gotta buy the hardware! I've been exploring this field for a few months and the only option I resigned to was an army of Tweet-a-watts. Maybe this combined with a few tweet-a-watts will take care of the vast majority of loads. Now to find the time...

Thanks!

I'm interested in load disaggregation so as to be able to infer activities of daily life of the elderly living alone.

I have been running a small scale trial to collect household power and water use data. I have studied disaggregation schemes and would like to co-operate with like minded individuals to develop effective open source algorithms.

I've been using current only monitoring because I've been using an off the shelf monitor from Current Cost. I believe that current and voltage monitoring, probably with 24 bit resolution and maybe transient acquisition may be required to effectively disaggregate all but the major appliances. One of the problems is that I want to have a sensor that users can self install without having to employ an electrician so getting voltage date is hard. Onzo and Navetas do this (I think using different approaches) but it all seems to be tied up in patents. Onzo have also done lots of work on disaggregation in the cloud. I'd prefer to see this done in the home - and the system should, ideally, learn appliance characteristics from acquired data rather than from an appliance database and user configuration.

Any ideas welcome.

 

Just a very quick reply for now (would love to write more but am racing to meet a deadline!)

I'm 1 year into a PhD on disaggergation and my ultimate aim is to produce an open-source software package for disaggregating aggregate data from off-the-shelf meters like Current Costs and emonTXs (and "proper" smart meters when they start to be installed).  I'd love to help out (or collaborate!) in any way...  if you're based near London then perhaps we could discuss over a coffee?

I agree that being able to resolve changes of 1 or 2 watts (or less) is probably important.  As is having readings of both real and reactive power.

I wrote a series of three blog posts on disaggregation here: http://jack-kelly.com/taxonomy/term/104

And here's my literature database: https://www.zotero.org/jack_kelly/items

Doing disaggregation well isn't easy (and no body is able to do it with anything like 100% accuracy for all appliance types yet)

I think some appliances will have a definite signature (a lamp, for example). Others will have more than one signature (a dehumidifier will probably have standby, running, and defrost modes, but all will be a predictable load. But then you come to devices like laptops where playing a game or watching a full screen movie (no, not THAT type!) can cause the standard power consumption to double or more. 

So I think a mixed approach would be needed. Central monitoring will be fine for certain loads, but you're going to need appliance based monitoring for devices like computers, TVs, etc.

Having said that, a signature isn't just current - it's also phase angle, and that might help to determine what a particular load is.

Also I guess it depends whether you want to monitor a 20W CFL somewhere in your house going on, or a particular bulb. If you're content with knowing a bulb has been switched on somewhere then it's easy, but if you want to know which particular bulb it is then the monitoring is far more complex.

If you'd like to see signatures of lots of different appliances then see http://www.tracebase.org/

Been a year or so - glad to see there's activity on this front!

Jack - awesome work! Best of luck with your PhD process and I sincerely appreciate your open sourcing this work. It takes a lot of time to develop the algorithms involved and having even a basic open source implementation of NILM would really help the effort along!

Have you been in touch with Mario Berges or Zico Kolter at Carnegie Mellon University in the US?They are both working on the NILM problem indepth and have begun to address the issue of the lack of data sets.

Mario Berges: http://tbc.ce.cmu.edu/people/faculty/berges.html

Zico Kolter: http://www.cs.cmu.edu/~zkolter/

1st International NILM Confernece abstracts/slides: http://www.ices.cmu.edu/psii/nilm/agenda.html

Tons of great resources there.

 

Hi Electron,

Yes, as you say, Zico Kolter and Mario Berges are both doing amazing work.  I'll been following their research output for a while now.  All great stuff!

By the way, if you are based in the UK and are working on some aspect of smart meter disaggregation / Non-Intrustive Load monitoring then please get in touch; we're hoping to plan a UK disaggregation meetup ;)

It looks like the Belkin stuff mentioned in the original post is getting closer to being a product (for both power and water):

http://www.belkinbusiness.com/everything-connected

http://www.belkinbusiness.com/echo-water-0

Many thanks for those interesting links!

Just a very quick note that a few of us are toying with the idea of setting up a wiki and/or online community to support electricity disaggregation (aka NILM / NALM / NIALM) researchers.  Some more details here.

As a forum newbie and complete idiot in the area of electronics, please could I ask for a pointer?

We have had Solar Thermal panels added, and are having problems with the Wagner control system.  Our installer has done his best, but has not fixed it yet. Thus I am attempting to monitor and log the temperature profile of our heat store cylinder that supplies heat for DHW and the underfloor heating.  That looks pretty straight forward with a Raspberry Pi and I am well on the way with that bit.

It would be nice to add into the log the state of the underfloor heating pump (on/off) and the solar thermal circulation pump (on/off).  In searching this site I have only come up with this Non-Intrusive Load Monitoring thread.  It has been very interesting (if mostly beyond my level of knowledge) but i s there a thread that looks at what I am seeking? Alternatively, can any of you experts point me to a nice road tested and proven way of generating a signal for the Rapberry Pi?

Many thanks.

I would suggest connecting a mains coil relay in parallel with each pump and using the isolated contacts to connect to your processor to log the on and off state of each pump.

Simple , safe and relatively low cost. suitable relays on ebay and the usual suspects.

 

HTH

Bob

We've just started work on an open source software toolkit for doing disaggregation: nilmtk.  It will be a good few months before it does anything useful though ;)

Wow, the screenshot you have there on your github https://github.com/nilmtk/nilmtk/ of load disaggregation looks amazing, is that being generated now?

Hiya!  Ooops, no, we should definitely make it clearer that nilmtk can't do that yet!  It was just an illustration of what NILM can do ;)  Sorry, I'll make sure we get that clearer!!!

Some time soon (2-3 months?  Maybe 6 months) nilmtk should be able to do that, though ;)

Hi,

I am Nipun and I am one of the contributors of nilmtk https://github.com/nilmtk/nilmtk/

The image used comes from my prior publication:

Nipun Batra, Haimonti Dutta, Amarjeet Singh, “INDiC: Improved Non-Intrusive load monitoring using load Division and Calibration”, to appear at the 12th International Conference on Machine Learning and Applications (ICMLA’13) will be held in Miami, Florida, USA, December 4 – December 7, 2013 [Preprint] [IPython notebook]

The entire code is made available as open source and you may refer to the IPython notebook linked above.

Thanks for interest!

I would like to bump this up in sight again. I was searching for ways to determine which devices are on a single ct and separate them from each other. After reading some papers and interesting webpages, I came across the nilmtk framework. I think it has very high potential.

I am thinking of building an exporter/importer for emoncms to nilmtk. I already did some basic tests with the export to csv function and importing that data into pandas, but I will need to make some specific conversions in order for the framework to work with the dataset. If anybody else is interested in joining the research/testing, I might be uploading it to github.

I am also very interested in trying this...

Definitely looking into this when I have some time.

I am interested too.

I am actually testing for work purpose a commercial nilm device that I cannot name and would love to have OS win...

I've stumbled across this before but have never looked deeper than their webpage: http://www.smappee.com/uk/