Archived Forum

Can you show some logs and tells us a bit about your setup, for example are you using an rfm2pi? 

emonHub isn't really geared up to filter by node id and whilst you maybe able to achieve this another way eg reduce the node id limit in emoncms (if it's a local install) it isn't ideal. if you are getting random nodes then the chances are you are also getting random packets with node ids that match your active nodes and these could fudge your data,. there is an extra "length of packet" check if you use all full "datacodes" strings for each active node but if a random packet of data, the expected size turns up it will get processed as good data.

Do you know the source?

Paul

PS maybe a moderator would be so good as to move this over to a new thread to save going off-topic here, since this is the closest we have to datacodes documentation :-)

Edit - done, Paul (moderator)

It's an rfm2piv2 on a custom readonly Debian wheezy install.
I currently only have one valid node 10 which is an emontx module. This is specified in the emonhub.conf with full datacodes definition for that node.
I do have a system of eq3 max heating devices (around 18 in total) which communicate over 868mhz and are received by a CUL device from busware.de so I suspect this is the cause of all the extra RF traffic.

I will provide some logs this evening when home.

Here's a log of the last couple of hours.  As you can see there's a lot of RF transmissions so I definitely need a way to filter the ones I want.

[Duplicate post deleted. Moderator (RW)]

Hi Matt, The log clearly shows alot of unwanted RF packets, however it's only showing the packets that emonHub is successfully filtering as the loglevel is "warning". These are not the packets you need to target.

Can you change the loglevel to "debug" and post again?

What group are you using? have you tried any others?

Group 0 is "open to all groups" and should be avoided

How close is the emonTx to the RFM2Pi ? If they are pretty close you may get an improvement by running your 868 rfm12's on 433MHz at a very reduced range but a different freq to the traffic. I haven't tried this but I'm inclined to think its feasible.

If they are very close is a wired solution possible?

We can definitely add a "node 10 only" filter if there's no other way but that's not a perfect solution as corrupt or alien packets starting with "10" can still slip through and this will be a change you make on your install only so will need to be re-implemented after updating emonHub in the future.

Log with debug enabled attached.  There's at least one packet that gets sent to emoncms spuriously in that one (node 19).

Group 173 is my home group.  But the group id of the spurious RF data is also random so appears in any group.

Not that close, and I've also got a couple of other devices (emonglcd, currently off) on the rfm12 network which need to be in different locations.  This is a "smarthome" with power/lighting/heating all using a mixture of wireless/wired protocols so the RF spectrum at 433Mhz and 868Mhz is noisy.  Forgot to mention 868Mhz also has ZWave traffic as well as the eq3 radiator thermostats so we're probably seeing that too.  There is also weather data being transmitted from the end of my garden.

I appreciate that there is a risk of valid node ids appearing in the random data but I would have thought we could come up with a solution where we check the length against the datacodes which would eliminate nearly all false-positives (note that all observed spurious RF are long datastrings).

I'm not likely to be the only one who will be using the system in a noisy RF environment and would prefer to see a solution that is part of the standard software that can benefit anyone who needs it.

Maybe something like the following:

Under the [nodes] section add a flag: restrictToDefinedNodes = 1 which when set would only treat the defined nodes and associated datacodes as valid.  Otherwise it passes everything as before.

Hi Matt,

I wasn't able to look into this over weekend but it was on my mind and I have thought of a couple of possible "enhancements" we could consider, You are right there maybe other users that might make use of additional filtering by node id, However I will need to look into the best way of implementing it. My current thoughts centre around a datacode "X" that can be used as either the interfacers default datacode to signify only "listed nodes" to be processed or as a "unwanted nodes" default datacode in the nodes section to block just that node.

This may offer a more usable solution but may not be instantaneous to implement, if needed we can always temporarily add a "node 10 only" filter I will have a look into it as soon as I get a chance.

Looking at your latest log I see the very long packet from node 29 has slipped through, Although it wouldn't eliminate all the unwanted packets in your first log, I do wonder if we should also have a "66 byte" cap to match the rfm's max payload and discard the obviously incorrect packets on arrival.  

Paul

At the cost of an extra byte, you may also want to consider a simple checksum appended to the message. It wouldn't help with blocking, but if implemented properly it would allow you to know if the message was corrupt or not observing to your protocol. One byte would not give an absolute guarantee of course, but in combination with a length and/or node check it would surely get most errors.

I agree, in fact I had actually presumed there was some form of "validation" in the JeeLib hence my initial question asking if Matt was using an RFM2Pi, followed by significant surprise when he said he was. I'm not totally up to speed on what happens "airside" of an rfm device, but can confirm my RFM2Pi handles alot of "noise" internally, if I disable "quiet mode" I see there's a lot of traffic that doesn't reach emonHub.​

Paul

As an experiment I've inserted into emonhub_interfacer.py a check for packetlen > 66:

        # Discard if first value is not a valid node id
        n = float(received[0])
        if n % 1 != 0 or n < 0 or n > 31:
            self._log.warning(str(ref) + " Discarded RX frame 'node id outside scope' : " + str(received))
            return False

        # Discard if packet len > 66 (max for rfm12)
        if len(received) > 66:
            self._log.warning(str(ref) + " Discarded RX frame 'string too long' : " + str(received))
            return False

        # If it passes all the checks return
        return received

Thinking ahead (and I'm aware there's some work in the experimental branch) we're going to want to support different types of serial data I think...

So we might want to specify in the config what is a valid packet for each datatype.  For example I'm processing a serial datastream from a mk2pvrouter which starts with the text "datalog event".

For an rfm12pi I guess we'd specify:

datacodes, maxlength.

​Specifying an X for the master datacode sounds like a simple solution for excluding other nodes.  Then we just need to add a maxlength parameter which defaults to 66.

Unfortunately I still seem to be getting packets through when I'm checking for max length of 66.  So think a node filter is required too.

In the mean time, for cleaning up the input list this post is useful: http://openenergymonitor.org/emon/node/3075

"I had actually presumed there was some form of "validation" in the JeeLib"

There's actually almost nothing in JeeLib, because he went for minimal packet size = minimum power. There is a checksum in there, but it won't help you if the 'foreigners' are also using Jeelib. The packet structure is here: http://jeelabs.org/2011/06/09/rf12-packet-format-and-design/
In fact, in the documentation, there's the comment "If [CRC] != 0 then rf12_hdr, rf12_len, and rf12_data should not be relied upon." and to me that indicates that you should be checking the CRC, it isn't done for you. (It is checked in the GLCD & Nanode sketches.)

Adding your own checksum with a different rule would of course allow you to identify your "own" nodes.

Just had a look at JeeLib etc and found the crc does indeed need to be implemented by the application, it also looks like this is done in the JeeLib version of RF12demo.

However the RFM2Pi sketches seem to depend on age and version, 

Matt - do you know which version RFM2Pi firmware you are running? and can you try adding "quiet = false" to emonhub.conf's [[RFM2Pi]] [[[runtimesettings]]] to see if there is were any "bad packets" being discarded?

This git commit suggests you where using the Full_RF12demo_atmega328.ino in Aug 2013, in a later commit it looks like there was a "bug fix: packets with bad CRC were not being rejected" in Jan 2014.

So there is a case for the possibility you're seeing all packets good and bad as crc is not functioning correctly.

Robert - I agree another checksum would add to the certainty, but unless I'm misunderstanding, I think the group is used in the checksum so even if the neighbors are using JeeLib a group change should overcome that, no?  

I'll check this evening and upgrade to the latest recommended firmware.  I'm using the version from my github: https://github.com/mattwire/RFM2Pi/commits/master/firmware/Full_RF12demo... which does ref the Jan 2014 crc fix but I don't remember if I flashed that version or not.

"Robert - I agree another checksum would add to the certainty, but unless I'm misunderstanding, I think the group is used in the checksum so even if the neighbors are using JeeLib a group change should overcome that, no?"

You're probably right there - I was getting tired and not thinking all the way through it! As the checksum at the end of the calculation should be zero, it's probably (without reading the library code!) just a sum (mod 256) of all the bytes in the packet, so everything will be included. It's not a bullet-proof test of course, but the probability that the sum is wrong by exactly 256 has to be quite slim - unless it's all zeros. If you're suggesting that the use of the checksum was missed, then that goes a long way to explaining a lot of the junk that people are seeing.

 If you're suggesting that the use of the checksum was missed, then that goes a long way to explaining a lot of the junk that people are seeing.

I'm a little hazy on this myself, but as I understand it the "stock" RF12_Demo_atmega328.ino that was/is installed to all shop bought rfm12 RFM2Pi's will ONLY pass packets that have a zero checksum, despite having a "q" or "quiet" option the quiet boolean variable doesn't appear to be utilized in the code anywhere. So you can enable/disable "quiet mode" but you will actually only ever get just the packets with a zero checksum.

However the original version of the Full_RF12demo_atmega328.ino (I believe) was intended to work the same way, ie pass just good packets with no option to disable "quiet, but the later bugfix suggests the CRC wasn't previously working and therefore I assume good and bad were passed until the bugfix was applied.

So in theory only the users that installed the "Full" version between june 2013 and jan 2014 should be seeing bad packets passed as good.

This also explains why my own rfm12 RFM2Pi shows no bad packets when I believe "quite mode" is disabled and yet the rfm69 RFM2Pi is bombarded with bad packets, I thought it was the better receiver picking up signals from far afield but no, apparently it's just the 12 isn't able to show them.

Looking at the latest RFM2Pi firmware which is installed to the shop bought rfm69 RFM2Pi's and looks compatible with an rfm12 RFM2Pi too, both the CRC and the "quiet mode" seem enabled for both rfm12 and rfm69, I can confirm they work as expected on a rfm69 RFM2Pi so it could worth trying a firmware upgrade to the RFM2Pi before going any further.

However there's alot of assumption in there so I may be well off base, especially as I'm not fluent in "arduino" yet.

Curiosity got the better of me as usual and I thought I would try the various firmwares, unfortunately I couldn't get the early "full" version to run on my rfm12 RFM2Pi and suspect it may due be a change in JeeLib rf12_config() is depreciated in favor of rf12_configSilent() and rf12_configDump(). 

As I didn't fancy installing an old JeeLib or updating the sketch I skipped the later version too and just tried installing the latest RF12demo.12 aka the rfm69 sketch to the rfm12 RFM2Pi and found many bad packets that were silently discarded previously by the "stock" rfm12 firmware.

To use this latest firmware on rfm12, 2 lines need editing before compiling

#define RF69_COMPAT 1     to      #define RF69_COMPAT 0

and

#define SERIAL_BAUD 38400      to       #define SERIAL_BAUD 9600

With RF12demo.12 installed you will also get the version returned when using "v" command, so the auto-baud selection in emonHub "can" work, I say "can" because it doesn't appear to work every time. But we are aware the "auto-baud selection" needs attention and it can easily be skipped by defining the baud in emonhub.conf.

The quiet mode can also be enabled/disabled with command "q" .

Paul

I've updated to the latest Full_RF12 firmware: https://github.com/openenergymonitor/RFM2Pi/commit/2dd48a9bf6497390966fc69df6ece953732526ce

I also tried the RF12_demo (cutdown) version too and both showed node 10 and occasional node 22 (which doesn't exist) so I'll leave it til tomorrow and see what we get!

As of 1100 today I only have one spurious node (22) with 9 values, triggered 9 hours ago.  So much much better with the updated rfm12pi firmware!

I do still have the custom packetlen < 66 in emonhub_interfacer.py as described above so not sure if this is having any impact.  Now it is "stable" I'll probably leave this for a few days before coming back and investigating further.  Got a few other projects to sort out :)

Matthew

Good to hear it's improved, we can still consider a selective "filter by node id" maybe, the 66byte limit however may not have the expected results, firstly there are instances where the "sent payload" is added to eg the rfm69 rssi, the node, possibly the group (if group 0 used) so may need more thought, plus the if >66 may be counting characters rather than bytes when performed on a string.

See how it pans out and we can look into these things if needed. If the node 22 is reoccurring and the only anomaly it may be worth trying a group change, there's no guarantee it will help but if the stray foreign packet is consistent (and alone) a group change should change the fixed element of the checksum.

Paul

Last night I removed the check for length 66 and changed my system to group 201.  Now there are no unexpected packets!