Archived Forum

Thanks for the kind words Glyn!

The latest changes have now been merged into the default "development" branch as they seem quite stable and thoroughly tested. so no need to go to the "testing" branch as that will now be subjected to further experimenting.

The most significant change is for rfm69 and but there are some other less exciting changes since rc1.0

  Jee/RFM2Pi serial baud auto selection (currently selects 9600 or 57600 depending on device firmware)
  Display RFM2Pi firmware revision and current settings (post vRF12demo.11 firmware only)
  Improved log messages
  Improved settings qualification
  Widened compatibility with other Jee type devices
  Reintroduced the float datacode "f"
  Option of using pre-timestamped payloads via serial or socket for non-live data input
  Cleaner exiting for multi-threading
  Plus various other minor improvements

To get emonhub rc1.1 you can pull the changes in from git, remember use "rpi-rw" first if running read-only.

cd ~/emonhub
git checkout development
git pull
sudo service emonhub restart
cd -

Paul

The update does not appear to auto select the correct baud rate as suggested above.

I'm using a RFM12B, and following the update above (I was already on the emonhub 'development' branch) it appears from the emonhub error log that it has selected 57600 baud, which has caused instability in my system and stopped all feeds being updated.

Any ideas what has gone wrong?

Paul

tut.. there always has to be one :-)

Sorry not off-hand no. I've been using that part of the code for months now with no issues, It would be a great help if you could post the log so I can see the steps it's taken.

emonhub tries to communicate with the rfm2pi at each baud and if it doesn't get a the expected response it moves on to the next baud, so that would suggest it got an acceptable response which is not a surprise if it is then working to some degree. Do you know what firmware you are running on rfm2pi? is it stock or modified? 

I have tested with as many devices and firmwares as possible but may need to "tune" the selection process if I've missed something.

The auto-selection can easily be overridden by just defining the baud you want in the rfm2pi init_settings eg "com_baud = 9600", i assume there is no pre-existing baud set in emonhub,conf.

Paul

I emailed you the log last night Paul to your outlook address, but now pasted below.

The RFM2Pi firmware is stock (preloaded - but was purchased direct from Martin Harizanov - not from OEM shop) and there are no 'baud' entries in the emonhub.conf

Paul

2014-12-09 21:25:31,148 INFO Set up reporter 'emonCMS' (buffer: memory | size: 1000)
2014-12-09 21:25:31,153 INFO Setting emonCMS url: http://localhost/emoncms
2014-12-09 21:25:31,155 INFO Setting emonCMS apikey: set
2014-12-09 21:25:31,158 INFO Creating EmonHubJeeInterfacer 'RFM2Pi'
2014-12-09 21:25:31,163 DEBUG Opening serial port: /dev/ttyAMA0 @ 57600 bits/s
2014-12-09 21:25:35,189 WARNING Device communication error - check settings
2014-12-09 21:25:35,192 INFO Setting RFM2Pi frequency: 868 (8b)
2014-12-09 21:25:36,196 INFO Setting RFM2Pi group: 210 (210g)
2014-12-09 21:25:37,199 INFO Setting RFM2Pi quiet: 1 (1q)
2014-12-09 21:25:38,204 INFO Setting RFM2Pi baseid: 15 (15i)
 

The auto-selection can easily be overridden by just defining the baud you want in the rfm2pi init_settings eg "com_baud = 9600"

Added, and I'm back on 9600 baud with the feeds now updating OK

If you have any ideas why it didnt auto select, let me know, and I'll try them out.

Paul

I have just tried this several times with my rf12 rfm2pi and it's working exactly as expected, so I'm at a bit of a loss as I can't recreate it, heres my "set-up" part of the log. from there it logs as usual.

2014-12-10 17:28:40,636 INFO EmonHub Pre-Release Development Version (rc1.1)
2014-12-10 17:28:40,639 INFO Opening hub...
2014-12-10 17:28:40,645 INFO Logging level set to DEBUG
2014-12-10 17:28:40,649 INFO Creating EmonHubEmoncmsReporter 'emonCMS'
2014-12-10 17:28:40,655 INFO Set up reporter 'emonCMS' (buffer: memory | size: 20000)
2014-12-10 17:28:40,661 DEBUG Setting emonCMS interval: 5
2014-12-10 17:28:40,664 INFO Setting emonCMS url: http://emoncms.org
2014-12-10 17:28:40,668 INFO Setting emonCMS apikey: set
2014-12-10 17:28:40,672 INFO Creating EmonHubEmoncmsReporter 'emonCMS2'
2014-12-10 17:28:40,678 INFO Set up reporter 'emonCMS2' (buffer: memory | size: 20000)
2014-12-10 17:28:40,683 INFO Setting emonCMS2 url: http://emoncms.org
2014-12-10 17:28:40,686 INFO Setting emonCMS2 apikey: set
2014-12-10 17:28:40,690 INFO Creating EmonHubJeeInterfacer 'RFM2Pi'
2014-12-10 17:28:40,696 DEBUG Opening serial port: /dev/ttyAMA0 @ 57600 bits/s
2014-12-10 17:28:42,702 DEBUG Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2014-12-10 17:28:46,712 INFO RFM2Pi device firmware version & configuration: not available
2014-12-10 17:28:46,716 INFO Setting RFM2Pi frequency: 433 (4b)
2014-12-10 17:28:47,720 INFO Setting RFM2Pi group: 210 (210g)
2014-12-10 17:28:48,724 INFO Setting RFM2Pi quiet: 1 (1q)
2014-12-10 17:28:49,729 INFO Setting RFM2Pi baseid: 15 (15i)
2014-12-10 17:28:50,734 DEBUG Setting RFM2Pi interval: 5
2014-12-10 17:28:50,737 INFO Creating EmonHubSocketInterfacer 'Socket'
2014-12-10 17:28:50,741 DEBUG Opening socket on port 50011
2014-12-10 17:28:50,767 DEBUG RFM2Pi broadcasting time: 17:28
2014-12-10 17:28:50,772 DEBUG RFM2Pi acknowledged command: > 4b
2014-12-10 17:28:51,223 DEBUG RFM2Pi acknowledged command: > 210g
2014-12-10 17:28:51,671 DEBUG RFM2Pi acknowledged command: > 1q
2014-12-10 17:28:51,897 DEBUG RFM2Pi acknowledged command: > 15i
2014-12-10 17:28:52,364 DEBUG RFM2Pi acknowledged command: > 0s
2014-12-10 17:28:52,590 DEBUG RFM2Pi confirmed sent packet size:  -> 4 b

I will continue to look into it and study the RF12Demo sketches, 2 details could help me out, can you confirm what firmware the rfm2pi is running? and if you have minicom or similar could you stop emonhub for a moment and send a "v" to rfm2pi and post the results and also a "?" the responses to these commands are what emonhub is looking for.

Something else just occurred to me, can you confirm if you are auto-starting emonhub from boot or if you have started/restarted emonhub alone using commands ie not rebooted?

Paul

edit - just reread your post above

I did a full reboot after the update, and emonhub normally starts as a service automatically.

Regarding minicom; I've installed it, and can see the data arriving in dev/ttyAMA0 but unsure how to send a "v".... I've tried typing, "v" but it just brings up the minicom commands list.

(It's taken me 5 minutes to find out how to exit from minicom!!! what a awkward piece of software)

Paul

Since I'm not able to reproduce this, I'm finding it hard to pinpoint what could be happening. My current thoughts are along the lines of either there is a difference in the firmware's in question which is not complying to emonhubs expectations, which can only really be determined by seeing the serial response to the commands used, or I vaguely recall some months ago when I first put the code together, I experienced some issues with the rf12 rfm2pi at boot up, this was overcome and hasn't reoccurred in many months, but without experiencing the issue I'm sort of clutching at straws here.

The crux of the issue in the early attempts seemed to be that the rfm2pi was printing many lines of help text at power up, then when emonhub sent out the test command and caught the end of the help text it thought it was in response to it's test command and accepted the current settings as correct.

If that was to be the issue it would only fault at power up and first use of the rfm2pi, When emonhub is restarted using "sudo service emonhub restart" while the Pi remains powered up and the rfm2pi is already initialized, the auto selection would then work as expected.

If you want to try this you will need to "comment out" the new "com_baud" line before restarting emonhub

sudo service emonhub stop

Then edit the conf file (sometimes it can take several seconds before all the threads are closed as it tries fo finish posting before closing).

sudo service emonhub start
tail -f /var/log/emonhub/emonhub.log

If you cut and paste these 2 lines in together and [enter] it will start emonhub and switch immediately to the log so you don't miss the start-up lines, if it still didn't auto-select correctly, exit log [ctrl-c] and uncomment the new line in the conf again. When you save the file it will automatically delete and recreate the interfacer with the set baud, no need to restart emonhub or the Pi, 

As you save and exit the conf if you immediately hit the scroll up key(twice probably) to re use the "tail log" line and [enter] you should just be able to catch the interfacer being deleted and recreated in the log.

If it's not that I'm (currently) lost, if it is that we know what were up against.

Paul

Edit - If you are ok having defined the baud in the conf I can wait to see if Glyn is able to recreate this on a non-live system rather than interfering with your data collection. it's up to you I do understand if you have it working you may not want to experiment at the cost of lost datapoints.

Hi guys,

Good news, I've fixed the issue I was having (same as Paul Reed) https://github.com/emonhub/emonhub/issues/58#issuecomment-66480108. The issue was I had got confused as to which emohub.conf file I was using, one of the files has a line in it overwriting the baud rate to 57600. Reverting back to the stock SD card image and using the emonhub.conf on the boot partition everything works just perfectly :-)

Emonhub now auto detects RFM12B 9600 baud and RFM69CW 56700 baud. I tested running emonhub at boot and manually and both on a Pi Model B and Pi Model B+. Every time ot successfully auto detected the baud. 

I'm using the emonhub.conf  file on the /boot partitionI was posting to a remote emoncms server (not emoncms.org but a test emoncms server we have running here in the lab) with local logging the to Pi disabled (as standard on the ready-to-go) SD card. 

I tested with three separate  RFM12Pi's with RFM69CW all running the latest: https://github.com/openenergymonitor/RFM2Pi/tree/master/firmware/RFM69CW_RF_Demo_ATmega328/

And four separate RFM12Pis with RFM12B all running the same code that's been on the last 1500 PCS of RFM12Pis that we've sold, we've not changed the code on the RFM12Pi with RFM12B since we first lauched the unit in the shop as far as I can remember: https://github.com/openenergymonitor/RFM2Pi/tree/master/firmware/RF12_Demo_atmega328

@ Paul Reed: if you have a spare SD card could you try flashing a fresh emonSD 13-08-14, edit the emohub.conf on the /boot partition and do a git pull and test if the serial baud auto detection is working for you? 

Thanks Glyn, that's fantastic news (for emonhub not so much for Paul though), is it possible a rfm2pi supplied by Martin has a different/earlier RF12demo sketch?

Paul

The rfm2pi was the second version (which fits into standard pi cases) and was purchased from Martin around 5th March 2013 (which was before oem was stocking them).
I'll ask Martin if he can recall.

Yes very good chance it could have slight differences, I don't have control over what Martin loaded nor do I have one here to test with. The good news is that you should easily be able to update to our 'stock' firmware direct from the Pi by following: http://wiki.openenergymonitor.org/index.php?title=RFM12Pi_V2#Upgrading_RFM12Pi_Firmware_Direct_from_the_Pi

Guys,

I think the firmware version I shipped these around March 2013 is this one: https://github.com/mharizanov/RFM2Pi/tree/d570b55221a2aa03da161e2c569cd4b6950a1364/firmware/RF12_Demo_atmega328

​

On another subject, I see you plan to use 57600 baud on 8Mhz clock, that would by default introduce 3.7% error possibility, check http://www.wormfood.net/avrbaudcalc.php?postbitrate=57600&postclock=8

This is the reason I used baud 38400 for the bootloader on the RFM2Pi. It may seem little, but if you think about it, 4 out of 100 bytes received on the serial may be garbage.. better use 38400 instead of 57600 baud.

Hi Martin, thanks for the link, I don't see anything obviously different with that sketch but will look closer at it.

Thanks for the pointers on baud too,I think I have seen 38400 in use on another Jee type device.so may include it in emonhub's short list anyway.

I've also compared the two sketches and found just two changes, lines 42 & 43 (Stack size  increase), other than that they appear to be the same.

I Followed Glyn's advice above re updating to stock firmware, and after removing "com_baud = 9600" from init_settings, restarted emonhub, and 9600 baud was auto successfully selected.

So in short, I'm not sure what the problem was, but all good now!

Thanks

Paul

@mharizanov,

 > 4 out of 100 bytes received on the serial may be garbage..

Hmm, not so - this is still an asynch link, each frame reception is independent (within limits) of the previous.

Net effect is that disagreement over clock rate between transmitter and receiver does not  accumulate during a block transmission - if the clock error is well below half a bit time by 'character' frame end, it's cool.  3% is within bounds unless driving capacitive loads like very long cables that could induce data dependent skew.

edited for clarity 17Dec

What is a frame in this context, is it a start bit, 8 data bits and a stop bit?

Yes.  Ideally the receiver samples the incoming waveform at the center of each bit cell in the frame. Clock skew will cause the sampling point to drift away from the cell center progressively for each bit cell sample. Provided the skew is less than half a cell width by the end of the frame, it is harmless.

The next 'start' bit kicks off the same process again with no error accumulation from the previous frame.

unless driving very long cables

And for the purpose of this discussion, you don't start measuring that length until the signal has been converted from USB to asynchronous serial.  On most Arduinos, the serial "cable" is actually PCB traces.  If you're using an external programmer, then it includes some traces on the programmer, the 6-pin header, and then some traces on the target board.  Either way, a much nicer environment than metres and metres of serial cable.

As another datapoint, I run a Mega2560 at 8MHz and do all image downloading at 57600 baud.  My hex file is about 100K and I've downloaded it hundreds of times with no problems at all.   It doesn't work at all at 115200, and I guess the error table linked above explains why...  7.8% error there.

I'd welcome any help troubleshooting the following issue.

I have two Pi side by side, one with the RFM12 and one with RFM69 which I just had delivered from the oem shop.

compare the following minicom output from the old board:

$ minicom -b9600 -D/dev/ttyAMA0
22 25 0 247 1 125 0 0 0 0 0 176 2 109 2 0 0
 -> ack
 7 24 0 5 0 98 0 44 2
 -> ack
 25 91 0 0 0 72 2 27 0
 24 70 0 0 0 240 2 24 0
 20 25 0 71 2 130 0 0 0 0 0 0 0 168 0
 -> ack
 25 91 0 0 0 75 2 27 0
 21 27 0 0 0 41 0 35 0 52 0 92 0 96 0
 -> ack
 24 70 0 0 0 240 2 24 0
 22 25 0 247 1 125 0 0 0 0 0 76 2 89 2 0 0
 -> ack
 25 90 0 0 0 74 2 27 0
 24 70 0 0 0 240 2 24 0
 20 25 0 71 2 130 0 0 0 0 0 0 0 172 0
 -> ack
 25 90 0 0 0 76 2 27 0
 21 27 0 0 0 41 0 35 0 53 0 92 0 96 0
 -> ack
 24 70 0 0 0 240 2 24 0
 22 25 0 244 1 126 0 0 0 0 0 139 2 68 2 0 0
 -> ack
 7 24 0 2 0 105 0 58 2
 -> ack
 25 90 0 0 0 76 2 27 0
 24 70 0 0 0 240 2 24 0

With the following from the second Pi (simultaneous):

$ minicom -b57600 -D/dev/ttyAMA0

OK ²š          ‚‚‚ÒÑÁA2 Ҳ4 ° ¨KM§%DHøOK µ² ?????M$&ÔS  IԊš‚‚‚ÁA0 0 0 0 ±·² ˜JK&¦RÔò
                                                                                           ԍ­5OËˠ™M¤'      
ي Kµ굳 ™        ‚‚¢Š‚‚Ík 0 ͵³˜HN&*²‚‚ªÙÁR 
    ­5
       OK 2´           ‚‚‚’¢‚ÉA24 0 (-60) 
OK 5ɴ ™M$&”)Š’²‚‚‚‚‚‚ŠšÊʂ’‚ÙáA2 0 0 ٨KM¦%DÔò
                                                      ԍ­5OK ³¹ ²šLH&2‚ª‚ªÂ’Bj²ŠÊʂj)]> ack
OK ²µ N&$       º²’‚’Ý݁a (-58) 
OK 24 70 ° ˜LH&        I’¢‚ÁQ-56) …OK 5² ™M$&ÔS      IԊš‚‚‚‚Áa 0 0 0 0 170 0 (­´±J¤Ôò
                                                                                           ԍ­5OK ™M$§       ٭¾X¬­HøOK 24 ·°?????˜H&$&     I’¢‚Âj²‚Êʂ*)ÿOK µšLM$&”SŠ‚ŠÉÝ@0 0ɠ°˜HL¨¦S’‚‚‚ÉɁa 0 (-60) Š -¾ áãë     á

I can sometimes get a sane looking frame (emonHub has managed to pass data to emonCMS for a couple of my nodes), but still... obviously something is seriously amiss!

I'm at a loss, unfortunately. I've confirmed that the RFM12 board gives the same output in minicom on either Pi, so the issue is definitely in the way I'm interfacing with the RFM69 board.

Glynn, any thoughts on this? I don't really know how to proceed since using minicom is about as low level as I have any understanding of. I've tried a couple of different baud rates, but as you'd expect that doesn't improve anything...

Clearly the asynch driver is disturbed by the RFM69CW activity, presumably at interrupt time.

Most individual lines start ok and some data is intact at line end. The (-nn) is the new RSSI indication. Is received buffer length important to minicom?

Hi guys, 

I think we could be experiencing exactly what Martin mentioned in a post above...thanks a lot for your insight Martin. 57600 baud is out of spec for an ATmega328 @ 8Mhz. 38400 is the max baud should be used. It seems that most of the time we get away with working out of spec, however as you have shown this is not always the case. 

I will re-compile the RFM12Pi RFM69CW code today and issue a flashable .hex file.  

Paul: Will emonHub recognise 38400 baud? 

Updated .hex files and sketch on GitHub to use 38400 baud https://github.com/openenergymonitor/RFM2Pi/tree/master/firmware/RFM69CW_RF_Demo_ATmega328

​emonHub does not currently autodetect 38400 baud. It must be set in emonHub conf under init_settings of EmonHubJeeInterfacer

com_baud = 38400

OK, sweet. I'll give this a try when I get home and update.

Update: emonhub does now auto detect 38400 baud if your using the latest 'testing' branch. Expect merge into 'development' soon. https://github.com/emonhub/emonhub/issues/58

OK, cooking on gas!

pi@pi-dev ~/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328 $ minicom -b38400 -D/dev/ttyAMA0

Welcome to minicom 2.6.1

OPTIONS: I18n 
Compiled on Apr 28 2012, 19:24:31.
Port /dev/ttyAMA0

Press CTRL-A Z for help on special keys


[RF12demo.12] O i15 g210 @ 433 MHz q1

Available commands:
   i     - set node ID (standard node ids are 1..30)
   b      - set MHz band (4 = 433, 8 = 868, 9 = 915)
   o   - change frequency offset within the band (default 1600)
               96..3903 is the range supported by the RFM12B
   g    - set network group (RFM12 only allows 212, 0 = any)
   c      - set collect mode (advanced, normally 0)
  t          - broadcast max-size test packet, request ack
  ..., a - send data packet to node , request ack
  ..., s - send data packet to node , no ack
   q      - set quiet mode (1 = don't report bad packets)
   x      - set reporting format (0: decimal, 1: hex, 2: hex+ascii)
  123 z      - total power down, needs a reset to start up again
Remote control commands:
  ,,, f     - FS20 command (868 MHz)
  ,, k              - KAKU command (433 MHz)
Current configuration:
 O i15 g210 @ 433 MHz q1
OK 53 27 0 0 0 91 0 82 0 100 0 132 0 135 0 (-51) 
 -> ack
OK 52 25 0 75 2 171 0 0 0 0 0 0 0 177 0 (-42) 
 -> ack

Whoooohoo!!

Only problem now is those high node id's due to the acks. this is the same thing fluppie007 saw, I will look at PM's and see if I can determine what his fix was.

Only change I've made is to change mySerial.print((int) rf12_hdr); to mySerial.print((int) (rf12_hdr & 0x1F)); to correct the node numbers output to the Pi.

I think this comment may relate to this line but maybe Glyn can shed some light as I'm not entirely sure.

RFM69CW_RF12_Demo_ATmega328.ino#L641

Interesting. I haven't tried to change the sketch as I'm not quite at the stage of compiling that stuff over ssh on the Pi, but if I manually print those node IDs & 0x1F with a quick perl script then I get the expected values... if someone can explain what that is all about I'd be interested to understand (we're throwing away the first three bits, right? But why?)

If you see http://jeelabs.net/projects/jeelib/wiki/RF12demo#Eeprom-Layout

The first 3 bits don't apply to the node id, that bit I get ok, but I don't get why they are included if they don't apply and how they gets affected by using acks. Maybe someone can enlighten us both.

Ok, so the eeprom layout threw me, right concept, wrong reference. I have since worked out the rf12_hdr refers to the packet's header byte and those 3 bits do include the "request ack" bit. In a normal non-targeted broadcast those 3 bits would be 000 but when an ack is requested they are 001, adding 32 to the node id.

http://jeelabs.org/2011/06/10/rf12-broadcasts-and-acks/

That explains why acks "cause" the issue so I assume the use of the whole byte for the node id is just an an oversight due to the in frequent use of acks, I have therefore submitted a pull request for the fix. https://github.com/openenergymonitor/RFM2Pi/pull/1

Aha, right. That makes complete sense - nice one!

If Glyn has time to update the hex etc. on GitHub I'll confirm that this is resolved. Looking forward to seeing my RSSIs...

Have you tried emonhub yet? the nodes that are out of range will still be processed by emonhub and should be visible in the logs if loglevel set to debug. Just emonCMS will not accept the data from node id's over 32.

The non-ack'd nodes should also be making it through to emoncms with the rssi's.

Out of curiosity, How will the acks work with 2 receivers ?

​

Hex file has been updated! Great work guys, tracking down this bug

https://github.com/openenergymonitor/RFM2Pi/blob/master/firmware/RFM69CW_RF_Demo_ATmega328/RFM69CW_RF12_Demo_ATmega328.cpp.hex

OK, I can confirm that I've got a fully working parallel solution now - RFM69 and the old board both bringing in the whole set of sensors. Great work guys!

The gas people came this afternoon and replaced my old gas meter with a "smart" one. For my purposes this means I can stop strobing an IR LED to detect the reflective 0-digit and just use a reed switch - it's all go in my little emon world today! \o/

(On which note, the emonTH did an admirable job of monitoring the gas - it was about 95% accurate which isn't bad considering the iffy way I was detecting the consumption using optical reflection. The batteries lasted an eternity - many months - even though it is outside, protected by just a slim plastic casing)

Regarding the "double ack", it will just work as you hope, that is, the emonTH broadcasts the data and then waits for confirmation. It doesn't matter which base station ACK hits it first. What you do afterwards with each base station could be significant though (for instance, you could harmlessly submit temperature from both base stations to one emonCMS feed, but you wouldn't want to do that with a gas pulse counter!)

In the general case my reliability of the emonTH nodes themselves is excellent. It's not perfect, for instance my attic / external emonTH crashed out twice this week just after fresh batteries were fitted (but had run seamlessly for months prior to that) but the ack helps I think.

Great to hear! It's interesting that you use the emonTH for pulse counting, it was designed for humidity and temperature monitoring, however I also had in mind pulse counting by breaking out the irq port on the terminal block. Would you be able to write a paragraph about your setup pulse counting on emonTH with a photo of your setup I'll post it up on the oem blog. An update to the emonTH will have a dip switch on board which could be used to switch between pulse counting and temperature and humidity operation. We should smart another thread, so we don't go off topic! 

Big thanks to Paul for help debugging rfm12pi and excellent ongoing emonhub dev 

The "38400" baud auto selection is now merged into the main "development" branch and "57600" option has been removed.

This may cause a little hassle for some early adopters of rf69 if they do not install the new firmware. But since it's easy to define "com_baud = 57600" in the conf it's better removed sooner rather than later.

current version is now rc1.2