RFM12PI receiver goes hard down sometimes

I have got a raspberry pi with emoncms (HD version) and three separate EmonTX transmitters, covering my solar generation, surplus power diverters, and thermal store temperature.

I have been running it for about 3 months now and it has been very successful.

The problem I have is that every few days the RFM12PI wireless receiver goes hard down.

I think that I have isolated the problem to the wireless receiver because I can still use the emoncms web interface when this problem occurs, and I can still input data manually via the json interface.

Also when the problem occurs I lose data from all 3 wireless transmitters simultaneously.

On my rasperry pi, I have tried stopping and restarting the rfm12piphp process but the problem remains.  What is more, the problem survives a software reboot of the pi (sudo reboot).  However, normal reception resumes immediately if I disconnect the pi's power supply and plug it in again.

I have repeated the above scenario several times and I am confident that I am reporting the facts correctly.

I have also tried restarting the rfm12piphp process with the log option.  The result was that the system created a log file but nothing was written to it.

the attached screenshot shows that the json interface can still recieve data (node 1) but nothing is coming from the wireless transmitters (nodes 8, 9 in the screenshot).

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

What version of the RFM12Pi have you got? V1 with attiny or V2 with ATmega328? The V2 is tested to be very reliable. 

It might be the software on the pi. EmonHub is the current activity mentioned and reliable rfm12pi raspberrypi gateway service 

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

Thank you for responding.  I have looked at the board and along one edge is printed "RFM12Pi V2.5.3" - so I assume that that's version 2

The reason that I assumed that it wasn't the software on the pi is that the problem continues after a software reboot of the pi (ssh to the pi command line and issue sudo reboot command) but is immediately fixed if I power off / power on the pi. 

I deduce from this that a software reboot of the pi does not actually "reboot" the RFM12Pi whereas a power cycle obviously does. 

As far as solving the problem is concerned, my idea is that I think I could write a monitoring program that would detect when the RFM12Pi had stopped receiving, then I would like that program to send a command to the RFM12Pi to reset itself - I have no idea if sending such a command is possible or how I would do it.

I do know that problems are not always where they seem to be and I'm open to any suggestions.  The problem occurs randomly.  Once the problem occurs it never goes away of its own accord, but once I get the RFM12Pi going again it usually works fine for several days.

I've already written some software that monitors the voltage level of my battery bank and feeds the data into emoncms via the json interface.  I'm planning to develop this to a point where the pi becomes the command centre for the whole of my off-grid system.  That's why I need to find a way of addressing the reliability of the data collection process.

I attach a screenshot showing a graph of solar generation and battery bank voltage.  You can see that the solar power generation data (which comes via the RFM12Pi) suddenly stops at 12:47 (the brown area on the graph) whereas the battery voltage data (which is automatically input using the emoncms json interface, not via the RFM12Pi) carries on (the blue line on the graph).  NB this is not zero values, it is an absence of data.

Thanks in advance for any help anyone can give.

 

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

Apologies for the long delay, I can only investigate this problem when it occurs, which is totally randomly (as far as I can tell).  After several weeks of things working perfectly, the problem has just occurred again, and I have made a little bit more progress in investigating it.  Here is the scenario:

1. I notice that emoncms has stopped receiving data from all 3 emontx transmitters simultaneously, while still getting data that a separate process is injecting via the json interface.

2. I connect to the raspberry pi with ssh and reboot it

$ ssh pi@192.168.10.106
pi@raspberrypi $ sudo reboot

3. The problem has not been fixed.  Still no data from any of the emontx while the json service is still fine.

4. I decided to use minicom to try and investigate the rfm12pi interface.

$ ssh pi@192.168.10.106

pi@raspberrypi $ sudo service rfm12piphp stop
pi@raspberrypi $ minicom -b9600 -D/dev/ttyAMA0

5. Minicom shows just the welcome screen, no sign of any data [Screenshot AfterReboot.png]

6. I physically unplug the pi from its power supply and plug it in again.

7. As soon as the pi reboots, the problem completely fixed. 

8. I use minicom again to look at the rfm12piphp interface again (repeat of step 4)

9. This time minicom shows a steady stream of data from the rfm12pi [Screenshot AfterPowerCycle.png]

To my mind, this shows that the problem is not on the raspberry pi, it must be on the rfm12pi expansion board itself. 

(This makes it sound like I knew what I was doing.  In fact it was all painfully slow, ie all bound up with learning Linux, discovering minicom and how to use it etc, etc)

I infer that rebooting the pi by software command does not reboot the rfm12pi, whereas obviously power cycling the pi does reboot the rfm12pi. 

I am interested in anything that can help me get closer to a solution - for example (a) is there something I could do to diagnose the problem more precisely (b) is there some way of sending a command to the rfm12pi to reset itself, or (c) do I need a firmware upgrade to the rfm12pi or something like that?

I've also noticed an older thread with what looks like the same problem
http://openenergymonitor.org/emon/node/2947

, but I'm not able to understand enough of that to know if it's relevant.

 

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Your logic seems sound and the issue does indeed appear to be with the RFM2Pi board which, as rule, are extremely reliable, I would recommend updating the firmware on the RFM2Pi as there were some issues with random lock-ups ( for example see http://openenergymonitor.org/emon/node/3584 ) which have been alleviated with firmware updates.

A guide to updating the firmware directly using the Pi is found in the wiki http://wiki.openenergymonitor.org/index.php?title=RFM12Pi_V2

Paul

 

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

Thank you very much for that.  I hadn't seen thread 3584 before, despite searching - it describes my problem exactly.

Looking at https://github.com/mharizanov/RFM2Pi/tree/master/firmware/RF12_Demo_atme... I see that there is a version of RF12_Demo_atmega328.cpp.hex that is dated 10 April 2014.  I purchase my rfm12pi on 3 July 2013 so obviously it will have an older version.

I will be big and brave and attempt the firmware upgrade.  I plan to do this from the Raspberry pi; the instructions at http://wiki.openenergymonitor.org/index.php?title=RFM12Pi_V2#Upgrading_R... seem clear. 

However I do have one concern.  I see at the same github location that there is also an update for Optiboot328_8mhz_RF12_Demo.hex  Would I need to flash that one as well?  If so, can I also do that from the raspberry pi?  The instructions at http://wiki.openenergymonitor.org/index.php?title=RFM12Pi_V2#Flashing_th... don't really say either way.

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

I think you just need the firmware updated that's all I recall doing to mine.

Hopefully someone will correct me if I'm wrong but I believe you would need an ISP programmer to change the bootloader and it probably hasn't been modified itself, the updated version is probably just the combined original bootloader and later firmware.

Paul

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

No, as Paul correctly said you don't need to flash the Optiboot. That's the serial bootloder wich is flashed via ISP in the factory. The serial bootloder enables subsequent firmware updates to be done over serial. 

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

I have had a go but I'm not sure if it worked.  

I followed the instructions at http://wiki.openenergymonitor.org/index.php?title=RFM12Pi_V2#Upgrading_R...

I think that the git pull worked because I got files with new datestamps.

However, when I ran the avrdude command, I got lots of error messages saying

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x78

which are leading me to think that the firmware upgrade did not work.

Here is the entire output:

pi@raspberrypi ~/RFM2Pi/firmware/RF12_Demo_atmega328 $ sudo avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 38400 -U flash:w:RF12_Demo_atmega328.cpp.hex 

avrdude-original: Version 5.11.1, compiled on May 23 2012 at 11:08:25
                  Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
                  Copyright (c) 2007-2009 Joerg Wunsch

                  System wide configuration file is "/etc/avrdude.conf"
                  User configuration file is "/root/.avrduderc"
                  User configuration file does not exist or is not a regular file, skipping

                  Using Port                    : /dev/ttyAMA0
                  Using Programmer              : arduino
                  Overriding Baud Rate          : 38400
done with autoreset
                  AVR Part                      : ATMEGA328P
                  Chip Erase delay              : 9000 us
                  PAGEL                         : PD7
                  BS2                           : PC2
                  RESET disposition             : dedicated
                  RETRY pulse                   : SCK
                  serial program mode           : yes
                  parallel program mode         : yes
                  Timeout                       : 200
                  StabDelay                     : 100
                  CmdexeDelay                   : 25
                  SyncLoops                     : 32
                  ByteDelay                     : 0
                  PollIndex                     : 3
                  PollValue                     : 0x53
                  Memory Detail                 :

                                           Block Poll               Page                       Polled
                    Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
                    ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
                    eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
                    flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
                    lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
                    hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
                    efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
                    lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
                    calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
                    signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

                  Programmer Type : Arduino
                  Description     : Arduino

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x78

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xfe

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x80
                  Hardware Version: -1225059712
                  Firmware Version: 344576.-1227354920

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x78

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xf8

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xf8

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x78
                  Vtarget         : 0.0 V
                  Varef           : 34456.0 V
                  Oscillator      : 10.698 Hz
                  SCK period      : 3328572456.6 us

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x1e

avrdude-original: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x78
avrdude-original: stk500_initialize(): (a) protocol error, expect=0x14, resp=0xe0
avrdude-original: initialization failed, rc=-1
                  Double check connections and try again, or use -F to override
                  this check.

avrdude-original: stk500_disable(): protocol error, expect=0x14, resp=0x78

I'm pretty sure that rfm12piphp was stopped OK, because there is a gap in my data.

The good news is that I was able to restart rfm12piphp afterwards so at any rate I haven't damaged it, but I have no idea if the firmware upgrade actually worked. 

Thanks once again for all your help.

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

No, that doesn't look right !

It looks good up to "Description     : Arduino" but it doesn't complete reading the device, so will not of worked.

To reduce the chance of damaging anything just try reading the device only and when that has been successful try flashing again. use the same command without " -U flash:w:RF12_Demo_atmega328.cpp.hex " this will just read the device and should give you an output that starts as above but instead of the errors you should get

                  Programmer Type : Arduino
                  Description     : Arduino
                  Hardware Version: 3
                  Firmware Version: 4.4
                  Vtarget         : 0.3 V
                  Varef           : 0.3 V
                  Oscillator      : 28.800 kHz
                  SCK period      : 3.3 us

avrdude-original: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude-original: Device signature = 0x1e950f
avrdude-original: safemode: lfuse reads as 0
avrdude-original: safemode: hfuse reads as 0
avrdude-original: safemode: efuse reads as 0

avrdude-original: safemode: lfuse reads as 0
avrdude-original: safemode: hfuse reads as 0
avrdude-original: safemode: efuse reads as 0
avrdude-original: safemode: Fuses OK

 

I found when I did mine it failed a couple of times due to low (or unstable) voltage, when I used a larger power supply it sailed through first time, It may be something completely different in your case but it may be worth trying to improve the power supply, if you have non-essential usb devices unplug them (even wifi dongle if you can) or even use a different SD card so you can unplug the HDD.

Paul

 

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

Success!  I proceeded as follows:

  • stopped the rfm12piphp service
  • changed to the directory containing the new firmware
  • issued the avrdude command without the " -U flash:w:RF12_Demo_atmega328.cpp.hex " at the end and saw that it output all the information suggested in previous post, down to and including "Fuses OK"
  • issued the avrdude command again, with the " -U flash:w:RF12_Demo_atmega328.cpp.hex " at the end and saw that it proceeded through "reading", "writing" and "verifying" stages without any error messages.
  • restarted rfm12piphp service and saw that emoncms was receiving data again

My plan B had been to boot the pi from another card as an SD-only device (as distinct from the HDD version which I am running) and try it that way, but obviously that wasn't necessary.

Only time will tell whether the new firmware solves the original problem of the rfm12pi board locking up randomly, but I am of course optimistic.

Also, I was enlightened by Glyn's comment:

the Optiboot ... serial bootloder enables subsequent firmware updates to be done over serial.

That clarified something that I had been struggling to understand.

 

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

I just had it happen for the first time in many months on the latest (as far as I am aware) firmware, looks like a watchdog on the rfm12pi might be a good idea

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

Happened again so I've swapped radios

Andrew Wedmore's picture

Re: RFM12PI receiver goes hard down sometimes

As the original raiser of this thread, I would just like to record that the solution (upgrade the rfm12pi receiver firmware to the latest version) has worked for me.  That is to say, it has now been working solidly for a whole month, which was not happening before.

I also have "monit" installed on the rasberry pi, in case the receiver software ever crashes out, but this may just be belt and braces - receiver crashing out was not the problem here.

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

It did for me for many months but recently occurred again, I too used to use monit maybe it's time to install again but first I'll see how the radio swap goes. I'm seriously considering a wired solution for more critical systems such as CH control, I've already had to turn off acks for most nodes as they appeared to be blocking each other. Maybe the radio spectrum is especially busy in these parts.

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

Happened again yesterday but at least now I can su poweroff then powercycle the RaspberryPi from my mobile thanks to Juice (ssh client for Android) and a web app I built myself.

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

Hi Brain,

Sorry to hear of the issue your having with the RFM12Pi. I am keen to try and track this issue down. Please could you confirm what version of the RFM12Pi you have and what firmware your running. Have you updated to the latest version?  

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

It's a V2 with the latest firmware plus a small amendment to increase the buffer size as here https://github.com/mharizanov/RFM2Pi/blob/master/firmware/RF12_Demo_atmega328/RF12_Demo_atmega328.ino to overcome an issue with packetgen limiting the size of transmissions . I'm not clear whether this has actually been implemented in your latest developement versions.

I have the same firmware on 2 V2s both exhibit the same occasional lock ups. I was thinking that implementing a watchdog in the firmware to reset on lockup would be the way to go.

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

Looks like the embedded link has queued my response for moderation so I'll repeat my answer. It's a V2 with the latest firmware I'm aware of. The link was to the firmware version on GitHub that I'm using.

Robert Wall's picture

Re: RFM12PI receiver goes hard down sometimes

Patience Bra1n, patience! Anyway, I've promoted you to 'OEM Power User' so you should be OK now.

Bra1n's picture

Re: RFM12PI receiver goes hard down sometimes

Who Hoo ... Promotion :)

dBC's picture

Re: RFM12PI receiver goes hard down sometimes

I was thinking that implementing a watchdog in the firmware to reset on lockup would be the way to go

You can also use the wdog to help you find out where it's hanging.  The AVR wdog has a nifty "interrupt the first time, and reset the second time" feature.  During the ISR for the first firing, you can store away system state (and even the PC off the stack) into non-volatile storage such as EEPROM.  The second firing will then reset you, and you just need to add a way to display the system state previously stored away.

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Not sure if I should be hijacking this thread but I have a similar problem. I've had my system running flawlessly for about 18 months, diverting away with Martin's sketch and transmitting temperatures from my tank.

However occasionally l have had power failures, in one case the power supply to the pi failed, other times it's been the mains supply and sometimes I've shut down the pi with sudo shot down.

Not every time but often enough when I get power back on the pi the radio won't restart. Doing a bit of checking, what happens is the green led on the rfm12pi flashes once but then doesn't go into it's running state of flashing every few seconds.

I've been solving this by just restarting again which normally gets things going but just today this seems to have stopped working. Any pointers as to what to try next gratefully received. I have done a search but this post seems to be the nearest to the issues I'm having although I do need to add that I have never had comms fail once the rfm12pi is up and running. My issue is only on startup.

 

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Isn't that just brilliant! Sod's law at work here. 

It's working again. I had rebooted before writing the post above. Sat down to write the post, then went back to where the pi is installed and hey presto the little led is flashing away fine and I can see the data in my dashboards...

Guess I'll just have to more patient until the number of retries gets excessive.

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

Not sure whether to post here or start a new thread.

I am having identical problems but with a RFM69Pi. I can get data for days or weeks, but then it will stop receiving and the pi will need a power cycle to get it working again.

Is there a command to reset the radio which I could try?

 

Ian Eagland's picture

Re: RFM12PI receiver goes hard down sometimes

Hi

Just to add that I have a similar problem as well. I am waiting for it to fail again so I can check if the RFM led is still working. (It's so small I did not even know it was there until Paul (emonhub) told me. Typical, since I learned about the LED it has not failed!

Regards

Ian  

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Hi Steve - Can you confirm if the led on the rfm69pi is flashing intermittently to signify it is receiving but not passing the data or just dead?

Hi Ian - That's been quite a while now, fingers crossed a ?

There is no built in command to reset the rfm2pi and I'm not sure it would work if the AVR is getting stuck anyway. It would be good to see a watchdog in the sketch too but I am working on a watchdog in emonhub that if the rfm device doesn't pass anything for a while emonhub will attempt a communication and if no response is received it will then physically hard reset the AVR using the reset line (used during programming) by pulsing the gpio pin.

This will be in the next emonhub (hopefully) but in the meantime I have attached a python script download and install using (rpi-rw 1st if using read-only)

 

sudo wget -O /usr/bin/reset_rfm2pi http://openenergymonitor.org/emon/sites/default/files/reset_rfm2pi.txt

sudo chmod +x /usr/bin/reset_rfm2pi

Then you can reset the rfm2pi any time using the command sudo reset_rfm2pi. If you find this works to reset the rfm2pi without restarting emonhub or the Pi then maybe adding the watchdog mentioned above would be the way to go. 

Paul

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

It dropped out again today and I can confirm there is no flashing light.

I will download and install the script and post the results when I get a chance to test it.

 

Edit:Woke this morning to find no inputs for 6 hours. Tried the script but it didn't bring it back to life

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Actually you will have to stop the emonhub service before using the script to make the serial port available to reset the rfm2pi and then restart it again afterwards, if you are using the emonSD image you can use emonhub -x to stop and emonhub -s to start again (otherwise sudo service emonhub stop and sudo service emonhub start). 

If you can confirm the led flashes periodically and that the reset script works whilst all is ok, then when it faults you can compare responses, eg you should get a longer led flash immediately after resetting the AVR as that is part of the setup. If that happens when testing but isn't happening when at fault either the serial port or the AVR is locked for some reason. If the longer flash does occur but the reception (periodic flashes) don't resume then it would point to the AVR resetting as expected but the rfm module is not working.

Does the emonhub.log shed any light? unlikely in this instance but worth checking.

Paul

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

Total contents of emonhub.log any time I have checked it is:

2015-05-15 08:01:55,435 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-05-15 08:01:55,438 INFO Opening hub...

I'm not seeing the longer flash (or any change in the flashing) while testing but I can see that stopping and starting the service is working

2015-05-15 08:01:55,435 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-05-15 08:01:55,438 INFO Opening hub...
2015-05-17 06:11:40,221 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-05-17 06:11:40,224 INFO Opening hub...

I'll update after trying on a locked system, if and when it happens again.

 

 

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

If you change the loglevel to DEBUG in emonhub.conf you will see far more info.

Do you see the single longer flash when restarting the rfm2pi? The script is intended to "reset" the rfm2pi's AVR in a way that should trigger the initial "setup()" routine as if it had just been powered up.

It's easy to miss the extended led flash at "setup()" as it only happens once and although I say "extended" as it's longer than the usual flash, it is still quite quick. but it should happen immediately after powering up (or using the reset_script), but restarting emonHub or rebooting the Pi will not "reset" the rfm2pi so no extended flash will be seen.

For the absence of a flash when using the script to have any meaning you will need to confirm that it normally does occur. 

Which model Pi are you using is it a B, B+ or 2B ?

Paul

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

I have noticed the longer flash when I power up the pi. I am not getting that when I run the reset script.

I have a model B raspberry pi.

I have set the log to debug level so that I can get more info when it stops

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

It looks like the script isn't working, I had only previously tested this on a Pi 2B, so to rule out any gpio numbering issues I have just tried it on a Pi B with a rfm69 RFM2Pi and it worked as expected. I used just these 3 lines

 

sudo wget -O /usr/bin/reset_rfm2pi http://openenergymonitor.org/emon/sites/default/files/reset_rfm2pi.txt

sudo chmod +x /usr/bin/reset_rfm2pi

​sudo reset_rfm2pi

 

Do you get any error messages when trying to use the script ?

what response do you get to " ls -la  /usr/bin/reset_rfm2pi​ " ?

Are you using Raspbian OS is it up to date?

Does " sudo apt-get install python-rpi.gpio " confirm the gpio module is installed? (latest version?)

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

I get no errors using the script.

I installed the script by copying and pasting the lines from the post (from the 14th) above straight into a command shell. The OS is installed from the image dated 13/3/15 & updated today. GPIO module is installed and newest version.

pi@emonbase ~ $ ls -la /usr/bin/reset_rfm2pi
-rwxr-xr-x 1 root root 406 May 14 11:28 /usr/bin/reset_rfm2pi

​I compared the script in that file to what was posted above and it is exactly the same. To be sure I deleted the file and re-downloaded it using the above steps in case it was corrupted somehow but there was no change.

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Well that's bizarre!

This should be quite straight forward, it's nothing new or complex the script just borrows from the "avrdude-rpi" script that's already tried and tested. You could try adding a print line after the "GPIO.cleanup()" line  to at least confirm the script has been run. 

        ...
        GPIO.cleanup()
        print ('A reset of the RFM2Pi has been attempted')
except ........

I don't have a RFM69Pi to test but cannot think of any reason it would be different off-hand. I have tested on Pi B, Pi 2B, RFM2Pi(rfm69) and RFM2Pi(rfm12) on 2 different firmwares. I have tried it with emonHub running and stopped, I have tried it read-only too not sure what else to suggest.

You could also try increasing the reset duration from "sleep(0.12)" to "sleep(1)" as it is not timing critical when not being used for with avrdude for actual programming.

Paul

 

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

I put the print statement in and it comes up on the screen while running the script, so I know the script is running through. I also changed the time delay but still no flash.

I appreciate your help in this.

I can try this on a Pi2B, but not until next weekend. Maybe someone else will see this before then and give it a try.

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Hi there,

I have the exact same problem. But I am using a rfm69Pi v3.1, a Pi2 with Raspian and the latest firmware for the Pi2 and rfm69pi. 

I just found the reset script mentioned earlier. If my rfm69pi board hangs again, I will try the script... Will the script also work with the Pi2? Just asking because the Pi2 has more GPIOs...

I noticed somthing today. When I power down the Pi and disconnect the board from the GPIOs, the board has it's old config (group id, node id etc). But when it "hangs" and I power down the Pi, it's resetted to the defaults... Other node and group ID and the freq is set to 433 Mhz, but I'm using a 868 mhz version...

Regards,
toberkel

[A duplicate of this post has been deleted. Please observe the notice on the right "NEW USERS, PLEASE READ - Posts held for moderation" - Moderator (RW)]

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

How do you manage to powercycle the Pi via ssh when it's powered off? 

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

I shut it down via ssh ( sudo poweroff ), and then go and physically turn it off and on again

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Hi toberkel, would it be possible for you to try the script prior to "hanging"? This would not only give you a insight as to what normally happens to gauge what actually happens when in a fault condition, but may go a long way towards helping fathom out why the script won't work for SteveM, plus if it's not working we may be able to correct that before you need it.

You cannot "powercycle" either the Pi or the AVR via SSH as there is no control of the power rails, since "powercycling" the Pi is only needed to interrupt the power to the GPIO to cause the AVR to reset this script is intended to negate the need for "powercycling" and therefore permit remote resets.

The "forgetting the frequency" is an interesting find as that suggests a problem related to the eeprom if it's the cause but if it's an effect maybe it's the frequency that gets changed and the RFM2Pi is not "hanging" but just not seeing any data on the alternative frequency and assumed to be hanging? although the latter I would expect to be resolved by restarting emonhub as correct frequency would be set if the RFM2Pi settings did not match the intended frequency

It would be useful to know if there is any abnormal log entries in emonhub.log, you can set loglevel to DEBUG and be able to locate the last good RFM2Pi packet quite easily if this reoccurs

Paul

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

Hi guys,

We have noticed the same problem with the emonPi, which shares much of the code with the RFM69Pi. It seems to be a JeeLib RFM69CW issue. It's strange that I have never had my RFM69Pi do the same in many many months of opperation. 

It seems the issue is with the RF driver rather than the ATmega328 or emonHub. It seems the RF stopps receiving data from my emonTx and emonTH but the ATmega328 is still doing local sampling on the emonPi and succesfully serial printing on node 5. The RF comes back to life when I change the group to something else then back to group '1' which is what I use in my house. 

Here is a minicom serial output showing the ATmega328 doing local sampling on group 5 then me brining the RF back to life by changing the group:

I tried today recompiling pulling in latest changes to JeeLib: https://github.com/jcw/jeelib/commits/master

There are various branches to JeeLib, it seems to be a few bugs to do with RFM69CW. Could this be the same issue we are experiencing: ​http://jeelabs.net/boards/6/topics/6374

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Glyn - When it fails again could you try changing a different setting eg change nodeid and back again? or better still when it next stops without letting emonhub alter anything on reboot (disable RFM2Pi interfacer before reboot) "powercycle" and check to see if it has reverted to group 210.

I wonder if the values retained in the eeprom (or absence of such) are causing an issue when accessed by the code and by changing a setting the eeprom is being overwritten, in toberkel's case, "powercycling" is reloading the default of 433Mhz, the same scenario for SteveM would be successful as he is using all defaults.

Paul​

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Hi Paul,

thanks for your detailed answer... I will execute the script after I returned from work.

When the board "hangs" and I connect via minicom to it, the frequency is till set to 868 mhz and the groud id is also 235 (which I am using).

I will come back to you with the script results...

Regards,
toberkel

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

I just executed the script without any error or output... My values still get updated, so I think everything is fine... Hopefully it will work when the Pi "hangs" again too

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

@paul All EEPROM stuff has been stripped out for the emonPi so this is not the issue

https://github.com/openenergymonitor/emonpi/tree/master/Atmega328/emonPi_RFM69CW_RF12Demo_DiscreteSampling

​I had the same issue again this morning, so using latest JeeLib has not solved anything. I tried restarting emonHub but this did nothing. As before changing the network group then back again brought it back to life. It seems it's requiring an RF initialize to start working again.  

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Hi,

this morning I had a freeze again. The script wasn't working...

I also tried to change the group id and afterwards change it back, but I failed... After typing "210g" the board replied with "config save failed" and the group id still was 235... So I had to powercycle the pi. After login I had to change all the settings from the rfm69pi board... Node id, group id and frequency was set to defaults :-/

Regards,
​toberkel

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

"this morning I had a freeze again. The script wasn't working..." Unfortunatly this doesn't have much meaning until you confirm whether the script works normally. Do you see a longer LED pulse immediately after using the script?

Looking at the sketch "config save failed" can only be a result of a failed eeprom write, since Glyn is not using the eeprom one can only assume the data being written to the eeprom is the issue as you are clearly experiencing eeprom issues if the settings are "not retained" in which I include erroneous overwriting.

Glyn - I notice (in the rfm69 sketch) the configSilent() returns the node id and a truth test is done on that, is it therefore possible if the node id was lost, null or zero a "successful" eeprom write (with bad data) would be considered a fail due to the 0 node id ? if something along those lines happened it might not be so obvious to spot running the emonpi sketch.

Paul

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Hi... I just had a freeze again... I found a solution without rebooting or power cycling the pi... If I start minicom and use the "123 z" option, I can reset the board with the script... Afterwards I have to configure it again with the group ID and frequency I'm using... But hey, no reboot needed anymore... Maybe I can build the 123z command into the script or something so I can restart it completely with the script or automatically... Anyone an idea how to send the 123z via the script?

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Is it possible to allow me to post without a review? Thats a bit anoying... ;)

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

Hi guys,

I have had success fixing a similar issue I had been experiencing with the emonPi where the RFM69CW would hang. This now seems to have been fixed by latest commits to JeeLib library. I have re-compiled the RFM69Pi firmware using latest JeeLib. I have bumped the RF12_demo version number from 12 to 13 to enable us to tell the difference. If you're experiencing hanging issues please could you try the new firmware. 

https://github.com/openenergymonitor/RFM2Pi/blob/master/firmware/RFM69CW_RF_Demo_ATmega328/V13_RFM69CW_RF12_Demo_ATmega328.cpp.hex

You can update by running the update script pointing to V13_RFM69CW_RF12_Demo_ATmega328.cpp.hex

https://github.com/openenergymonitor/RFM2Pi/blob/master/update-RFM69

Please let me know the results. 

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Hey Glyn,

thanks, I will try this when I'm home! :)

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

You could be on to something there Glyn, I never use the hex files, I always compile locally which maybe why I don't see the same issues since I will of unintentionally always had the latest JeeLib.

So is this hex file the same rfm69 "v12" sketch just recompiled with latest JeeLib and no other changes to the sketch?

toberkel - That command shouldn't "restart" it should put it into a sleep state requiring a powercycle to exit. 

https://github.com/openenergymonitor/RFM2Pi/blob/master/firmware/RFM69CW_RF_Demo_ATmega328/RFM69CW_RF12_Demo_ATmega328/RFM69CW_RF12_Demo_ATmega328.ino#L314

​or a proper "reset" signal which is what I was trying to do with the script. The Pi has never needed a reboot for this issue it's just away of powercycling the RFM2Pi without disconnecting it.

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

@pb66

Q. "So is this hex file the same rfm69 "v12" sketch just recompiled with latest JeeLib and no other changes to the sketch?"

A. Yes, exactly 

Ian Eagland's picture

Re: RFM12PI receiver goes hard down sometimes

Hi

Well its been a long wait but again my test has stopped posting. I can confirm that the led on the rfm had stopped flashing. Unfortunately I cannot investigate restarting because of brain fade. I shut the pi down via ssh (intending to shut down my main server!)

I have restarted the pi and its now posting away again. It  worked OK for about 12 weeks before it failed.

Regards

Ina

 

 

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

I have updated the firmware as indicated above. Note that the reset_rfm12pi script still doesn't cause the light to flash after this update, but I assume the reason for the update was for stability rather than control.

I'll post here if I get another hang, this usually has been between 2 - 28 days.

 

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

@pb66

Yes, but with my way I don't need to powercycle the pi... I send the 123z command via minicom and afterwards I execute the mentioned script. Then, when I start minicom again, the board is responding and after I configured the propper settings (frequency, group id) it shows the packets again. So, maybe I can automate this process...

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

Thanks a lot for testing so promptly. 

Ian Eagland's picture

Re: RFM12PI receiver goes hard down sometimes

Hi

I have tried to update firmware as above but get avrdude: command not found.

Is this normally installed in the emonhub or do I need to upgrade software?

Also I have RFM12pi v2.6 pcb with an RFM69 sub pcb. Is this new V13 firmware correct for this set up.

Regards

Ian 

 

 

 

glyn.hudson's picture

Re: RFM12PI receiver goes hard down sometimes

You should be able to install it and auto reset by git cloning running the update script here: https://github.com/openenergymonitor/avrdude-rpi

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

Could someone please remove the flag "post needs aproval" for my account? :D

toberkel's picture

Re: RFM12PI receiver goes hard down sometimes

One of my posts is missing... :-/

SteveM's picture

Re: RFM12PI receiver goes hard down sometimes

One week later... Hung again :(

I'll swap out the Pi with a a Pi 2 over the weekend and see if there is any improvement.

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

@toberkel - sorry I've missed your posts somehow. If the reset script works to restart the rfm2pi then it is working and can therefore operate anytime, I think the LED activity must differ on some firmwares then. So the rfm2pi board is being reset you are just not aware as we have been expecting a longer LED flash. No need to use 123z or write an additional script.

The fact you need to re input the settings normally held in the eeprom is the key here although we can tell if it's the cause or the effect but either way they should be present and correct.

@SteveM - I would be very surprised if it made any difference, It seems the LED flash may not be a definite sign that the rfm2pi is being reset. 

Paul

 

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

I think I may have now experienced this. I noticed none of my RFM network inputs had updated on emoncms since 7am yesterday, checked the LED on the emonTx in the garage and that was still flashing as expected, but the rfm2pi (rfm69@433mhz running standard oem rfm69 firmware, compiled locally with latest libs not from hex) was not flashing, when I ssh'd in emonhub was still running just no signs of any rfm traffic.

Since I had the reset script still installed from the tests above I tried it (without stopping emonhub) and everything started back up once the rfm2pi's "option instructions text" had passed.

I cannot confirm if there were any settings lost on the rfm2pi because this one runs the default 15,210,433 settings, but it does mean if emonhub was made to respond to period of no receipts it could initiate the reset (and also resend the settings if needed).

This would side step but not fix the root issue.

It also confirms our suspicions about it being the rfm2pi (either fw,hw or influenced) rather than the Pi or emonhub as I did nothing but momentarily ground the avr's reset line via the gpio and script.

Paul

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

I have had the same issue over the last few weeks. This is on a new hardware setup. Pi2, rfm69pi and a fresh install of V9 rc2 running on HDD as per Paul's installation manual.

Paul in the other thread you wrote

Bramco - The drop-outs maybe the rfm2pi locking out and rebooting the Pi also restarts the rfm2pi, see the latter part of the RFM12PI receiver goes hard down sometimes thread for more info. the test would be to ssh in and reset the rfm2pi by pulsing the reset pin (pin7 of the Pi's gpio).

The issues here may apply but it would be unusual that no packets get through, these issues normally arise from clashes and/or bit syncing issues, the bit syncing is unlikely unless you have all the temp sensors disconnected (zero values) and although the extra sensitivity of the rfm69 may introduce some clashing some data would still get through.

Paul

The next time it happens I will try the reset, I'm assuming this is what's in your script mentioned above?

And just to be clear, this isn't about clashes or bit syncing issues, it's that the rfm69 has stopped. No flashing light. As I mentioned in the other thread I think, I'm pretty sure I haven't got any Pi power issues, it's on a 2W power supply with a new USB hub on a separate power supply for the HDD.

Of all the folks that have contributed to this thread with the same issue. Have you sorted this or are you just putting up with it?

Simon

EDIT

PS to the above.

@Glynn   -  wouldn't it be sensible to make the flashes on the rfmxxpi boards more discernable. So extend the flash for normal behaviour/rf packet received to say 1 second? and make the flash at setup something different, e.g. three quick flashes. From a lot of contributions it seems that lots of people don't notice the flashes until it is pointed out to them.

Obviously haven't looked at the code to see how easy this would be but I think it might help.

Simon 

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Bump!!

Hi once again.  :-)

Anyone from the thread above have a definitive solution to the rfm69pi suddenly stopping? Or do we just have to put up with occasional hard failures like this?

@ Glyn - any thoughts on improving the heartbeat and startup signals on the led? I.e. making them more easily discernable.

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

@Bramco - the problem in this thread was establishing a pattern and narrowing the diagnosis. some users have not been able to confirm the reset script even works during normal functioning and It is pretty difficult to establish if something reacts differently when faulty when the expected behavior is not confirmed.

Plus there are several different devices, several different firmwares etc. It would be good to get to the bottom of this but the lack of case data prevents any progress. have you tried the lateset firmware hex? (v13 - same firmware later JeeLib)

Paul

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Thanks Paul for the summary. I was hoping some of the folks that have been on this thread might have found a solution in the interim but not thought to report it here. 

It would be great to rename this thread with a SOLVED prefix  :-)

Anyway, I'll check versions later and report back.

I still think that given quite a few users have seen this and reported it (and there's probably more out there that just do a reboot and don't report it) that it would be a good idea to get to the bottom of it or at least incorporate a solution to this in emonhub or somewhere. As you know I'm not Linux fluent, so haven't a clue really where to start to think about a solution.

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Hi Paul,

Following the instructions here -> http://wiki.openenergymonitor.org/index.php/RFM69Pi_V3

I get the following from avrdude which seems to be complaining about the user configuration file, i.e. the firmware file.   On re-reading it seems to be the config file for avrdude.

Also in your note above you mention V13. Is the base file now V13? or should I be using this file not the base file? See the directory listing below.

Simon

pi@Homepi ~/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328 $ ls
Blink.cpp.hex
Optiboot328_8mhz_RFM69CW_RF12_Demo_ATmega328.cpp.hex
RFM12_Demo_ATmega328.cpp.hex
RFM12_Demo_ATmega328.cpp.hex.readme
RFM69CW_RF12_Demo_ATmega328
RFM69CW_RF12_Demo_ATmega328.cpp.hex
V13_RFM69CW_RF12_Demo_ATmega328.cpp.hex
pi@Homepi ~/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328 $ avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 38400 -U flash:w:RFM69CW_RF12_Demo_ATmega328.cpp.hex

avrdude-original: Version 6.1, compiled on Jul  7 2015 at 13:18:47
                  Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
                  Copyright (c) 2007-2014 Joerg Wunsch

                  System wide configuration file is "/etc/avrdude.conf"
                  User configuration file is "/root/.avrduderc"
                  User configuration file does not exist or is not a regular file, skipping

                  Using Port                    : /dev/ttyAMA0
                  Using Programmer              : arduino
                  Overriding Baud Rate          : 38400
avrdude-original: Using autoreset DTR on GPIO Pin 7

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

Yes, you can ignore the "user configuration file" messages they are just confirming there is no "per user" conf file found, the system wide conf file is there.

The file you have chosen should work, as far as I recall that is the same sketch as the "V13_" version, only JeeLib was updated between the files being compiled, unless something has changed i think "V13_" should be the one to use, you should aware the V13 prefix was just to distinguish it from the existing compiled file V13 is still v12, perhaps V12a would have been more appropriate. 

The output you show seems correct just not completed and/or any error message. Did it exit to the command prompt? Did it take long? Was there any led activity? is the Pi in read/write mode?

Did you try the reset script to confirm the reset pin/gpio connection? You can also test avrdude and the connection without attempting an upload by leaving the -U option off the command line you are using, it should read the AVR fuses and provide a reassuring print out confirming comms.

Paul

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Hi Paul,

avrdude didn't give any error messages, it simply exited to the command prompt and it was pretty immediate, no led activity and the pi is I'm pretty sure in r/w mode. It's a Pi2 with rfm69 set up following your installation instructions with a USB HDD.

Just tried the reset script which just returns to the command prompt with no output. I've tried it with emonhub stopped as well as running - exactly the same behaviour.

How do I confirm the reset pin/gpio connection?

Running avrdude by leaving off the -U option gives me exactly the same output as running it with the -U option. I'm a little concerned that the output indicates it is 'Using autoreset DTR on GPIO Pin 7'. From reading the other documentation this refers to pin 4.

Sorry to be so much trouble here but it would be good to get to the bottom of this if there is a problem with the instructions.

PS It might be an idea to add the info about your reset script to the documentation on the rf2pi in the wiki.

Simon

Bramco's picture

Re: RFM12PI receiver goes hard down sometimes

Just an update to link to a solution to the avrdude not doing anything.

See this post by http://openenergymonitor.org/emon/node/11747 by Ingmar Verheij.  Basically you need to use an old version of avrdude.

Simon

pb66's picture

Re: RFM12PI receiver goes hard down sometimes

if you are watching the output from the rfm2pi, either in a serial console or by tailing emonhub.log and use the reset script you should see the setup text to confirm it was reset, I think the reset script would be useful, I will be including it as part of a watchdog function in the next emonhub version, so that if there is no traffic from a rfm2pi for a given time it will be reset automatically, but this wouldn't "fix" rfm2pi problems, just make them tolerable.

The timing ofthe reset in avrdude 6.1 seems to be different and extending it seems to help but I have not found a single timing that works everytime yet, reverting to the previous version just falls back to the tried and tested timing for the reset.

pin 7 and pin 4 are actually different naming conventions for the same physical pin.

Lets us know how you get on.

Paul

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.