EmonPi losing connection to EmonTX

I have a new system consisting of an EmonPi and EmonTx purchased ready built from the store, with EmonPi components updated as per the guides.  The EmonPi is in the garage monitoring solar generation and receiving grid import/export data from the EmonTx located in the house next to the utility meter. Over the ten days since the system has been running, Emonhub has stopped receiving data from the EmonTx many times.  The lack of connectivity occurs irregularly, from 10 minutes to over 48 hours between drop-outs and the EmonTx appears to be operating normally during when dropouts occur.  The only way to restore connectivity is via a hardshut down/restart of the EmonPI.  Soft reboots from ssh shell hang the EmonPi on reboot.  Although the signal strength of the EmonTX signal at the EmonPi is not wonderful (RSSI -74), the immediate reconnection on reboot and the lack of dropped packets logged while "working normally" suggests to me that the issue is hardware/firmware related on the EmonPi.

I'd really appreciate any pointers as to how to troubleshoot and resolve this, being as I am a complete beginner to these devices (but willing to learn!)

Thanks in advance

Andy

emjay's picture

Re: EmonPi losing connection to EmonTX

@AndyS,

Can you put together an environment capable of rebuilding and downloading the EmonPi binary?

There is a one-liner change to potentially lift that RSSI level out of the noise...

 

AndyS's picture

Re: EmonPi losing connection to EmonTX

Thanks for the feedback. I'm going to have to get the environment sorted shortly, as the stock 9V supplies give inaccurate VRMS results and power outputs are way off as a result of this and, perhaps, the CT measurement calibration : I'll need to work through the calibrations on both, which will also require a rebuild.  Just a case of finding time! 

As far as I can see the RSSI changes very little over time.  I don't understand why the EmonPi stops picking up the signal then does not reestablish connection.  Does the device only scan for remote signals when booted?

pb66's picture

Re: EmonPi losing connection to EmonTX

There have been some instances reported where the rfm module on the emonPi (or rfm2pi) has forgotten it's settings, this issue can be diagnosed/confirmed by changing the group to something else and back again via the emonhub.conf, it will only recognise a change so the "change and back again" is required to force the resending of the originally set group.if it is this issue packets should start to flow again pretty much imediatly.

Another possibility is that the emonTx (you haven't told us anything about it yet ) is sending a payload that has a lot of zero values, usually 6 unused temp values. This causes a "bit slip" issue and isn't normally terminal so the fact it doesn't remidy itself makes this less likely, do you normally see a long of zero values when it's working? If is it a late firmware with the positive fault values of 300 or a very early emonTx from prior to the 6 temps being added it shouldn't be a concern.

unfortunatly there is no way to see any unsuccesful packets on the emonPi as the ability to unset the "quiet" option has been removed so it's a process of elimination.

When you do the calibration process make sure you note the changes elsewhere or disable the emonPi update software since it will overwrite your changes (calibration) without any permision or confirmation when run. 

The env for uploading the binary is there already on the emonPi but hardcoded to upload from the emonpi git repo, you could point the repo to your own fork of the repo and just compile and save your own firmware to git from your PC but those changes will probally get overwritten in a emonpi update process too.

Try the group change and let us know about the emonTx payload.

Paul

emjay's picture

Re: EmonPi losing connection to EmonTX

unfortunately there is no way to see any unsuccessful packets on the emonPi as the ability to unset the "quiet" option has been removed so it's a process of elimination.

Hmm - really? Well that's fixed the missing packet issues at a stroke ! ;-)

AndyS's picture

Re: EmonPi losing connection to EmonTX

Thanks Paul,

I'm back to work so limited time to learn!.  The EmonTX is transmitting packets as follows (from Emonhub on EmonPi):

NEW FRAME : OK 7 199 253 132 0 45 0 0 0 19 98 180 0 184 11 184 11 184 11 184 11 184 11 1 0 0 0 (-78)

which are reported on the Nodes as 300DegC temps for absent sensors. The device has three active CT sensors, the AC-AC supply and one temperature sensor.

I'll try the Emonhub.conf change next time I find the connection has dropped and post back results.

I was a bit surprised how far out the solar generation was (+200w at 1700w, +15w at 0w), so will have to sort the calibration: thanks for the warnings. 

Andy

pb66's picture

Re: EmonPi losing connection to EmonTX

That frame looks good, there's no long runs of zeros and although a RSSI of -78 isn't great, I have a similar node that doesn't miss enough packets for it to be a concern 

I would be interested to hear if the settings "nudge" the emonPi's receiver back into play, see the Feeds stop updating thread for more info and a handy test script.

emjay - sorry my comment made it sound like the code was chopped in production, more accurately put I should have said the required code from the RFM2Pi/RF12demo firmware didn't make it into the emonPi's firmware. It was "removed" during the transition not since release. But yes it hides away any corrupt or damaged packets as there is no way of knowing the "not quiet" isn't actually working so it appears your set up has no imperfections. This combined with the non-RF packets being decorated to look like RF makes it less noticeable when you are not receiving any RF so debugging RF on the emonPi certainly isn't an easy task.

Paul

emjay's picture

Re: EmonPi losing connection to EmonTX

@Paul,

Is there some way to re-instate optional bad packet logging?  There is probably a gremlin  lurking in the RFM69 handling, but without some trace capability, it remains elusive.

What has proved useful is the timestamped comparison of good/bad CRC packet traces with RSSI appended.  Bit synch slip was found this way, perhaps the lock up cause will appear, though this might need tracing the command sequences which seem to have a different character level 'slip' happening.

 

pb66's picture

Re: EmonPi losing connection to EmonTX

@emjay - I am not familiar enough with the emonPi firmware to comment on how much of a task it would be nor why it's not there in the first place, passing the data into emonhub gives us timestamps and logging, both are essential to debugging as has been proven on many occasions.

Paul  

AndyS's picture

Re: EmonPi losing connection to EmonTX

@Paul - thanks for the pointers: I have the script ready. As per the usual law, there has not been another break in the transmissions since I started the thread.

Andy

AndyS's picture

Re: EmonPi losing connection to EmonTX

Well, it took six days for the connection to drop this time which was a record.  However, connectivity stopped in the middle of the night for some reason.  Using EmonHub config to change the group out and back to 210 reestablished the comms immediately.  Good to know I don't have to go out to the garage any longer to get reporting back!

Andy

 

glyn.hudson's picture

Re: EmonPi losing connection to EmonTX

Hi guys,

I have been working on a fix for this issue.

Please, can you confirm that you are running V2.4 firmware. and are still having drop off issues? To check firmware version, power down (fully) then, on powerup the LCD will display firmware version. If you are not running V2.4 then please run emonpi update. V2.4 was compiled with a known working version of jeelib RF driver. See master branch change log

------------------------------------------------------------------------------------------------------

If you are running V2.4 and still experiencing drop-offs, I have taken things a step further and introduced a watchdog to 'poke' the RFM60CW every 60s to keep it awake. This should certainly fix the issue but it's a bit of a hack. I have pushed this to V2.5 firmware which is on the 'rf_hang' development branch. To switch to rf_hang development branch you will need to connect via ssh then run

$ rpi- rw 
$ cd emonpi 
$ git pull 
$ git checkout rf_hang 
$ git pull

Then run emonPi update as usual via the web interface. Check on the LCD to see V2.5. Please let me know if this fixes your issue.

Here is the commit to the 'rf_hang' dev branch detailing the fix /hack to poke the RFM69CW to keep it alive: 

https://github.com/openenergymonitor/emonpi/commit/beb96e092b6591a4cba48...

rf_hang branch changelog

I've been running it for a few days now on my test systems, and it seems to be working well with no detrimental effects. 

AndyS's picture

Re: EmonPi losing connection to EmonTX

Now at 2.4: the commissioning update was done on 8th March, so just missed the 2.4 release I guess.  Do you want me to test 2.5 for you, or monitor on 2.4 for a while?  Your support is much appreciated: thanks.

Andy

AndyS's picture

Re: EmonPi losing connection to EmonTX

Red Herring Warning applies, but I notice that the  LCD on the EmonPi now goes off after some period of inactivity, requiring a button push to get the display illuminated.  It did not before under 2.3.

glyn.hudson's picture

Re: EmonPi losing connection to EmonTX

Please use V2.4, I am keen to hear if this fixes your issue. If the issue happens again when running V2.4 then at that point please update to V2.5.

The LCD operation is independent to the firmware version. The emonPi update also pulls in updates to the python LCD service and Emoncms, the LCD backlight turning off after 5min so is intended operation to save energy and light pollution! If you don't like this you can change L in emonpi/lcd/emonPiLCD.py :

https://github.com/openenergymonitor/emonpi/blob/master/lcd/emonPiLCD.py...

AndyS's picture

Re: EmonPi losing connection to EmonTX

Connection dropped yesterday again, with the group edit and reset via the config editor returning everything to operational.  I'll try V2.5 at the weekend, unless I get called away on business.

2.4 default calibrations are good enough that I don't have to worry about this any more: very substantial drop in the difference between the inverter measurements of Vrms, power etc and those of the EmonPi system.  Thanks for this!

Andy

glyn.hudson's picture

Re: EmonPi losing connection to EmonTX

Thanks for the update. Please let me know how you get on with V1.5. Remember you will need to change git branch of the emonPi repo. See instructions in my previous post.

V1.5 has been totally rock solid for me. 

AndyS's picture

Re: EmonPi losing connection to EmonTX

I just got back from three weeks on site. Connection looked to have been rock solid under 2.4 until yesterday: must be me back in the house!  Noted that 2.5 was now mainstream so ran the standard upgrade.  Will monitor again. Issues with apps under 2.5 (Undefined offset: 0 in /var/www/emoncms/Theme/memu_view.php on line 42) which I will now troubleshoot.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.