Archived Forum

Update - I have tried a few things such as turning off each of the EmonTX in turn to see what happens. I still find that with just the emonTH transmitting to the Pi the connection to these is lost after a few hours running. Is this some sort of hardware failure of the wireless module on the board?

Jez

Can you confirm where you are seeing the fault?

Have you checked the emonhub.log?

The Pi it self is totally up to the job, the 6 devices, 2 of which will have very long payloads will put the RF network under load and there is a possibility  probability of clashes etc, you may need to implement some traffic calming and/or control but these things are not just setttings they will involve coding changes (manly configurations), sketch uploads and abit of trial and error.

There may be a particular issue that is the biggest culprit so debugging may find a single solution that makes the error level of that much traffic a little more palatable,

Paul

Paul

All I can see on the emonhub.log is that the the data from the transmitter nodes just disappear from the log. No sign of any errors.

I have gone back to a transmitter configuration that previously had been working with no problem - so the 2 CTs connected to the emonPi, one TX with 4 xCTs and 1 x TH. This configuration worked longer than the full set of transmitters, but eventually it lost the connection to both. The red LED on the TX is still flashing - so presumably still transmitting. Turning the power on the TX on and off to restart it does not re-establish the connection to the emonPi. This is what makes me think it is a hardware issue rather than software.

Jez

"All I can see on the emonhub.log is that the the data from the transmitter nodes just disappear from the log. No sign of any errors."

Is there any other data arriving via the rfm2pi? are there any "rfm2pi errors" prior to the packets ceasing?

Paul

Paul

There are no errors on the log. Is there a way to view the historic emonhub log to see if I can spot where the TX data stop? I have tried using $ tail -f /var/log/emonhub/emonhub.log in putty but that just gives the same live log feed as the log viewer. Current log view below

Jez

2015-09-23 05:35:35,235 DEBUG RFM2Pi 121112 Timestamp : 1442986535.23
2015-09-23 05:35:35,235 DEBUG RFM2Pi 121112 From Node : 5
2015-09-23 05:35:35,236 DEBUG RFM2Pi 121112 Values : [154, 5, 159, 235.82, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:35,237 INFO RFM2Pi Publishing: emonhub/rx/5/values 154,5,159,235.82,17.3,0,0,0,0,0,0
2015-09-23 05:35:35,240 DEBUG RFM2Pi 121112 adding frame to buffer => [1442986535, 5, 154, 5, 159, 235.82, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:35,241 DEBUG RFM2Pi 121112 Sent to channel' : ToEmonCMS
2015-09-23 05:35:40,296 DEBUG RFM2Pi 121113 NEW FRAME : OK 5 151 0 7 0 158 0 221 91 173 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 05:35:40,300 DEBUG RFM2Pi 121113 Timestamp : 1442986540.3
2015-09-23 05:35:40,301 DEBUG RFM2Pi 121113 From Node : 5
2015-09-23 05:35:40,302 DEBUG RFM2Pi 121113 Values : [151, 7, 158, 235.17000000000002, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:40,303 INFO RFM2Pi Publishing: emonhub/rx/5/values 151,7,158,235.17,17.3,0,0,0,0,0,0
2015-09-23 05:35:40,305 DEBUG RFM2Pi 121113 adding frame to buffer => [1442986540, 5, 151, 7, 158, 235.17000000000002, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:40,306 DEBUG RFM2Pi 121113 Sent to channel' : ToEmonCMS
2015-09-23 05:35:41,859 INFO emoncmsorg sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1442986515,5,154,6,160,235.77,17.3,0,0,0,0,0,0],[1442986520,5,186,5,191,235,17.3,0,0,0,0,0,0],[1442986525,5,154,4,158,235.9,17.3,0,0,0,0,0,0],[1442986530,5,143,5,148,235.09,17.3,0,0,0,0,0,0],[1442986535,5,154,5,159,235.82,17.3,0,0,0,0,0,0],[1442986540,5,151,7,158,235.17000000000002,17.3,0,0,0,0,0,0]]&sentat=1442986541
2015-09-23 05:35:42,634 DEBUG emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org
2015-09-23 05:35:45,243 DEBUG RFM2Pi 121114 NEW FRAME : OK 5 149 0 6 0 155 0 62 92 173 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 05:35:45,246 DEBUG RFM2Pi 121114 Timestamp : 1442986545.24
2015-09-23 05:35:45,247 DEBUG RFM2Pi 121114 From Node : 5
2015-09-23 05:35:45,248 DEBUG RFM2Pi 121114 Values : [149, 6, 155, 236.14000000000001, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:45,249 INFO RFM2Pi Publishing: emonhub/rx/5/values 149,6,155,236.14,17.3,0,0,0,0,0,0
2015-09-23 05:35:45,251 DEBUG RFM2Pi 121114 adding frame to buffer => [1442986545, 5, 149, 6, 155, 236.14000000000001, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:45,252 DEBUG RFM2Pi 121114 Sent to channel' : ToEmonCMS
2015-09-23 05:35:45,842 INFO emoncmsorg sending: http://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2015-09-23 05:35:50,277 DEBUG RFM2Pi 121115 NEW FRAME : OK 5 156 0 6 0 162 0 222 91 173 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 05:35:50,317 DEBUG RFM2Pi 121115 Timestamp : 1442986550.28
2015-09-23 05:35:50,319 DEBUG RFM2Pi 121115 From Node : 5
2015-09-23 05:35:50,320 DEBUG RFM2Pi 121115 Values : [156, 6, 162, 235.18, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:50,321 INFO RFM2Pi Publishing: emonhub/rx/5/values 156,6,162,235.18,17.3,0,0,0,0,0,0
2015-09-23 05:35:50,323 DEBUG RFM2Pi 121115 adding frame to buffer => [1442986550, 5, 156, 6, 162, 235.18, 17.3, 0, 0, 0, 0, 0, 0]
2015-09-23 05:35:50,324 DEBUG RFM2Pi 121115 Sent to channel' : ToEmonCMS

emonhub.log is just a text file and can be opened with a normal editor such as vi or nano, however the files can get quite bulky (5mb) and slow to handle on the PI, You can also use cat, echo, more or probably the better one is  the"less" command.

less /var/log/emonhub/emonhub.log

gives you a page by page view, then "G" will take you to the end of the file

? From Node : 10 should search out that "From Node : 10" string working backwards, ie find the last occurrence, this only works when it is in a failed state, 

You mat find it easier to either copy the log to a PC to dissect or reboot the Pi to clear the logs and monitor closely for the 1st fail, grabbing the logs then will minimise the log size and place the area of interest right at the very end.

To be honest with you I'm not convinced there will be any useful info but cannot be 100% sure. It looks like the emonpi board continues to report data but not from the rf devices, so there may be nothing than an absence of data to be seen.

Something else to try next time it occurs is to change the group to something else in the rfm2pi section of emonhub.conf, save the file and then change it back to your current group. I recall an issue Glyn had some time ago where the emonPi was forgetting the rf settings and forcing a "reload config" would kick start the reporting again, maybe the same, maybe not but worth a try!!

Paul 

Is that (-0) in the tail of NEW FRAME meant to be the RSSI value, or just some reporting artefact?

The "(-0)" has no purpose, it's hard-coded in the emonPi sketch to "match" the rfm69 packet appearance.

Paul

Paul

I have rebooted the EmonPi - and it then started to receive data from the emonTX again (node 9). The connection only lasted 20 minutes this time. I have downloaded the emonhub.log file (using WinSCP) and identified the last transmission in the log but can find no associated errors. The log at the point of disconnection is below. I will try your suggestion of temporarily changing the group and then back to 210 in the rfm2Pi config.

cheers

Jez

2015-09-23 19:44:36,045 DEBUG    RFM2Pi     951 NEW FRAME : OK 9 8 0 0 0 0 0 5 0 213 94 0 0 0 0 0 0 0 0 0 0 0 0 9 0 (-28)
2015-09-23 19:44:36,048 DEBUG    RFM2Pi     951 Timestamp : 1443037476.04
2015-09-23 19:44:36,049 DEBUG    RFM2Pi     951 From Node : 9
2015-09-23 19:44:36,050 DEBUG    RFM2Pi     951    Values : [8, 0, 0, 5, 242.77, 0, 0, 0, 0, 0, 0, 9]
2015-09-23 19:44:36,050 DEBUG    RFM2Pi     951      RSSI : -28
2015-09-23 19:44:36,052 INFO     RFM2Pi     Publishing: emonhub/rx/9/values 8,0,0,5,242.77,0,0,0,0,0,0,9
2015-09-23 19:44:36,053 DEBUG    RFM2Pi     951 adding frame to buffer => [1443037476, 9, 8, 0, 0, 5, 242.77, 0, 0, 0, 0, 0, 0, 9, -28]
2015-09-23 19:44:36,054 DEBUG    RFM2Pi     951 Sent to channel' : ToEmonCMS
2015-09-23 19:44:36,249 INFO     emoncmsorg sending: http://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2015-09-23 19:44:40,829 DEBUG    RFM2Pi     952 NEW FRAME : OK 5 199 0 5 0 204 0 216 91 190 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 19:44:40,833 DEBUG    RFM2Pi     952 Timestamp : 1443037480.83
2015-09-23 19:44:40,834 DEBUG    RFM2Pi     952 From Node : 5
2015-09-23 19:44:40,835 DEBUG    RFM2Pi     952    Values : [199, 5, 204, 235.12, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:40,836 INFO     RFM2Pi     Publishing: emonhub/rx/5/values 199,5,204,235.12,19,0,0,0,0,0,0
2015-09-23 19:44:40,838 DEBUG    RFM2Pi     952 adding frame to buffer => [1443037480, 5, 199, 5, 204, 235.12, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:40,839 DEBUG    RFM2Pi     952 Sent to channel' : ToEmonCMS
2015-09-23 19:44:45,906 DEBUG    RFM2Pi     953 NEW FRAME : OK 5 195 0 5 0 200 0 120 91 190 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 19:44:45,910 DEBUG    RFM2Pi     953 Timestamp : 1443037485.91
2015-09-23 19:44:45,910 DEBUG    RFM2Pi     953 From Node : 5
2015-09-23 19:44:45,911 DEBUG    RFM2Pi     953    Values : [195, 5, 200, 234.16, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:45,912 INFO     RFM2Pi     Publishing: emonhub/rx/5/values 195,5,200,234.16,19,0,0,0,0,0,0
2015-09-23 19:44:45,915 DEBUG    RFM2Pi     953 adding frame to buffer => [1443037485, 5, 195, 5, 200, 234.16, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:45,916 DEBUG    RFM2Pi     953 Sent to channel' : ToEmonCMS
2015-09-23 19:44:50,867 DEBUG    RFM2Pi     954 NEW FRAME : OK 5 195 0 4 0 199 0 137 91 190 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2015-09-23 19:44:50,871 DEBUG    RFM2Pi     954 Timestamp : 1443037490.87
2015-09-23 19:44:50,872 DEBUG    RFM2Pi     954 From Node : 5
2015-09-23 19:44:50,872 DEBUG    RFM2Pi     954    Values : [195, 4, 199, 234.33, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:50,874 INFO     RFM2Pi     Publishing: emonhub/rx/5/values 195,4,199,234.33,19,0,0,0,0,0,0
2015-09-23 19:44:50,876 DEBUG    RFM2Pi     954 adding frame to buffer => [1443037490, 5, 195, 4, 199, 234.33, 19, 0, 0, 0, 0, 0, 0]
2015-09-23 19:44:50,877 DEBUG    RFM2Pi     954 Sent to channel' : ToEmonCMS
2015-09-23 19:44:53,775 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1443037470,5,200,5,205,234.66,19,0,0,0,0,0,0],[1443037475,5,197,5,202,235.21,19,0,0,0,0,0,0],[1443037476,9,8,0,0,5,242.77,0,0,0,0,0,0,9,-28],[1443037480,5,199,5,204,235.12,19,0,0,0,0,0,0],[1443037485,5,195,5,200,234.16,19,0,0,0,0,0,0],[1443037490,5,195,4,199,234.33,19,0,0,0,0,0,0]]&sentat=1443037493
2015-09-23 19:44:54,838 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org

Temporarily changing the group in the rfm2Pi config did not help - the connection went again after an hour.

Jez

Hi Jez unfortunately the logs show very little except that node9 is a very likely candidate for the "run of zeros issue" discussed in the "Data loss due to RF packets getting corrupted" thread, if it's not the cause now it could well be a problem at anytime (including the same time ie more than one problem).

Leading up to node9 stopping were there any other intermittent packets missing or was it perfect and then just stopped dead? 

Where are the other nodes? are they connected? do all the nodes (except node5)  definitely stop together?

Do your emonTH's have dht22's fitted? there is another unrelated issue that can effect them discussed in the "emonTH Unreliable reading timing?" thread, but if they (tx's and th's) all stop together it could be something completely different.

"the connection went again after an hour." Does that mean it did bring it back but reoccured after an hour, this isn't intended to fix the issue but if the settings "nudge" restore a lost signal for any amount of time is is significant.

Debugging is hampered somewhat by the emonpi sketch being permanently in "quiet-mode", we cannot see the problematic packets, we can only the ones there is nothing wrong with.

Whether diagnosing or fixing I think you will be altering some code as I suspect this is very much a software issue rather than a hardware one, but I wouldn't rule it out completely yet, do you have a programmer?

Another route which is an additional cost but IMO very much worth considering (in fact I would probably recommend it) is to invest in a second receiver, this will allow you to "snoop" on the network without interfering, give you an alternative place to tinker and test if you wish to experiment with expanding your sysytem.

A jeelink is a perfect little device for this as it is a usb dongle that can be plugged into any pc/laptop (or for that matter the emonPi as a second reciever) the JeeLink (v3c) is just over 30 euros and maybe the simplest way to find the issue.

Paul

Paul

answers as follows:

Leading up to node 9 stopping were there any other intermittent packets missing or was it perfect and then just stopped dead? Where are the other nodes? are they connected? do all the nodes (except node5)  definitely stop together? As far as I can tell it looked fine up to the point of losing the connection. For this test, to simplify issues I just had the one TX (node 9) and one TH (node 21) - the data frm both stopped within a minute of each other - the TH slightly before as the data is less frequent.

Do your emonTH's have dht22's fitted? 3 of my 4 THs have DHT22s - one of these was the one used in the test yesterday. However, in my original set-up I had the Pi plus 2 x the THs running for over a month with no problems.

"the connection went again after an hour." Does that mean it did bring it back but reoccured after an hour, this isn't intended to fix the issue but if the settings "nudge" restore a lost signal for any amount of time is is significant. In my test I first tried changing the group while the transmissions were being received OK - about an hour after this was when the connection was lost - I then tried changing the group again - but this did not restore the connection - so "nudging" doesn't seem to work. Only way to get them back was to reboot the Pi.

Debugging is hampered somewhat by the emonpi sketch being permanently in "quiet-mode", we cannot see the problematic packets, we can only the ones there is nothing wrong with. - Can I change this setting in emonhub config?

do you have a programmer? Yes - I have the USB to serial programmer - but haven't had the courage to use it.

One more thing that might be worth mentioning in case it helps is that in the past I have seen spurious nodes appear in the input list. Only happened very occasionally and I haven't seen any for a while. I do have other wireless kit in the house - in particular a Honeywell Evohome zoned heating/hot water controller - but this is on 868MHz not 433 so this shouldn't be a problem.

I have now set up my system back to it's original configuration with just the Pi and 2 xTH - and pulled the batteries and turned off the power from everything else. Will see how this goes.

Jez

The set-up in original configuration (Pi + 2xTH) lasted just 1 1/2 hours before the connection to both THs was lost (at the same time give or take a minute). Tried changing the group from 210 to 0 and back again - but like last time this had no effect. So the question is why is this simple set up not working now, when it ran for weeks with no problem before? Hence my suspicion this is a hardware issue.

Jez

Any reason to think the Pi may be running warmer than previously?

There are clearly packets still flying that the Pi side is ignoring. Really need that 'quiet mode' off and/or an independent 'listener' as Paul suggested to debug further.

Any reason to think the Pi may be running warmer than previously?

You can check that with either of these two commands, not entirely sure why they give slightly different readings, perhaps there's more than one sensor buried in there.

pi@raspberrypi1 ~ $ vcgencmd measure_temp
temp=51.9'C
pi@raspberrypi1 ~ $ cat /sys/class/thermal/thermal_zone0/temp 
51382
pi@raspberrypi1 ~ $ 

Hi Jez, If you are convinced it is hardware you can contact the shop support and refer to this thread, at this point the fact the hardware is normally sound and most issues get resolved in the software that leads me to explore those avenues first, but you have the advantage of seeing the issue first hand so don't let me sway you if you want to contact the shop.

There are other options to try, we could use a very small script to reset the receiver from the command line as currently rebooting the Pi when there is possibly a problem with the rfm2pi part is a bit like throwing the main breaker to you house when your PC won't connect to the web.

We could go ahead and install the previously successful fixes to your devices, we know they can be problematic and since the symtoms are not always permanently displayed the fact it previously worked isn't 100% conclusive it's not the problem now. Installing sketches isn't difficult, once you've done a couple you'll be fine. 

Debugging is only difficult in this instance because of the lack of quiet mode, unfortunately it's missing from the sketch so although the setting is available and processed by emonhub, the "rfm2pi" part has no non-quiet mode to activate. I have no idea why it was removed so cannot say how easy it would be to add it back in to a sketch, and then loading a modified sketch is not quite as straightforward as pressing the update button either.

The alternative as I mentioned, is to use a JeeLink which comes pre-installed with receiver software so should be plug and play to expose all the stuff you currently cannot see. bear in mind the devices you have on your network could be subject to the 2 issues I've mentioned independent of any potential issue with the emonPi.so a jeelink could still be a valuable asset even when the emonPi itself is behaving.

The choice is yours :-)

Paul

I don't think there's anything to stop you using a (for the moment, spare) emonTH with your programmer as a receiver! It needs a sketch writing, but you have the hardware. I think Robin's PV diverter page has a receiver sketch that would be a good starting point - otherwise one of the GLCD sketches.

That is actually a really good idea, the RFdemo12 (or an rfm2pi) sketch will load straight up (set the COMPAT mode 1st) and the group, frequency and quiet mode can be set in the Arduino IDE using commands afterwards. I have a emonTx v3.4 right here testing a modified "rfm2pi" sketch, not sure why it didn't occur to me!

The pins used by the RFdemo12 sketch are the same for both the (non-RFu) emonTH and emonTx so you could use a emonTH since they are more abundant (and will need the revised sketches anyway).

Good call Robert! Would you prefer to try that Jez?

Paul

I have checked the pi temp with just the wired sensors running and then after a reboot to get the pi to see the THs. In both cases the temp was quite stable ranging from37.9degC to 38.5degC. So doesn't look like the processor is getting hot

Jez

@Jez - try downloading and running this little reset script next time it fails. all it does is pulses the reset pin to the emonPi board causing it to restart without restarting the Pi. this may narrow the field a little.

rpi-rw

sudo wget -O /usr/bin/reset_rfm2pi http://openenergymonitor.org/emon/sites/default/files/reset_rfm2pi.txt

sudo chmod +x /usr/bin/reset_rfm2pi

​sudo reset_rfm2pi

​you can view the script here or in the "RFM12PI receiver goes hard down sometimes" thread.

Paul