Archived Forum

Hi Norm

It looks like the issue is because the arduino is reconstructing the byte values received via the rfm12b before tx over serial. emonhub is expecting a string of byte values eg node 10 L, h, h, h, h, l, l, l, l should be 29 numbers between 0 and 255 (inc node id), which is why it is saying the received data length of 9 values is not valid. Likewise node 19 should be 7 byte values not 3.

Effectively you have 2 decoding stages one prior to the serial link and one after, one needs to be disabled. The quick fix would be to just remove or comment out the nodes/datacodes lines in emonhub.conf as I'm guessing the arduino is already configured for the emonTx v3 sketch (9 values shown in serial print). The more flexible and scalable approach would be to edit the arduino sketch once to just pass the raw byte values (like a RFM2Pi board does) and then any changes or additions to your sensor network could be configured in emonhub.conf rather than rewriting sketches each time.

If you want to see more of what is happening in emonhub set the loglevel to debug in emonhub.conf.

Paul

Hi Paul, thank for your reply,

I edited emonhub.conf as follows.

#######################################################################
#######################          Nodes          #######################
#######################################################################
#[nodes]

# List of nodes by node ID
# 'datacode' is default for node and 'datacodes' are per value data codes.
# if both are present 'datacode' is ignored in favour of 'datacodes'
#[[99]]
#       datacode = h
#       datacodes = l, h, h, h,

#[[10]]
#       datacodes = L, h, h, h, h, l, l, l, l

#[[19]]
#        datacodes = h, h, h

This is what I get now with DEBUG on.

2014-08-15 19:38:38,746 DEBUG 18 NEW FRAME : 1408088318.75 10 86271 910 10 0 346 15697 26010 0 14133
2014-08-15 19:38:38,750 WARNING 18 RX data length: 9 is not valid for datacode h
2014-08-15 19:38:43,636 DEBUG 19 NEW FRAME : 1408088323.64 10 86273 908 9 0 347 15698 26010 0 14133
2014-08-15 19:38:43,640 WARNING 19 RX data length: 9 is not valid for datacode h
2014-08-15 19:38:53,400 DEBUG 20 NEW FRAME : 1408088333.4 19 215 483 30
2014-08-15 19:38:53,404 WARNING 20 RX data length: 3 is not valid for datacode h
2014-08-15 19:38:58,693 DEBUG 21 NEW FRAME : 1408088338.69 10 86281 903 9 0 346 15702 26010 0 14135
2014-08-15 19:38:58,697 WARNING 21 RX data length: 9 is not valid for datacode h

I guess I'll have a go at a different Arduino sketch. Do you think I can run the RF12_Demo_atmega328.ino sketch on a Arduino UNO  with RFM12B without too much trouble?

Norm

Just had another thought, maybe the interface I'm using in emonhub.conf is incorrect for the Arduino/RFM12B/USB?

Is there an alternative?

[interfacers]

# This interfacer manages the RFM2Pi module
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtimesettings]]]

Any ideas?

That interface was designed to be primarily used with the RFM2Pi but intended to be compatible  with all standard "rfm" devices, the old standard was to send data over an "rfm" network in byte pair integers only so there is a default datacode of 'h' inbuilt to provide backwards compatibility to all the standard "byte pair integer only" sketches by default.

Your application presents a previously unencountered scenario which I think emonhub should be able to cope with.

Try over riding the default value by adding "datacode = False" to the interfacer settings in emonhub.conf

# This interfacer manages the RFM2Pi module
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtimesettings]]]
        datacode = False

Paul

Hi again Paul, nice to have someone helping me out! Much appreciated.

I have added datacode = False now this is all I get:

pi@raspberrypi ~ $ tail -f /var/log/emonhub/emonhub.log
1970-01-01 12:00:23,645 INFO EmonHub Pre-Release Development Version (rc1.0)
1970-01-01 12:00:23,648 INFO Opening hub...
1970-01-01 12:00:23,651 INFO Logging level set to DEBUG
1970-01-01 12:00:23,654 INFO Creating EmonHubEmoncmsReporter 'emonCMS'
1970-01-01 12:00:23,659 INFO Set up reporter 'emonCMS' (buffer: memory | size: 1000)
1970-01-01 12:00:23,663 INFO Creating EmonHubJeeInterfacer 'RFM2Pi'
1970-01-01 12:00:23,666 DEBUG Opening serial port: /dev/ttyACM0
1970-01-01 12:00:23,672 INFO Opened serial port: /dev/ttyACM0 9600 bits/s
1970-01-01 12:00:23,675 DEBUG Setting RFM2Pi datacode: False
1970-01-01 12:00:23,696 DEBUG 1 NEW FRAME : 23.7 10 8019 0 14476

No data shows up on emoncms.org

I'll post the whole emonhub.conf just incase I missed something obvious.

# SPECIMEN emonHub configuration file
# Note that when installed from apt, a new config file is written
# by the debian/postinst script, so changing this file will do
# nothing in and of itself.

# Each Interfacer and each Reporter has
# - a [[name]]: a unique string
# - a type: the name of the class it instantiates
# - a set of init_settings (depends on the type)
# - a set of runtimesettings (depends on the type)
# Both init_settings and runtimesettings sections must be defined,
# even if empty. Init settings are used at initialization,
# and runtime settings are refreshed on a regular basis.

# All lines beginning with a '#' are comments and can be safely removed.

#######################################################################
#######################    emonHub  settings    #######################
#######################################################################
[hub]

# loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
# see here : http://docs.python.org/2/library/logging.html
loglevel = DEBUG

#######################################################################
#######################        Reporters        #######################
#######################################################################
[reporters]

# This reporter sends data to emonCMS
[[emonCMS]]
    Type = EmonHubEmoncmsReporter
    [[[init_settings]]]
    [[[runtimesettings]]]
        url = http://emoncms.org
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yes I added it :)

#######################################################################
#######################       Interfacers       #######################
#######################################################################
[interfacers]

# This interfacer manages the RFM2Pi module

[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtimesettings]]]
        datacode = False

#######################################################################
#######################          Nodes          #######################
#######################################################################
#[nodes]

# List of nodes by node ID
# 'datacode' is default for node and 'datacodes' are per value data codes.
# if both are present 'datacode' is ignored in favour of 'datacodes'
#[[99]]
#       datacode = h
#       datacodes = l, h, h, h,

#[[10]]
#       datacodes = L, h, h, h, h, l, l, l, l

#[[19]]

#        datacodes = h, h, h

Thanks heaps!

Norm

The log entries are all dated 1970 which they weren't  beforehand, have you changed anything else ? if not just try restarting emonhub using

sudo service emonhub restart

if that doesn't sort the dates out, comment out the "datacode = False" line and restart emonhub as above, to see if it returns to reporting the correct time and date, adding the line as suggested should not have effected the time & date or logging settings, so we should sort that before we move on.

Where there any further log entries? the log posted shows nothing abnormal beyond the wrong dates and the "rfm" settings are missing which you will have removed from the conf file due to your set-up. The last entry looks incomplete the "NEW FRAME"  received indicates 3 values (8019 0 14476) from node 10 with a time stamp of 23.7 which although not the right time it corresponds to the 23.696 seconds after midday on 1/1/1970 in all the log entries.

I will try and set something up to replicate what you have, can you attach a copy of your arduino sketch?

Since you are just using serial to transfer data not control an rfm device it maybe better to create a simple EmonHubSerialInterfacer  rather than using the EmonHubJeeInterfacer but we can look at that once we know a bit about why it's misbehaving.

Paul

Also now I have had another look back at the code I think the datacode should of been set to 0 (zero) not False. (appologies) False indicates a default datacode is not specified (ie use per value decoding) rather than a specifying a default datacode of none (ie all values not to be decoded).

Try "datacode = 0" once the log dates are back.

Or I see no obvious reason (so not conclusive) why changing the type from EmonHubJeeInterfacer to EmonHubSerialInterfacer shouldn't work if you fancy trying it, the default type for the serial interfacer is 0 so the "datacode =" entry wouldn't be needed

Paul

Hi Paul,

I was rebooting the Pi after config changes, thanks for the restart emonhub tip!  (sudo service emonhub restart)

datacode = False seemed to cause the date problem above.

datacode = 0 results in the following errors.

pi@raspberrypi ~ $ sudo service emonhub restart
[ ok ] Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
pi@raspberrypi ~ $ Traceback (most recent call last):
  File "/usr/share/emonhub/emonhub.py", line 336, in <module>
    hub.run()
  File "/usr/share/emonhub/emonhub.py", line 90, in run
    values = I.read()
  File "/home/pi/emonhub/src/emonhub_interfacer.py", line 368, in read
    return self._process_frame(f, t)
  File "/home/pi/emonhub/src/emonhub_interfacer.py", line 106, in _process_frame
    frame = self._decode_frame(ref, validated)
  File "/home/pi/emonhub/src/emonhub_interfacer.py", line 201, in _decode_frame
    elif len(data) % ehc.check_datacode(datacode) != 0:
ZeroDivisionError: integer division or modulo by zero
^C

pi@raspberrypi ~ $ tail -f /var/log/emonhub/emonhub.log
2014-08-16 09:04:25,576 INFO Opening hub...
2014-08-16 09:04:25,579 INFO Logging level set to DEBUG
2014-08-16 09:04:25,581 INFO Creating EmonHubEmoncmsReporter 'emonCMS'
2014-08-16 09:04:25,587 INFO Set up reporter 'emonCMS' (buffer: memory | size: 1000)
2014-08-16 09:04:25,591 INFO Creating EmonHubJeeInterfacer 'RFM2Pi'
2014-08-16 09:04:25,594 DEBUG Opening serial port: /dev/ttyACM0
2014-08-16 09:04:25,599 INFO Opened serial port: /dev/ttyACM0 9600 bits/s
2014-08-16 09:04:25,602 DEBUG Setting RFM2Pi datacode: 0
2014-08-16 09:04:25,831 DEBUG 1 NEW FRAME : 1408136665.83
10 110446 346 1182 0 349 23836 27028 0 18536

Tried EmonHubSerialInterfacer instead of EmonHubJeeInterfacer doesn't seem to make any difference.

pi@raspberrypi ~ $ tail -f /var/log/emonhub/emonhub.log
2014-08-16 09:18:30,489 DEBUG 104 NEW FRAME : 1408137510.49 10 110869 158 1302 0 350 23878 27318 0 18618
2014-08-16 09:18:30,493 WARNING 104 RX data length: 9 is not valid for datacode h
2014-08-16 09:18:35,370 DEBUG 105 NEW FRAME : 1408137515.37 10 110871 96 1304 0 350 23878 27320 0 18619
2014-08-16 09:18:35,374 WARNING 105 RX data length: 9 is not valid for datacode h
2014-08-16 09:18:40,462 DEBUG 106 NEW FRAME : 1408137520.46 10 110874 16 1305 0 350 23878 27321 0 18619
2014-08-16 09:18:40,466 WARNING 106 RX data length: 9 is not valid for datacode h
2014-08-16 09:18:45,354 DEBUG 107 NEW FRAME : 1408137525.35 10 110876 152 1307 0 349 23879 27324 0 18619
2014-08-16 09:18:45,358 WARNING 107 RX data length: 9 is not valid for datacode h
2014-08-16 09:18:50,444 DEBUG 108 NEW FRAME : 1408137530.44 10 110879 148 1306 0 353 23879 27325 0 18620
2014-08-16 09:18:50,448 WARNING 108 RX data length: 9 is not valid for datacode h
2014-08-16 09:18:55,338 DEBUG 109 NEW FRAME : 1408137535.34 10 110881 188 1308 0 352 23879 27327 0 18620
2014-08-16 09:18:55,342 WARNING 109 RX data length: 9 is not valid for datacode h

I'm thinking emonhub can't handle the decoded RFM12B string. I've attached my Arduino sketch. Maybe we can modify so it prints the undecoded string? 

Thanks heaps

Norm

Oops made a mistake. I changed EmonHubEmoncmsReporter instead of EmonHubJeeInterfacer!

As you suggested I changed to EmonHubSerialInterfacer and it works!!!!!!!!!!

[interfacers]

# This interfacer manages the RFM2Pi module
[[RFM2Pi]]
     Type = EmonHubSerialInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtimesettings]]]
#       datacode = 0

pi@raspberrypi ~ $ tail -f /var/log/emonhub/emonhub.log
2014-08-16 09:31:50,817 DEBUG 49 Append to 'emonCMS' buffer => time: 1408138310.72, data: [10, 111269, 2542, 1422, 0, 354, 24308, 27621, 0, 18696], ref: 49
2014-08-16 09:31:50,921 INFO Sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1408138310.72,10,111269,2542,1422,0,354,24308,27621,0,18696]]&sentat=1408138310
2014-08-16 09:31:52,206 DEBUG Receipt acknowledged with 'ok' from http://emoncms.org
2014-08-16 09:31:55,822 DEBUG 50 NEW FRAME : 1408138315.82 10 111271 2353 1422 0 350 24310 27623 0 18697
2014-08-16 09:31:55,826 DEBUG 50 Timestamp : 1408138315.82
2014-08-16 09:31:55,828 DEBUG 50      Node : 10
2014-08-16 09:31:55,831 DEBUG 50    Values : [111271, 2353, 1422, 0, 350, 24310, 27623, 0, 18697]
2014-08-16 09:31:55,925 DEBUG 50 Append to 'emonCMS' buffer => time: 1408138315.82, data: [10, 111271, 2353, 1422, 0, 350, 24310, 27623, 0, 18697], ref: 50
2014-08-16 09:31:56,029 INFO Sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1408138315.82,10,111271,2353,1422,0,350,24310,27623,0,18697]]&sentat=1408138316
2014-08-16 09:31:57,411 DEBUG Receipt acknowledged with 'ok' from http://emoncms.org
2014-08-16 09:32:00,719 DEBUG 51 NEW FRAME : 1408138320.72 10 111274 2443 1422 0 351 24314 27624 0 18697
2014-08-16 09:32:00,723 DEBUG 51 Timestamp : 1408138320.72
2014-08-16 09:32:00,728 DEBUG 51      Node : 10
2014-08-16 09:32:00,731 DEBUG 51    Values : [111274, 2443, 1422, 0, 351, 24314, 27624, 0, 18697]
2014-08-16 09:32:00,829 DEBUG 51 Append to 'emonCMS' buffer => time: 1408138320.72, data: [10, 111274, 2443, 1422, 0, 351, 24314, 27624, 0, 18697], ref: 51
2014-08-16 09:32:00,934 INFO Sending: http://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1408138320.72,10,111274,2443,1422,0,351,24314,27624,0,18697]]&sentat=1408138320
2014-08-16 09:32:01,870 DEBUG Receipt acknowledged with 'ok' from http://emoncms.org

Paul you've been an awesome help! Many thanks.

Norm

I have not been able to replicate the date issue so I'm not entirely sure what is happened there. nor am I sure why when trying the "EmonHubSerialInterfacer" it appears to be using a datacode of "h" the default is "0" so unless specified in the emonhub.conf should remain as zero.

The first of the results sets you posted, for the "datacode = 0" appears to be a variable type casting issue, the zero is being processed as a string"0" rather than a integer, therefore it goes wrong at this point as string zero is True where as int zero is False. Therefore it misses the if statement intended to filter off and return frames that require no decoding and continues till it trips up here with the divide by zero error

The logic worked ok in testing because if a "datacode" was specified by a user it was assumed it would be adding a "datacode" and not possibly removing one. Therefore the casting of a default int zero to a string "h" or "L" etc didn't create an issue and curiously nor does removing a datacode which causes the default zero to be reloaded, probably due to configObj.

Anyway, I believe changing this line to also test for a "0" string will remedy your issue so in ~/emonhub/src/emonhub_interfacer.py try changing line 193 to " if not datacode or datacode == '0': " (not a permanent solution but should do what you want until emonHub is updated

What is obvious from this experimenting there needs to be some filtering/casting of datacode values the user can enter as there is a vulnerability, The intention is emonHub will raise exceptions and continue rather than crash on error, a bulk of the code is filtering and error handling. 

Although not an average application, I'm sure it will pop up again so I am keen to resolve this and I do appreciate your help highlighting and (hopefully) resolving this, However, as mentioned in my first post, in the long term I think altering your sketch to pass undecoded values is the way to go (I haven't looked at it yet). But if you prefer to decode in your sketch we can also retry the Serial Interfacer, your choice :-)

See how we get on with this fix first I guess

Paul

So maybe I should check PM's more frequently, 

normosb 15/08/2014 - 22:37
Hi Paul,

It works! thanks very much for your help! Change to Serial did the trick;

My latest thrend post went to the moderator :)

# This interfacer manages the RFM2Pi module
[[RFM2Pi]]
#    Type = EmonHubJeeInterfacer
     Type = EmonHubSerialInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtimesettings]]]
#       datacode = 0

Much appreciated!

Regards

Norm

Thank you for letting me know, that clears the uncertainty as to why the serialinterfacer didn't work and I'm now confident the latter fix would of worked also. great.

Thanks again for putting emonhub to the test. I have started adding additional "datacode" qualifying.

Paul

Hello,

I just set up my Raspberry Pi, but I'm getting similar errors.

2015-03-20 21:43:29,243 DEBUG 17 NEW FRAME : 1426887809.24 OK 10 42 2 0 0 0 0 0 0 (-51)
2015-03-20 21:43:29,248 WARNING 17 RX data length: 8 is not valid for datacodes ['L', 'h', 'h', 'h', 'h', 'l', 'l', 'l', 'l']
2015-03-20 21:43:31,611 DEBUG 18 NEW FRAME : 1426887811.61 OK 10 40 2 0 0 0 0 0 0 (-52)
2015-03-20 21:43:31,616 WARNING 18 RX data length: 8 is not valid for datacodes ['L', 'h', 'h', 'h', 'h', 'l', 'l', 'l', 'l']

My arduino uses the new RF96 board.

I'll include the sketch:

http://pastebin.com/ZWNe8Y4y

And my emonhub.conf

http://pastebin.com/eKW94TnD

I used the 'Write API Key' from my account. Correct?

Regards,

Hans

That sketch sends the node number followed by a payload of 4 integers (8 bytes). That is specified in

  typedef struct { int power1, power2, power3, power4;} PayloadTX;

therefore you should not need to specify datacodes. The default should be fine.
The datacode you specified is expecting:

  unsigned long, int, int, int, int, long, long, long, long.

And that is not what the sketch is sending, hence the error.

The "read & write" APIkey is correct.

Thanks.

Commenting out these lines did the trick. Everything is up and running now.