Archived Forum

If the connection is serial over USB, you can use the OEM gateway. Instead of using a OemGatewayRFM2PiListener, you'll use a OemGatewaySerialListener. The only parameter is the COM port.

Make sure that the TX formats the data like this:

NodeID val1 val2
NodeID val1 val2

Example for temperature and relative humidity fluctuating around 19°C and 60%:

12 19.2 60.1
12 19.5 60.0
12 19.3 60.4

(No restriction on number size and decimal part.)

Tell us how it goes.

nice, I didn't know OEM gateway could do that. How do you set USB serial baud rate of serial listener. What's the default? 9600?

Good point.

Yes, currently it is hardcoded as 9600/

Let's change this.

https://github.com/Jerome-github/oem_gateway/issues/16

Thanks, Jerome.

I assume that you are saying that I should make my own build  using the code the you have placed on github, rather than using the ready-to-go SD card image (22 oct) available on the emoncms.org website.  Could you confirm? Since I have never used Python before and my programming days are 20 years ago this will be a bit new for me .... Or should it also work with the ready-to-go image?

The ready-to-go SD card contains the same gateway, roughly, perhaps not the exact latest version, but never mind. If you don't have anything else on your RPi already, you can use the ready-to-go SD Card.

Or you can add the gateway to your existing system. Nothing to build in python, it is script/interpreted just copy and execute. Instead of copying, you'd rather git clone (see help anywhere), so that you can pull later revisions. See gateway readme for installation procedure (file permissions, config file,...).

If the baud rate you use is not 9600, just modify the code to yours and restart the gateway.

Ok, sounds good. I shall try both options. 

One more question: How do I find out which NodeID I need to choose that I presumably need to include in the Arduino sketch, before the values, according to your post above, and where in the sketch do I include it? The standard arduino sketch does not seem to transfer a node in the way you have explained it above.

Thanks a lot for you help!

Please point me to the sketch you are referring to.

The node ID is up to you to decide. Just don't use two nodes with same node ID.

See this page, especially the end: http://openenergymonitor.org/emon/buildingblocks/rfm12b2

Hi Jerome,

here is the sketch that I am using:

https://github.com/openenergymonitor/emonTxFirmware/blob/master/emonTxShield/Shield_CT1234/Shield_CT1234.ino

In my uneducated state I am assuming that the data is transferred from Arduino to Pi with the serial.print() method, however, i am not clear how to transfer the nodeID.

thanks again.

OK, now I understand the context.

It's not as straightforward as I thought but it should be feasible.

The Node ID is 10, which is the default. If you have only one TX, keep it 10.

https://github.com/openenergymonitor/emonTxFirmware/blob/master/emonTxSh...

I didn't realize you were using the default serial/USB. Well, this may be unclear to you, but I thought you were using a sketch you had modified to send data over serial, not radio.

Currently, in your sketch, the serial link is not really used to transfer data, rather to be used as a debug tool, a logger. But I suppose you can use it like you want with few modifications.

Yes, the instruction is Serial.println("My string");

The lines in setup() won't be understood by the gateway but shouldn't make it crash. You may leave them.

What you need to do is ensure you write

NodeID val1 val2
NodeID val1 val2

In loop(), add the following line at the beginning:

Serial.print(nodeID); Serial.print(" ");

Watch the output and modify the code until you get the expected format.

Then you can try the gateway.

BTW, baud rate is 9600, so no need to modify anything.

In the gateway config, you'll need to enter the serial port path. I don't know it, and I don't think it will be the default one. But work on the TX output before, and you'll see this later.

Hi Jerome,

I did the changes to the sketch and checked the output format and it is ok. Unfortunately you are right with you assumption that I also need to change something in the config file. I have a feeling that I need to replace the listener:

Listeners #
#############
[listeners]

# This listener manages the RFM2Pi module
[[RFM2Pi]]
    type = OemGatewayRFM2PiListener
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
    [[[runtime_settings]]]
        sgroup = 210
        frequency = 8
        baseid = 15
        sendtimeinterval = 300

# This listener gets data from a socket
[[Socket]]
    type = OemGatewaySocketListener
    [[[init_settings]]]
        port_nb = 50011
    [[[runtime_settings]]]

thanks for your help. Otherwise tomorrow I'll try the code from github.

Regards,

Frank

Here is an update if you are interested.

I updated the conf file, even though this may not be the problem. 

The gateway is running the initialization, but then does not get any data from /dev/ttyAMA0. I reckon the problem is that the Arduino/EmonTX shield does not send anything like it does when I connect it to the RPi. However, it always sends ok when I put the other SD card into the RPi.

Could it be that the RPi gateway image does not support USB? How could I add the USB support to the gateway?

/dev/ttyAMA0 is the Pi's UART serial port not the Pi's USB port.  You will need to change that to point at the USB port

The address of that will change dependant on which USB port you are using.

if you do a ls the dev directory you should be able to determine the right port (/dev/ttyUSB<something>)

Since there's only two USB ports on your Pi, it'll be /dev/ttyUSB0 or /dev/tty/USB1

Hey Jerome 

Great work on the serial gateway. I've just set up the oem gateway posting data from an Arduino connected via USB to both local and remote emoncms. The whole thing worked first time without any issues. I will try and do a quick blog post to highlight this functionality of the oem gateway. I'm not sure many people realize it can be used like this. It worked well on conjunction with ino to compile and upload code for the arduino all on the pi

http://openenergymonitor.blogspot.com/2013/12/developing-for-arduino-remotely-on.html 

Hi, thanks everyone, I figured it out now. In my case the correct port actually  is /dev/ttyACM0. works great, the oemgateway is now getting data from the Shield/Arduino.

However, its not uploading anything to the remote server (emoncms.org) as it should. Here is an excerpt of the log:

    2014-01-26 20:01:30,981 INFO Serial RX: 10 4 263.56 0.36
    2014-01-26 20:01:30,984 WARNING Misformed RX frame: ['10', '4', '263.56', '0.36']

Anyone any good ideas? Here is the listener in config file I have created. I left the runtime settings empty as Jerome suggest.

#############
# Listeners #
#############
[listeners]

# This listener manages the serial input
[[Serial]]
    type = OemGatewaySerialListener
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtime_settings]]]
 

Could it be that the Rock Solid Gateway Forwarder (RSGF)does not activate the USB port on the Pi? It only shows the serial port  /dev/ttyAMA0 and it is unable to open that one. Also the LED on the Arduino does not light up when I use the SD card with the RSGF. However, it does when I use the SD card where I have the full environment and the oemgateway code in python from jerome-github. Any help is much appreciated!

I've just done a quick blog post to highlight using the OEM gateway as a serial > emoncms gateway: http://openenergymonitor.blogspot.co.uk/2014/01/oem-gateway-serial-port-emoncms-link.html

Frank: could you post your entire oemgateway.conf fie so we can check all the emocms setting look OK (remember to strip out your API key!). It sounds like you are nearly there. 

I don't think the RSGF does any sort of disabling of the USB ports 

Hi guys.

I'm glad this feature proves useful.

Thanks for the highlights, Glyn. I admit my communication is not that good. You first link is wrong (trailing spaces in the URL, or something). In your second link, in the config example (serial -> emoncms), you may remove the socket listener part. It is just an example and not used, so it might be confusing.

I can't help on the RSGF/USB part. I use Raspbian and added the OEM softwares one by one.

I don't understand the error message.

2014-01-26 20:01:30,984 WARNING Misformed RX frame: ['10', '4', '263.56', '0.36']

This frame seems perfectly well formed to me.

I've been a bit dumb when writing the code, as I set several times the same error message. Frank, can you please modify oemgatewaylistener.py to ? Change each "Misformed RX frame" message to something different, like "Misformed RX frame 1", "Misformed RX frame 2", etc... So that we know at least where this happens.

Could it be a decimal separator issue ? Maybe the gateway doesn't understand the dot (.) separator. If the failed test is at line 73, you could add something like this, at least fir the sake of the test:

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewayliste...

# Log data
        self._log.info("Serial RX: " + f)

        received.replace(",", ".")

     
        # Get an array out of the space separated string
        received = f.strip().split(' ')

Now, I'm realizing we've got a much greater issue. Because I use int(val), all decimal parts are lost. This affects all persons using decimal numbers. Not the ones using the RFM2Pi module (99% of the users). I need to think this through...

https://github.com/Jerome-github/oem_gateway/issues/17

BTW, Glyn if you missed the serial -> emoncms feature, perhaps did you also miss the socket -> emoncms feature.

This allows to communicate with the gateway through a socket. There are examples showing how to do this. The client can be another application on the same host (for instance a software that can manage a specific sensor on the Pi) or on a distant host, provided the communication is routed to the Pi (beware of security issues if open to an external network, there is no password of API key, here).

There is even a listener that listens on a socket and repeats on the RFM2Pi, allowing the gateway to work the other way around, using the radio to command appliances.

Perhaps the config file is not documented enough.

Maybe we should provide a few different config files to chose amongst, instead of relying on the user to do proper modifications. I'm sure most users have the socket listener open but don't use it. Apparently, it is the default on the ready-to-go SD card:

http://openenergymonitor.org/emon/sites/default/files/Picture%202.jpg

In fact, int() does not remove the decimal pert, if fails with an exception, hence the error message...

Frank, please forget about my gibberish above and pull the latest version of the gateway, where this is hopefully fixed.

https://github.com/Jerome-github/oem_gateway

Please tell us how it goes after that.

(So I guess this affected 0% of the users up till now...)

Hi Joeroem & Glyn,

thanks for all your advice and sorry for having been offline the last two days. 

I have just changed and the  .conf file and also used the latest oemgatewaylistener.py. Thanks to Jerome it has no longer issues with decimal data, but it still fails to send.

The .conf looks like this now (do I have to use an API key for the local instance as well?):

# OemGateway Configuration file example
# Copy this as oemgateway.conf (or any custom name) and edit settings

# Each listener and each buffer has
# - a [[name]]: a unique string
# - a type: the name of the class it instantiates
# - a set of init_settings (depends on the type)
# - a set of runtime_settings (depends on the type)
# Both init_settings and runtime_settings sections must be defined,
# even if empty. Init settings are used at initialization, and runtime
# settings are refreshed on a regular basis.

# All lines beginning with a '#' are comments and can be safely removed.

####################
# Gateway settings #
####################
[gateway]
# loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
# see here : http://docs.python.org/2/library/logging.html
loglevel = DEBUG

#############
# Listeners #
#############
[listeners]

# This listener manages the serial input
[[Serial]]
    type = OemGatewaySerialListener
    [[[init_settings]]]
        com_port = /dev/ttyACM0
    [[[runtime_settings]]]

###########
# Buffers #
###########
[buffers]

# The two following buffers instantiate the same class,
# that formats the data for an emoncms instance.
# One sends the data to a local instance, the other one
# to a distant one.
# If active is set to False, the buffer neither records nor sends any data,
# but it holds unsent data until active becomes True.

[[emoncms_local]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = localhost
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        protocol = http://
        active = False
        path = /emoncms

[[emoncms_remote]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = emoncms.org
        apikey = nobodyneedsttoknow
        protocol = http://
        active = True
        path = /emoncms
 

The output says a lot more now (thanks to jerome's error messaging):

pi@raspberrypi ~/oem_gateway $ chmod +x oemgateway
pi@raspberrypi ~/oem_gateway $ ./oemgateway
2014-01-28 19:34:44,450 INFO Logging level set to DEBUG
2014-01-28 19:34:44,453 INFO Opening gateway...
2014-01-28 19:34:44,456 INFO Creating buffer emoncms_local
2014-01-28 19:34:44,460 INFO Creating buffer emoncms_remote
2014-01-28 19:34:44,471 INFO Creating listener Serial
2014-01-28 19:34:44,475 DEBUG Opening serial port: /dev/ttyACM0
2014-01-28 19:34:46,197 INFO Serial RX: 10
emonTX Shield CT123 Voltage example
2014-01-28 19:34:46,206 WARNING Misformed RX frame: ['10\nemonTX', 'Shield', 'CT123', 'Voltage', 'example']
2014-01-28 19:34:46,421 INFO Serial RX: OpenEnergyMonitor.org
2014-01-28 19:34:46,435 WARNING Misformed RX frame: ['OpenEnergyMonitor.org']
2014-01-28 19:34:46,707 INFO Serial RX: Adjusted by FH to work without RF
2014-01-28 19:34:46,716 WARNING Misformed RX frame: ['Adjusted', 'by', 'FH', 'to', 'work', 'without', 'RF']
2014-01-28 19:34:46,946 INFO Serial RX: 10 23427 428.71 69.23
2014-01-28 19:34:46,953 DEBUG Node: 10.0
2014-01-28 19:34:46,956 DEBUG Values: [23427.0, 428.71, 69.23]
2014-01-28 19:34:46,965 DEBUG Server emoncms.org/emoncms -> buffer data: [10.0, 23427.0, 428.71, 69.23], timestamp: 1390937686.96
2014-01-28 19:34:46,967 DEBUG Server emoncms.org/emoncms -> send data: [10.0, 23427.0, 428.71, 69.23], timestamp: 1390937686.96
2014-01-28 19:34:46,970 DEBUG Data string: &time=1390937686.96&node=10.0&json={1:23427.0,2:428.71,3:69.23}
2014-01-28 19:34:46,974 DEBUG URL string: http://emoncms.org/emoncms/input/post.json?apikey=blackedout&time=1390937686.96&node=10.0&json={1:23427.0,2:428.71,3:69.23}
2014-01-28 19:34:46,984 INFO Sending to emoncms.org/emoncms
2014-01-28 19:34:47,701 WARNING Send failure
2014-01-28 19:34:47,935 DEBUG Server emoncms.org/emoncms -> send data: [10.0, 23427.0, 428.71, 69.23], timestamp: 1390937686.96
2014-01-28 19:34:47,938 DEBUG Data string: &time=1390937686.96&node=10.0&json={1:23427.0,2:428.71,3:69.23}
2014-01-28 19:34:47,942 DEBUG URL string: 

Set active to True to send to local.

[[emoncms_local]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = localhost
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        protocol = http://
        active = False
        path = /emoncms

[[emoncms_remote]]
    type = OemGatewayEmoncmsBuffer
    [[[init_settings]]]
    [[[runtime_settings]]]
        domain = emoncms.org
        apikey = nobodyneedsttoknow
        protocol = http://
        active = True
        path = /emoncms

I deactivate by default since many people don't use local instance.

We should either provide config examples or strss out the fact that the example is just... an example and each parameter should be reviewed.

I'm glad the "decimal numbers fix" worked.

Hi Jerome,

I just did the change, however it neither seesm to be successful in seding to local nor to remote. Also I noticed that my voltage value has now increased by roughly factor 1.5??

The node is now a float as well instead of an int, is that ok?

Thanks.

2014-01-28 20:04:33,984 DEBUG Values: [23964.0, 435.68, 69.02]
2014-01-28 20:04:33,986 DEBUG Server localhost/emoncms -> buffer data: [10.0, 23964.0, 435.68, 69.02], timestamp: 1390939473.99
2014-01-28 20:04:33,993 DEBUG Server emoncms.org/emoncms -> buffer data: [10.0, 23964.0, 435.68, 69.02], timestamp: 1390939473.99
2014-01-28 20:04:33,995 DEBUG Server localhost/emoncms -> send data: [10.0, 23964.0, 435.68, 69.02], timestamp: 1390939473.99
2014-01-28 20:04:34,004 DEBUG Data string: &time=1390939473.99&node=10.0&json={1:23964.0,2:435.68,3:69.02}
2014-01-28 20:04:34,007 DEBUG URL string: http://localhost/emoncms/input/post.json?apikey=&time=1390939473.99&node=10.0&json={1:23964.0,2:435.68,3:69.02}
2014-01-28 20:04:34,024 INFO Sending to localhost/emoncms
2014-01-28 20:04:34,211 WARNING Couldn't send to server, URLError: [Errno 97] Address family not supported by protocol
2014-01-28 20:04:34,215 DEBUG Server emoncms.org/emoncms -> send data: [10.0, 23964.0, 435.68, 69.02], timestamp: 1390939473.99
2014-01-28 20:04:34,232 DEBUG Data string: &time=1390939473.99&node=10.0&json={1:23964.0,2:435.68,3:69.02}
2014-01-28 20:04:34,234 DEBUG URL string: http://emoncms.org/emoncms/input/post.json?apikey=blackedoutf&time=1390939473.99&node=10.0&json={1:23964.0,2:435.68,3:69.02}
2014-01-28 20:04:34,253 INFO Sending to emoncms.org/emoncms
2014-01-28 20:04:34,628 WARNING Send failure

Node being float is not elegant, I may change this. But it works for me here.

(You can check this kind of things by logging in your emoncms instance and entering the URL in your browser.)

I don't know about the voltage.

I don't know about the failures as well. What if you enter the same URL from your browser ?

Hi Guys,

i just figured it out. It was just the wrong path (/emoncms) in the .conf file. Now the remote send works!!! excellent, thanks for the help.

the local data still has issues (error 97: Address family not supported by protocol.) what do I need to do there?

2014-01-28 20:20:31,869 INFO Sending to emoncms.org
2014-01-28 20:20:32,181 DEBUG Send ok
2014-01-28 20:20:32,456 DEBUG Server localhost -> send data: [10.0, 23350.0, 428.76, 69.1], timestamp: 1390940431.64
2014-01-28 20:20:32,464 DEBUG Data string: &time=1390940431.64&node=10.0&json={1:23350.0,2:428.76,3:69.1}
2014-01-28 20:20:32,467 DEBUG URL string: http://localhost/input/post.json?apikey=&time=1390940431.64&node=10.0&json={1:23350.0,2:428.76,3:69.1}
2014-01-28 20:20:32,473 INFO Sending to localhost
2014-01-28 20:20:32,593 WARNING Couldn't send to server, URLError: [Errno 97] Address family not supported by protocol

Great! Thanks for helping debug this Jerome.

No, I was aware of the serial listener. That's pretty cool. How would this work in practice? Say for incidence I would like to send a data packet to a node. I know how to do this in minicom communicating directly with the RF12Pi, how would I do this remotely using the oem gateway?

I've also seen in some of the gateway code that there is an emoncms gui, how does this work? Does this use the same GUI that is already there under the raspberry Pi tab in emoncms? It might be good to use the on the rock solid serial gateway instead of the .conf file although I guess the file system would first need to be put into R/W mode in order to save the chance. 

It's fantastic the work you are doing developing the gateway, its maturing into a very tidy bit of python :-)

Hi Frank.

2014-01-28 20:20:32,593 WARNING Couldn't send to server, URLError: [Errno 97] Address family not supported by protocol

Never seen this, but I suspect the message is misleading.

Can you access your local instance from a browser on the Pi ? If you don't have a graphical session but only SSh access, you can try links/www-browser.

Did your enter your local API key ? I can't tell if you removed it from the log. I'm not 100% sure it is necessary when sending locally, I don't remember, but it'd be worth a try.

(Note that you don't have to reload the gateway when changing settings.)

Hi Glyn.

See the examples directory for a sample python code. Basically, chose a port number in the gateway config. Then, in the sample code, use the same port. And There you go. I tested this by sending the commands to change group/freq/nodeID and it worked.

If your use case is to send a packet depending on the output of another program, you can modify the program to write to the socket just like in the example. It is in python, but it shouldn't be more complicated in C, for instance. Most languages have libraries for this.

There is also a listener to get samples through a port. For instance if you have an application that manages a sensor.

You can open to the outside world if your router is configured so, but there is no auth, so anyone can send anything.

I designed the gateway starting from the raspberrypi module. The GUI is just the same. emoncms GUI writes parameters in DB and those are read by the gateway through the emoncms API (the gateway itself does not have acccess to the DB). The good point is the GUI. It is also its limitation : only parameters accessible from the GUI are usable. Currently, there is no way to use both in the same time. Besides, it is not really suited for the rock-solid since it needs the full emoncms stack with DB and all. And I like the text file, although threads like this one suggest that it can be confusing to new users.

I was thinking of dropping this feature, as I don't think it is used, and it is limiting, as compared to the text file.

I tried to design the gateway to be as generic and extensible as possible. Unfortunately, it makes the code more complicated than it could be. Perhaps some features are not used because scarce people know about them...

Nice I understand, just had a look at examples. Seems pretty straightforward. I agree, I like the text file config, it's simple and works. Maybe closer integration is needed for the emoncms GUI to work well, best talk to Trystan about that. 

Hi Jerome,

how do l obtain a Local Api key? Do l need to install emon CMS?

Frank

Franknfurther,

Yes the script is designed to write data into an emoncms database.
Therefore if you want to store the data locally using this script you will need to install emoncms. 

Once you have installed emoncms you can get the API key in the same way as you did for the emoncms.org (it will be a different number)

Frank, AFAIK, there are two flavors of ready-to-go card. This is because SD cards wear of if written intensely.

One is used read-only and does not contain emonCMS. It is meant to be used s a repeater.

The other one has emoncms and should be used with an external hard-drive.

If you chose the first, it might not be a great idea to install emonCMS on the SD card.

(I leave this for the others to explain, I run my own Raspbian + emonCMS and have never used ready-to-go SD cards. Perhaps should we find them great nicknames to clarify the ambiguity).

Hi guys, that makes sense, I get it. I am still on a steep learning curve here. 

I kind of mistakenly thought that the data could be buffered locally short-term (i the RAM) without having to install emoncms. Wishful thinking on my side.

Thanks for your help!

Glyn, do you have any idea why the rock-solid, ready to go SD card does not activate the Arduino? The Rasbian version works fine and the Arduino starts sending. Could it be that the serial port needs to be opened manually?

The data is buffered short-term in RAM already.

https://github.com/Jerome-github/oem_gateway/issues/6

The 1000 here

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewaybuffe...

is an arbitrarily hardcoded (and possibly very conservative) value. You can increase it.

Say you expect data strings of length 20 characters (Node + a few vals). Assume 8 bits per character (could be wrong). That is 20 bytes per strings. Current max of 1 000 would be 20 ok. Then 1 000 000 would mean 20 Mo, still safe.

I haven't been investigating this. Any feedback welcome.