Archived Forum

Its my understanding that those settings are just for writing to the database.  So try this:

[[emoncms_local]]
  type = OemGatewayEmoncmsBuffer
[[[init_settings]]]
[[[runtime_settings]]]
  domain = 192.168.1.77
  apikey = <READ AND WRITE API KEY FROM Input API help on YOUR LOCAL EMONCMS INSTANCE>
  protocol = http://
  active = True
  path = /emoncms

[[emoncms_remote]]
  type = OemGatewayEmoncmsBuffer
[[[init_settings]]]
[[[runtime_settings]]]
  domain = emoncms.org
  apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  protocol = http://
  active = False
  path =

I reserve emoncms_remote for emoncms.org, so if you are not using emoncms.org set active to false.

Thanks

I have done that and started oemgateway.

I am not seeing any inputs so I must have something wrong. Where does the log file end up?

Regards

Ian

Are you sure that the data is getting to the oemgateway script?

Oemgateway either gets data from a RFM12Pi on the same computer (via serial port) or via listening on a socket.

You can setup the log file location from the command line otherwise it is directed to sys.stderr

Have a look at the source code https://github.com/Jerome-github/oem_gateway/blob/master/oemgateway approx line180.

Hi Ian, Did you get this sorted? 

Having tried this last night I can confirm the oemgateway forwarder used in the RO image will forward both socket and serial packets, to emoncms locally on same machine, same network or remotely hosted (in fact it can do all 3 at once if you add another buffer in oemgateway.conf) so these details should work for you.

[[emoncms_local]]
  type = OemGatewayEmoncmsBuffer
[[[init_settings]]]
[[[runtime_settings]]]
  domain = IP.OF.LIVE.PI
  apikey = API KEY OF LIVE EMONCMS
  protocol = http://
  active = True
  path = /emoncms

[[emoncms_remote]]
  type = OemGatewayEmoncmsBuffer
[[[init_settings]]]
[[[runtime_settings]]]
  domain = IP.OF.TEST.PI
  apikey = API KEY OF TEST EMONCMS
  protocol = http://
  active = True
  path = /emoncms

Getting oemgateway to install and run was a bit fiddly due to some permission issues, but once it started everything worked really well

Paul

Hi

Thanks. Not working yet because I think I had not configured port number correctly.

I set that to 80 which is what I thought it is but get an error:-

2014-03-28 13:04:46,954 INFO Logging level set to DEBUG
2014-03-28 13:04:46,958 INFO Opening gateway...
2014-03-28 13:04:46,963 INFO Creating buffer emoncms_local
2014-03-28 13:04:46,970 INFO Creating buffer emoncms_remote
2014-03-28 13:04:46,975 INFO Creating listener RFM2Pi
2014-03-28 13:04:46,981 DEBUG Opening serial port: /dev/ttyAMA0
2014-03-28 13:04:46,988 INFO Setting send time interval to 0
2014-03-28 13:04:46,992 INFO Setting RFM2Pi | frequency: 4
2014-03-28 13:04:47,997 INFO Setting RFM2Pi | sgroup: 210
2014-03-28 13:04:49,002 INFO Setting RFM2Pi | baseid: 15
2014-03-28 13:04:50,006 INFO Creating listener Socket
2014-03-28 13:04:50,011 DEBUG Opening socket on port 80
2014-03-28 13:04:50,016 ERROR [Errno 13] Permission denied
2014-03-28 13:04:50,021 ERROR Could not open port 80
^C2014-03-28 13:04:58,990 DEBUG SIGINT received.
2014-03-28 13:04:58,994 DEBUG Closing serial port.
2014-03-28 13:04:58,997 INFO Exiting gateway...

Still investigating!

I will change config to your example.

Regards

Ian

the default port oemgateway listens to is port 50011, it may be better to change it back and match it at the other (emontx) end since you are not sending direct to emoncms now.

Paul

Hi Paul

I am a bit lost on port addressing(but learning!).

My Live PI is on a fixed IP internet connection. My emontx is located remotely and sends data over the internet.

I have port forwarding set up on the router for the emoncms as follows.

# Service Name   External Start Port   External End Port   Internal Start Port   Internal End Port  
4 Emon                  80                               80                              80                             80                            

Internal IP address

192.168.1.9

I changed the oemgateway to listen to port 5011 and it starts without an error message. I then changed the internal port forwarding to match 50011.

This has an immediate effect on the oemgateway  which is giving warnings like this:-

2014-03-28 16:04:43,630 WARNING Misformed RX frame: ['GET', '/emoncms/input/post.json?apikey=********06d069745bfe8dac70040261&node=28&csv=0,0', 'HTTP/1.0']
2014-03-28 16:04:43,864 INFO Serial RX: Host:
2014-03-28 16:04:43,869 WARNING Misformed RX frame: ['Host:']
2014-03-28 16:04:44,075 INFO Serial RX: Accept: text/html
2014-03-28 16:04:44,079 WARNING Misformed RX frame: ['Accept:', 'text/html']
2014-03-28 16:04:44,283 INFO Serial RX:
2014-03-28 16:04:44,287 WARNING Misformed RX frame: ['']

Sadly it also has the effect that I can no longer log in to my Live PI over the internet. I can log in to the live PI using the network address but the inputs are not updating. So I am not sure what to do next. It would seem I need to give oemgateway access to port 80 but I don't know how.

Regards

Ian

[Edited: APIkey fudged - RW]

I'm not familiar with port forwarding but I would of thought you could leave the live pi exactly as it was and change emontx to send to LIVE.PI.IP port 50011 and then enter whatever address your emontx was originally sending to in the oemgateway.conf in one buffer along with the test pi address in the other buffer. that should retain your previous access as you are just putting oemgateway in between emontx and live pi.

Paul

Hi Paul

Not so easy. I am using a nanodeRF  located at different premises and I don't really want to lose data collection.

My emontx data is all collected by the nanode.

My Live PI server  is working well and has done for many months. I was setting up a test slave PI to evaluate emoncms V8 before I upgrade and to help with the testing of V8 if I can.

The nanode is sending data to a static address set by this line of code:-

static byte hisip[] = { 85,***,188,38 };

Out of interest does anyone know if the port number can just be added to this in the normal format:-

static byte hisip[] = { 85,***,188,38:50011 };

Regards

Ian

[Edit: IP address fudged - RW]

Hi Ian - You should maybe mask some digits off in your last post !

I'm guessing this isn't going to work as even if it did send to that ip/port it will clash with existing port forwarding. It seems from the previous prints the oemgateway isn't able to decode the complex url from nanode when using port 50011 so giving it access to port 80 probably won't resolve the issue. You maybe able to get around it by changing url template at the nanode but I think this would then be insecure as it appears the oemgateway requires no authentication, in fact it will add its own API key validating everything it receives. 

So probably oemgateway will need modifying for this to work

Paul

You can't do "static byte hisip[] = { 85,***,188,38:50011 };"  because that is trying to fill an array, and the syntax is illegal - 38:50011 is not an integer. You'll need to delve very deeply into the Ethercard library to find out how to add the port (always assuming that you can). So while I can tell you why it won't work, I'm sorry but I can't tell you how to fix it.

Hi

Many thanks.

I will forget this for now as it was only for testing purposes. Robert -  thanks for changing the IP address. It may have been interesting  to see how well my router firewall works as there are only 4 ports exposed and they are all pointing to uncritical web pages. However probably best not to find out is does not work the hard way.

Lesson learned, don't cut and paste without thinking.

Regards

Ian

Hi.

Not sure I get the point, but I think there is a misunderstanding.

The gateway is a repeater. It gets data from the emonTX (generally locally) and sends it to one or several emoncms servers. Those servers can be anywhere, even on the same machine, in which case the domain is localhost. The name "emoncms_local" is just a convenient name, it can be changed and any number of "emoncms buffers" can be created ion the gateway. This config file is just an example. Anything can be changed.

When the gateway sends data to an emoncms server, it formats it as a http request. Which means it sends it to that server on the 80 port (I suppose) or the https one, if configured so and if this works on server side (never checked). Apache (or nginx, or whatever) on the server side passes it to the emoncms instance on that server, and the rest is history.

The socket listener of the gateway (the one that opens a port) is very specific. It opens a new port (I chose 50xxx to be sure it was a free port, since it is not currently reserved, you can pick another open one if it is already open) and reads it as you would read a COM port. It expects raw data of the form

NodeID val1 val2 val3

See the example in the gateway directory. There is no "mainstream" application of this. It is here to help anyone wanting to send data from a custom application like a sensor reader.

It is not clear to me what you want to do. If you want to receive data on an emoncms instance on one Pi, then forward it to another, well, this is not possible. emoncms does not do this. And the gateway is not concerned, since the data is sent to emoncms, not to it, on the first Pi.

The solution is to send to both Pis from the emonTX. There was a thread about this. I don't think there is another way, unless such feature was added in emoncms since then (which I doubt).

That is if my assumption is correct that you're using a TX with an ethernet port and formatting the data for emoncms. By "formatting" I mean that the TX creates a http request with the API key and all.

The gateway is meant to get raw data and create the http request. If your TX is equipped with network interface and does that, you just don't need the gateway.

You could use the gateway as a repeater if you sent it raw data, but I don't see the point (there could be good reasons in specific cases, and testing is a specific case). However, it would mean open a port, route it, and it is not really meant to be done : there is no authentication such as API key, so currently it should be used on a local network.

I hope this is not too confusing. Please ask for clarification if needed.

Hi Jérôme

I have since learned that oemgateway does not do what I was trying to do.

I have a nanodeRF sending data over the internet to a raspberry PI (without the RFM12Pi expansion board) acting as an emoncms server at a fixed IP address. I have recently set up a  second PI again without RFM12Pi expansion board as a second emoncms server. I thought the oemgateway could be used to forward the data from the 1st Pi server to the 2nd PI server.

I now understand that this is not how oemgateway works.

This is not important as I was only trying to do this as a way of testing the latest version 8 of emoncms before I actually upgrade.

Many thanks for replying

Regards

Ian

OK.

I think you could modify your Nanode code to send to both servers, if they are both reachable from it.

I tried that some time ago, I could send the data to the two servers, but handling the acknowledgements was difficult as I couldn't identify the sender; and I gave up (mainly because the Nanode was losing favour and everybody seemed to switching to using RPi's).

Would it be possible to add a url listener to the oemgateway with the listeners incoming apikey set within it's entry in oemgateway.conf? So if the received url string contains the correct api then add the basic data values to the existing buffer system. I know it won't be quite as easy as that but it would be a very valuable addition to the oemgateway.

This would not only resolve this particular need but also expand on the currently limited option of only being able to forward rfm2pi data from emoncms.

Dave (aka Schism) for instance is currently able to forward from one emoncms to another via the rfm2pi module interface but if he had non rfm inputs they would be excluded.

I on the other hand can use oemgateway to feed 2 local installs and emoncms.org, forwarding all input types to all emoncms servers as long as the inputs are created locally.

With what's currently available the easiest solution for Ian would may be to replace the nanode rf with a Pi running the rocksolid ro version of oemgateway, sending to 2 emoncms servers using the same ip but different paths eg /emoncmslive and emoncmstest and resolve the addresses locally, but I'm guessing the test emoncms may not be important enough for the extra expense/hassle.

So how difficult would it be to add a url listener (assuming it can be done of course) ? 

Paul

Hi Paul

Your suggestion would also provide an up to the minute automatic backup!

Regards

Ian

I'm not convinced it is a good idea. I mean, it may be a twisted way of doing something that could be done from emoncms.

In fact, after a quick thought, I think it's more a job for emoncms.

Maybe you can open an enhancement request on emoncms for this. You can also open one on OEM gateway. I don't plan on doing this soon, but it will be here for here record. And maybe the one on emoncms will open a fertile discussion.

Or maybe you'll convince me that the gateway is a place for that.

It strikes me it's more like RAID than a backup?

Ian - yes it would, in fact much better than just a data back-up because if a server went down either due to a fault or for intentional maintenance/upgrade and was back-up again before the oemgateway buffer duration was reached (currently about 1.5hrs for me) there would be no loss of data on that server once it was back online and caught up with the buffered backlog.(assuming of course the oemgateway hasn't gone down with it ie same machine or same power supply failure)

Robert - I hadn't thought of it that way but yes like a mirror array raid except a complete shadow server possibly even located somewhere different eg emoncms.org that if a server went down you could just log-in to the other and carry on. not that anything we're monitoring is quite that important but it's a nice incidental feature.

Jérôme - Basically having decided the url forwarding was definitely a useful feature for the emoncms project, based mainly on the usefulness of the limited rfm2pi only forwarding part of emoncms and the forward all local data functions of the oemgateway, My thought process was simply why exclude standalone RO oemgateway from using it? if added to oemgateway both standalone and emoncms servers reap the benefit in one go as I believe the oemgateway is being adopted as the standard gateway to emoncms.

I guess either place would suffice but to me the oemgateway is really the traffic hub for the system and as this function is explicitly about routing data,, it just seems the most logical and flexible option,

Paul

Anyone, please follow discussion on this specific point in the github issue, if you want to be sure I'm notified.