Archived Forum

Hi JS

Good to see the socket interfacer getting some exercise, I think it's a much under used feature. 

I'm not an expert in this area, but wouldn't keeping the connection open block any other devices? Is there a reason not to close the connection? or rather is there a major benefit to keeping it open? I'm not particularly opposed to the concept, other than I think the original idea was to have a "rfm2pi type listener" for "depositing" any local and network packets. in the past I have had several scripts posting to a single port on my own set-up.

Paul

Hello Paul,

There should be no issue keeping the connection opened. And even though, if the client opens the socket, it should be responsible for closing it : it knows when it has nothing left to send !

I don't think it would prevent any other connection from another device. I'll test that. Also, If the server is forcibly closing the connection, client may have been in the process of sending data. So data is lost. I think client side connection will remain for 5 minutes in "CLOSE_WAIT" which can be a problem if you open a socket every second : you'll get 300 CLOSE_WAIT connection. But I'll check that also.

So, if you post data fast enough, I tend to think that the overhead of opening a socket each time is worse that keeping a connection alive. But to be honest... I don't know what "fast enough" would be.

JS

I look forward to the results of your testing.

That code was inherited from oemgateway and was originally written  two and  a half years ago, I'm not opposed to changing it but I would like to be 110% sure it's the right thing to do before I commit, ideally yes the client should know when it's finished and sign off, but where does that leave emonhub if it doesn't, emonhub must be able to continue functioning correctly even if a client crashes, stalls or just isn't written well enough to close the connection.

I would need to do some swatting up to make the call myself so hopefully some more knowledgable devs will chip in here. My main concerns are that emonhub is resilient to client error and that the connection is non-blocking for other nodes to post to the same port.

Otherwise maybe an optional setting to make it an exclusive "kept open" connection could be introduced ?

Paul

Hello again,

I modified my script so that I open a new socket each time I need to send data, and it works well.

I checked the TCP connection is properly closed on both side, even though the client is not closing the connection. On emonpi side, the connection is cleared almost right away. On my sensor side (I am testing this with an Arduino Yùn), I have some socket in "TIME_WAIT" : ~55 if I post every second, no more than ~20 if I am posting every 3 seconds.

If I also close the socket in my script, I get the very same behavior.

I also tried adding a conn.shutdown in EmonHubSocketInterfacer.py but no change.

Anyway, I guess unless you want to post data at a very high rate (I would say more that 1/sec), opening and closing the socket each time seems ok. So having a persistent connection may not be that usefull in the end... Sorry for that. And in addition, forget I said it should not be a problem... it may require quite a bit of work  to make it accept more that 1 connection at a time.

Finally, I got it to work properly just by adding the missing imports into EmonHubSocketInterfacer.py, that is

import socket
import select
import Cargo

changing

        # create a new cargo
        c = new_cargo(rawdata=f)

to

        # create a new cargo
        c = Cargo.new_cargo(rawdata=f)

and by opening (and closing) the socket at each posting.

Here is my test script (run on my Yùn)

import time
import sys
import datetime
import socket

# Interval between sensor reading
readInterval=1   # in seconds

# emoncms properties
emoncmsHost = "myip"
emoncmsSocketPort = 20100

while True:
    emoncmsSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    try:
        emoncmsSocket.connect((emoncmsHost,emoncmsSocketPort))
    except socket.error as msg:
        print str(msg)
        time.sleep(readInterval)
        continue

    payload="31 22.1 2.3 36 -18"
    print "Sending " + payload
    payload = payload + "\r\n"

    try:
        emoncmsSocket.send(payload)
    except socket.error as msg:
        print str(msg)
        
    emoncmsSocket.shutdown(socket.SHUT_RDWR)
    emoncmsSocket.close()

    time.sleep(readInterval)

I'm glad you found it works ok for you now. I was just reading up on it myself and it seems to be very much the "standard" way going by the general examples. It was still useful for me to better understand the code, thanks for testing and sharing your findings too.

Paul

I've submitted a pull request for the missing imports to the OEM "emon-pi" emonhub repo, (this isn't an issue in the original emonhub). Could you please confirm the edits as I haven't tested them personally.

Added missing imports

Paul

Pull request has been merged, please could you confirm this works for you https://github.com/openenergymonitor/emonhub/commit/2a45af573f0a381bf50942a8c5552359455050d3

​Thanks.

Hello,

I just updated my emonpi, and it works for me !

Thank you for your help.

JS.

This is a great solution. I spent some time today to test it with a RPi 2 and the 22 dec 2015 image. You can configure emoncms, mqtt, openhab (what else?) without connecting anything physically to your serial port. 

With a bit of imagination, you can create mathematical functions and display them in your feeds...

Good work, thanks

Walter

Paul,

Can you point folks to the documentation for the socket interface?

I'm assuming if you are using the original emonhub, i.e. not the one on the emonPi then you don't need the pubchannels line in [[[runtimesettings]]]

Thanks

Simon

If anyone is interested in doing this from an esp8266 system, then see the attached file. I couldn't find a decent example or any instructions as to how to do this - and yes I searched.....

These are the instructions for using the code that I put at the top of the file

// Replace the ssid and password for your wifi router
// Also replace the ip address of your local emoncms server
// and if necessary the port number you want to use.
// You can also set the node id , it's set to 15 in this example

// you may need to use 'sudo ufw allow 20100/tcp' to allow the packets through the firewall

// Tested sending to a PiB with Jessie Lite and emoncms 9.3 and the original emonhub installed through the Raspberry Pi
// installation instructions at https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi

//      /etc/emonhub/emonhub.conf   needs the following section

/*  
# This interfacer manages sockets
[[Socket]]
   Type = EmonHubSocketInterfacer
    [[[init_settings]]]
        port_nb = 20100
    [[[runtimesettings]]]
        pubchannels = emonCMS, emonCMSlocal

In my case emonCMS refers to the .org copy and emonCMSlocal refers to the local emoncms. 
Change these names to match what you have called your reporters. 
*/

I'm sure the code could be improved but it works for me (it's a cut down version of the full code I'm using to manage my boiler and heat bank on an esp03)

yes documentation is even thinner than usual on this (socketinterfacers) as they never really took off, they currently only tend to only get used in custom scenario's and each user has thier own way of doing things and specific settings or code.

I will try and write something up but finding the time is not easy, I really must get emonhub v2.0 off the ground too.

I will at least get a few notes together with a range of links, the other trouble being the various past versions and settings may not all apply now, for example Simon yes the pubsub channels are only required on emonPi (as you've discovered) .

A note on the channels (not specific to socketinterfacers though) if the emonpi variant has the same intended operation as the experimental branch it's based on then you should only need the one "emoncms" channel and both your local and remote emoncms's could subscribe to that one channel, this means you do not need to list every target in every interfacer. The only time you would "need" to do it the way you have is if say, you had 5 input interfacers and only 4 of those went to one emoncms and 5 to the other for example.

The "experimental" branch also has default channel settings for each interfacer so only the more exotic set-ups and braver users had to tackle setting the channels, meaning the settings will not even be visible on a totally default setup.

Again I need to do some investigating into what did get carried over to the emonpi variant as the "experimental" socket interfacer has provision to use an apikey for use outside the LAN and also can accept a supplied timestamp in the payload rather than apply one on arrival, this last feature is very useful if trying to post a list of data, a script can loop through back-up data or downloaded data etc.

Paul

Hi Paul,

I actually left the pubchannels in but it doesn't seem to affect anything which is good. I'm pretty sure the data gets to both the .org version and my local version.

I'm not sure I'm even going to use this as I already have some code in my full version of the esp s/w for using the http interface.

As for whether folks would use this from outside the network I'm not sure, I wouldn't as the esp controller is visible from outside and I can see the emoncms data from the .org version which is a duplicate.

I guess for V 2.0 what you'd need to do would be to collect requirements from the forum. I wonder if you asked the question whether you needed both the socket interface or the http interface then they would say only one is needed. But then it would have to be http into emonhub not into emoncms.

One thought on the documentation, it might be a good idea to start with to add to the emonhub/emonhub space on github a note about the history and different versions, this would at least clear up any confusion. Took me a while to realise that the emonPi version was different, even though I'd probably read the posts during the development. 

Anyway, hope the file I posted might help other folks get the answer more quickly. It was a good exercise for me to work out how things worked.

Simon

Ahh yes, I have just come back to say about the channels as I have just posted on your other thread and the penny dropped you are using v1.2 emonhub.

The big advantage of using a socket interfacer to post to emonhub locally and then emonhub post to emoncms (local or remote) is each of the devices only needs configuring to find emonhub and then you can post to as many places as you like and change it freely without having to upload a new sketch. Plus if using v1.2 and a remote instance of emoncms the data is buffered in emonhub where as directly send data will be lost if there is a server or network issue beyond the lan.

Paul

Agree about moving everything through emonhub, it certainly makes things easier than replicating the posts for every system you want the data to reach as well as the buffering.

Simon