Archived Forum

Hello Frank, sounds like a great development, yes this is one of the intended features of oem_gateway/emonhub. I would take a look at the new emonhub development repository and maybe open an issue there, can you share your code there too?

https://github.com/emonhub/

Yes your right about the flush method, we've been working on the next version of emonhub, which may have a send all version of flush included. You might be interested in some of the wider discussions going on there about development direction of oem_gateway/emonhub

Hi Trystan, I will gladly share the code of the additional listeners once I have figured out how to do this on github. However, I have noticed that I have been working with an older/different version on https://github.com/Jerome-github/oem_gateway that I downloaded less than a week ago.

Anyway, the change I made to the flush method was simply replacing the first IF statement with a WHILE statement. So basically data is sent from the buffer until the buffer is cleared. Works for me for now.

Another change I had to make was to change the time.sleep() period in the oemgateway.run() loop to 10 seconds. This is because with the Modbus meter you can get the data from the registers at any time, rather than having to wait for it. Might it be an idea to have this as a user-defined global parameter stored in the config file?

I'm still using a copy off oem gateway myself, I guess "officially" the OEM gateway version you are using from Jerome-github is still the current one.

Please start an "issue" to discuss your listener modifications, we are extremely keen to make emonHub v1 as "future-proof" as we can by ensuring the initial structure is such that can be developed easily. we have pondered on potential future requirements but a real life example would be great.

If the changes are relatively small you can just "copy & paste" those lines with comments into an issue at emonHub even though it's oem gateway code, 

https://github.com/emonhub/emonhub/issues?direction=desc&sort=updated&state=open

Paul

Hi Frank.

Yes, the data is flushed one line at a time. But the main loop is supposed to never be stuck, so it is not an issue, rather a design feature.

I don't want to flush all the data at once, because if there is too much data in the buffer, the other tasks are not performed (the listeners are not called) as long as everything has not been send. Worse if network is down.

Could it be your problem is due to one of your custom listeners working in blocking mode ? I mean, if one of your listeners has a read function that gets stuck until something comes in, then this won't work. This is wrong because if nothing ever comes in, the program is stuck. All other operations are delayed.

If this is the case and you don't know how to do otherwise, you may simulate multi-threading by creating another application that would send the data through a socket to the SocketListener.

I've been playing with Modbus recently and I'm interested in your listeners. There are python libraries we could use to add RTU and TCP Modbus features to the Gateway. The difficulty in making a generic code might be the multiplicity of possible configurations. If it takes as many config lines as lines of code, users may prefer using their own code than figuring how the Listener works. Just a quick thought. I may be wrong.

To share your code, just create an account, clone the OEM Gateway repository in githug, clone your clone on your machine, apply your modifications, then push to your github clone.

I hope this helps.

Hi Jerome & Paul,

sorry for the delay in responding as I have been travelling. Thanks for your comments and interest.

If you let me know the github command that I have to use then I will upload the code to github. I am not experienced with networked programming and github and the like.

Regarding Modbus: I have been using the MinimalModbus library. The background is that while I am the proud owner of an official smart meter from the local Dutch grid company with serial interface, this smart meter is actually quite dumb because it only gives me the balance between household consumption and solar production. However I wanted the production and the consumption to be shown separately, therefore I invested in this DIN-Rail Modbus RS485 meter device (Euro 35) and installed it next to the smart meter. One distinct difference of the Modbus compared to the normal serial interface is that I have to request the data from the registers of the meter, whereas the grid company meter will send me a serial telegram by itself once about every 10 seconds. The implication is that I cannot have a sleep time of only 0.2 seconds in the main loop, otherwise I shall end up with 300 datapoints per minute from the modbus meter, while I am getting 6 datapoints per minute from the smart meter.

So in effect I am getting the production/consumption balance in Watts from the smart meter and also get the pure production in Watts from the aforementioned meter, and then deduct the production from the balance in order to get the pure consumption. All calculations are currently being done on emonCMS.org

Next thing I am planning to do is to make a SerialListener for a friend of mine who lives in Germany and has 2 separate grid company meters each with a serial interface, one for solar production and one for consumption. Furthermore, I want to make a buffer that uploads the data to pvoutput.org (in addition to emonCMS). This is because I have uploading the solar data for the past 2 years to pvoutout.org and I like a few features there too.

Generally I really like the overall structure of the gateway, especially the logic of creating buffers and listeners. If I may make a few suggestions in terms of architecture that would be useful from my point of view:

• would it be possible to create two independent polling/cycling instances within the same __main__ instance, one for polling the listeners at a pre-defined interval rate set by a global parameter (set through config file), and another one for cycling through all the buffers, with a possibly different interval rate. The reason for this is that I am getting data in different ways at different intervals, and I want to upload to several servers at different intervals.
• every couple of days my script gives an error and stops (due to the serial interface listener getting no or strange data). In this case I would like the Raspberry to catch this error and reboot. Can this be done and how? Will a command like os.system("reboot") be appropriate to use? Ultimately I want the gateway to run with a 100%  up time.
• The possibility to perform some pre-processing of data before it is being uploaded to a server. However, I am not yet exactly sure what this pre-processing will entail.

I look forward to your reactions.

Regards,

Frank

PS: I have just added the code of the two listeners to an issue on github/emonhub

Hi Frank, thanks for posting your code on emonhub's github. for us to take a look at, 

Good suggestion to have multiple polling intervals, I intend to explore the possibilities of an I²C listener at some point. my thoughts were to use a countdown timer value within the Listener that gets checked and decreased each loop of the main rather than changing the loop speed.

Currently emonHub's intention is to deliver the data as soon as possible following receipt, the data is tagged with the time upon receipt and when sending is delayed by network or server issues the time code is used by emonCMS, (I'm not familiar with PVoutput.org but would expect something similar) So by controlling when the data is logged by emonHub could "align" different sources maybe. 

With regards to  error free running, it is important to us that emonHub is stable and reliable, Since the OEM gateway is already extremely reliable, emonHub is lucky enough to have a solid foundation to build on.  So I would hope that we could find and eliminate the cause of the failures to avoid the need to reboot.

Have you had these serial problems since running both listeners? or would it be worth trying each one independently for a few days  to establish if it's one or the other that triggers the issue, or if (more likely) it's only when they are both running that causes issues?

The jury is still out on pre-processing as you can probably tell by the discussions on github, Up until recently the aim was to faithfully forward data without any processing beyond recombining the byte-pairs into ints from the RFM2Pi, since considering the nodes module, which deals in byte-size data only the topic of processing the data into real values has opened a large can of worms. It is my own opinion that a Listener should process received data enough to present real values for  forwarding and any additional processing, scaling or calculations should be done by emonCMS etc. But there is no reason why a dispatcher shouldn't  perform some custom processing if a certain data type is required by the target eg bytes by the nodes module.

Excellent discussion points :-)

Paul

Hi Paul,

I had issues with both listeners. The ones from the Modbus listener were because the metering device I use is at intermittent intervals not ready to be read; in these cases I have chosen to change the code to ignore the resulting serial interface error. Since then it has  been working well. The serial interface reader sometimes had issues with the parser of the telegram, however, is has been running faultlessly for the past 5 days now. The gateway really has become a nice little datapump... however, I think it would be nice if there was the option of a reboot, just in case of an error. The hub really is expected to be up 24/7, which we will be difficult as the range of meter devices is being expanded.

Instead of a countdown timer in the run loop we could also just store the time of the latest received read from the listener, and then use the time difference to see if enough time has passed for the next read to be received. The same for the buffer, we store the time of the last upload, and see if its time to upload again. If we however want to do this for both the receipt of data and the upload of data, we may want to do some averaging of the uploaded data.  

Cheers,

Frank

Hi Frank.

Using the actual time read with an interval would be good, the rfm2pi listener has similar code for updating the clock on a GLCD so it wouldn't be difficult to implement and that would then be independent of loop & sleep times, giving a more controlled and regular read interval.

Not so sure about the sending though, as sending doesn't always guarantee receipt  so a slow network or 1 server down will have knock on effects, there is also bulk upload to contend with plus we are planning a persistent buffer in a future version that will cope with longer server outages etc, but who knows ?

The expectation of having the hub running 24/7 is the reason for not wanting to reboot the Pi. the buffered data would be lost, as would any incoming data during the booting phase. My OEM gateway runs for quite literally months without any hiccups. the only time it is not running flawlessly is when there is a powercut or I'm experimenting/changing something, I don't even bother checking it anymore, I'm that confident.

Whilst I really like the idea of the custom listeners I honestly think you maybe better writing a little python script  (as also suggested by Jerome) That script could listen for the 'telegram' and poll the other meter (either on an interval or triggered by the telegram)' the script can forward  a complete dataset to the hub via a socket. this has the advantages of "aligning" the 2 data sets, any stalling would not effect the hub and if prone to glitches then the script could be restarted using cron every hour/day or week without restarting hub or pi. 

Paul