Archived Forum

Hi.

There is nothing done currently regarding this, and no plan to my knowledge.

The alternative gateway I'm working on does this partially:

https://github.com/Jerome-github/oem_gateway

https://github.com/Jerome-github/oem_gateway/issues/6

Hi Jerome,

Looks good, I have a suggestion why don't you create a table in the database that records the raw data from the last good transmission to the next good transmission, and truncate when transmission s good, or  a threshold if no data is uploaded after 5 attempts start logging to the DB, always attempt to upload the oldest data first, once all is successful truncate the table,

The only thing is the public emoncms needs to understand the values received are historic first. I am not sure on how received data is stored, does the local emoncms transmit with the data with a timestamp for that row or is the timestamp created on the receiving cms when it receives data ?

I have 2 monitoring systems in place while I fine tune the emonTX, I use THEOWL Intuition-PV as a secondary, I must add though THEOWL can be anywhere from 5% to 20% so it's actually useless which is why I got the emonTX, however during the 2 day while my internet was off THEOWL buffered all data till it returned and when it did returned it uploaded the data, so I have no gap's in my feeds, which is pretty cool and why I prompted this discussion.

Unfortunately I live in a country where the ADSL service is really shocking and unreliable.

As Jerome mentioned the python gateway does this by storing the unsent values in RAM this is clever as it means you wont be using up SD card writes on your raspberrypi.

You could use the python gateway instead of raspberrypi_run.php to both direct data to you local emoncms installation and to forward the data with the buffering to emoncms.org or other remote server.

Hi Jerome

With this last outage of emoncms, the buffer worked like a charm the only thing is it only lasted for about 1 hour and outage was 21+ hours, 

it would be awesome if we can rectify or come up with a solution to ensuring that we loose zero data over long outages.

An idea I had was to send the data to a proxy service, either locally to another PC or on the internet, this proxy service would then attempt to upload the data once the connection is re established. Or on failure keep details in memory and once buffer size has reached 1000 set system to ipe-rw and flush the buffer to a file then set the system to ipe-ro again, and continue doing this until such time a connection is reestablished, Once the connection is resolved, flush the file to the live servers and delete the file. ?

Shot

Hi.

Normally, you should have lost everything except the values from the end of the interruption period. In other words, I assume you lost all values except for the last hour of the interruption. If not, please tell me, it would mean I missed something.

The easy and short-term solution is to increase the (hardcoded) buffer size here:

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewaybuffe...

1000 stands for 1000 requests, regardless of the number of samples in the request. This is totally arbitrary, I didn't think of what kind of RAM usage it represents. Perhaps 100 000 is totally feasible. I'd need to look into python's string representation.

We could add this as a parameter, but I'm not sure it is so useful.

I didn't think of sending to another server, as the use case I was thinking of was rather a local network issue, but why not ?

The file buffer was my next option. See here:

https://github.com/Jerome-github/oem_gateway/issues/6

Yes, this could mean having to "ipe-rw", which means give the user that runs the gateway the privilege to do so.

I may not follow every thread on the forum, so feel free to comment on Github about this issue to be sure I'm notified.

Thanks Jerome,

I shall use githib going forward, you implementation works 100%, I am purely looking to see if we can't buffer to loger periods or idefinatly till connectivity is restored. I'll take a look at the hard buffer and see how this effects the sytem if we increase it to 100 000.

Thanks again.

I think 100 000 is probably safe for a 24h break. Hard to tell as it depends on the data and all, but a quick and conservative estimation makes me think it should be fine.

Actually, it would be interesting to use the resource module to see how the memory usage grows when the link is down (easy to simulate: just change the target URL to a dummy one).

http://docs.python.org/2/library/resource.html

http://pythonforbiologists.com/index.php/measuring-memory-usage-in-python/

I have set the buffer to 100 000, I created a fake enter to emoncms in my hosts file to simulate a network issue,

After 3 hours I hashed the new entry out and it seemed fine, data is being put back into emoncms, the only issue I see here is that it's take a long time, eg. to buffer out the 3 hours of downtime to emoncms.org took about 6 hours to complete. 

Memory usage on the Pi was negligible. I am going to test again, this time I'll try 12 hours.

My only concern is the time it takes to catch up. basically meaning I can't have a power outage for this period. but that said it's still better than nothing.