Hi. I have created a script where I upload collected data from a csv file to Emoncms using the bulk function. It seems to be working, but in some cases when I am posting large amounts of bulk data in the same packet, it appears to be ignored by the server.
If I instead divide the data into smaller bulk packets and post them one by one it appears to be working, so there seems to be some limitation of how big bulk packets can be send.
Can anyone tell me if this is in fact the case, and if so, what is the max amount of bulk data I can send in one packet?
Re: Bulk data max upload size
I assume you are referring to emoncms.org, not a local instance?
There never used to be any restriction. Emonhub's bulk updating was restricted to a max batch size of 250 packets to throttle the impact of the server coming back on line following an outage and everyone trying to deliver their backload at once. I have tested the operation of that many times, but not recently, I have to admit
There have been some posting restrictions added to emoncms.org recently, but I didn't think this would affect bulk upload, since that is the purpose of the bulk upload - to deliver large amounts of data in an efficient way, to reduce server workload. However, you are not the first person to post on this issue recently, so there may well be a new restriction, intentional or not. (see "Bulk Upload Error")
Which bulk upload format are you trying?
emonhub uses the full timestamp, rather than a relative reference, for each packet and "sentat=" not "time=".
Paul
Re: Bulk data max upload size
Hi Paul
I am uploading to emoncms.org and I am using the sentat format. I have an installation on my own server also, but didn't test it there yet.
I have changed my program, so that it is only allowed to use a limited amount of time for bulk upload. Then it has to pause the upload in order to do other things in between, and then resume the bulk upload. This has solved the issue with not being able to upload everything at once.
With this change I am currently uploading approximately 100 "lines" with 12 values in each, and this seems to work.