While developing the new application specific dashboards recently, I found that I needed to be able to request data from the server for multiple feeds with the exact same timestamps so that you could directly run calculations such as the energy balance or 'virtual store' calculation using multiple feeds.
When a request was made to the emoncms data request api which looks like this:
http://emoncms.org/feed/data.json?id=71667&start=1426031940000&end=1426064520000&interval=30
The datapoints returned where not guaranteed to be aligned to the request start time and interval making this cross feed comparison difficult.
Yesterday's update makes it much easier to do this kind of request, if you call
http://emoncms.org/feed/data.json?id=71667&start=1426031940000&end=1426064520000&interval=30&skipmissing=0&limitinterval=0
for multiple feeds you will be guaranteed to get a json output with the same number of datapoints and the same timestamps.
I've added a new default data viewing graph that loads when you click on the eye on the feeds page which makes it easier to explore this new data request API. Its possible
The new paramaters are skipmissing and limitinterval.
skipmissing = 0 will return null if there is no value at the requested timestamp, skipmissing = 1 will skip the datapoint entirely returning an output similar to the older data request api.
limitinterval =1 will limit the minimum request interval to the interval of the feed being used so that if you have a 10s phpfina feed it wont return data at less than 10s.
CSV Output, theres also a new CSV output box in the data viewer graph which makes export to a spreadsheet easier to do.
There are still a few teething issues with the new request method that need ironing out, but hopefully the result will be a more useful data request method that makes multiple feed post processing easier.
kWh zoomer bug fix - as part of this update I have also fixed the kwh zoomer visualisation which wasnt displaying the years correctly.
Testing, if you wish to test this on your own emoncms install you can find these features in the newgetdatamethod branch here to be merged into master once all related bugs are sorted https://github.com/emoncms/emoncms/tree/newgetdatamethod you will also need this to run the latest application specific dashboards module https://github.com/emoncms/development/tree/master/Modules/app
Re: New emoncms get data request method and data viewer
I'd like to try this out. Would you mind posting a pointer to some data backup/restore instructions please?
Thanks, John.
Re: New emoncms get data request method and data viewer
Have you checked the user guide? - https://github.com/emoncms/emoncms/blob/master/docs/Backup.md Paul
Re: New emoncms get data request method and data viewer
Hi there,
I downloaded newgetdatamethod with git clone (using --branch on master git), then I downloaded development branch and then I copied Module directory from development into emoncms_new directory. (I followed this tutorial https://github.com/emoncms/emoncms/blob/Extended/docs/Upgrading.md with some adjustments), but it doesn't work. The Apps menu is appeared, but if I click on Solar PV there is a blank screen.
Why?
Thanks.
[Please notice the message that tells you when your post is held for moderation, and do not post repeatedly. You only waste everyone's time. - Moderator (RW)]
Re: New emoncms get data request method and data viewer
Last time I tried switching Git branches, I made a complete mess of it!
Could you give the Git commands please to switch to the 'newgetdatamethod' branch and also the reverse - to switch back to origin/master.
If I run $ git branch -a I can't even see the 'newgetdatamethod' branch...
Re: New emoncms get data request method and data viewer
Hi Paul
You will need to pull in the changes before switching
and to switch back it should just be
although since git can be .....well, a bit of a git sometimes I would do a back-up first, just in case !
Paul
Re: New emoncms get data request method and data viewer
Thank you!
Re: New emoncms get data request method and data viewer
Thanks Paul, the problem was that the 'newgetdatamethod' branch was not visible from $ git branch -a
A bit of reading suggested $ git fetch origin which updated my local git database, and now I can see 'newgetdatamethod' and have checkout to it.
Unfortunately, the changes made between v1 & v2 of 'My Solar' have introduced an bug stopping it being displayed over https as it requests an insecure XMLHttpRequest endpoint and therefore the page is blocked as a security weakness.
v1 worked OK.
I've logged it as an issue on git
Paul
EDIT 14/3 - Also get an error message when viewing default graphs, but only on mobile device!
Notice: Undefined offset: 2 in /var/www/emoncms/Modules/vis/vis_controller.php on line 74
Strangely, I don't get an error message via laptop
PS - I have cleared the browser cache and cookies.
Re: New emoncms get data request method and data viewer
Here is a screenshot from Chrome's 'developer tools' showing the error message which appears to point back to the use of jquery-1.9.0.min.js although the first version of 'My Solar' still used that library, and yet worked fine.
Re: New emoncms get data request method and data viewer
Hi,
Great improvement !
This should allow people to download many feeds into the same CSV file in one click with all timestamps aligned and no missing point. Today, I'm wasting my time to do that in open office ...
According to me, such a multidownload tool is a must-have feature !
Eric
Re: New emoncms get data request method and data viewer
The Modules/app files within development branch were updated today by Trystan, which has resolved the 2 issues that I raised above.
Works great now using https.
Paul
Re: New emoncms get data request method and data viewer
FYI - in the My Heatpump app the notes down the bottom state the room temp should be a feed named 'room_temp', which was how it originally worked. But the latest version is expecting 'heatpump_room_temp'.
Spent a while wondering why that one feed on the dashboard wasn't working after I upgraded!!
Re: New emoncms get data request method and data viewer
OK, I admit I'm a bit behind the times, but I'm hoping somebody can expand a bit further, as I'm trying to setup v8.5 and having this exact issue mentioned by Paul Reed, but on a regular laptop computer:
Paul
EDIT 14/3 - Also get an error message when viewing default graphs, but only on mobile device!
Notice: Undefined offset: 2 in /var/www/emoncms/Modules/vis/vis_controller.php on line 74
Strangely, I don't get an error message via laptop
PS - I have cleared the browser cache and cookies.
The setup I have is probably as basic as possible, as I've just followed the instructions ( https://github.com/emoncms/emoncms/blob/master/docs/WindowsInstall.md) to set up the "Test" database and used the method in Step 10: http://localhost/emoncms/input/post.json?node=1&csv=100,200,300
And then I get that same error message when I try to look at the graph using the eye choice. But the graph itself seems to be correct.
Any help is greatly appreciated.
Re: New emoncms get data request method and data viewer
If you switch to the 8.5.1 Extended branch, the error is not present.
I don't know what has changed between 8.5 & 8.5.1 Extended - maybe Chaveiro fixed it.
Paul
Re: New emoncms get data request method and data viewer
Thank you very much Paul!
I did the one that was suggested, as opposed to the cutting edge one. And you'll understand why when I ask the ridiculously basic question, is the emoncms-8.5.zip file from https://github.com/emoncms/emoncms the right one? It says 8.5, not 8.5.1 and actually the apparently emoncms version 8.5.0 is listed as emoncms-master.zip, so rather than just possibly wreck what I've already done I thought I'd ask.
Also, while asking ridiculously basic questions, what are the "branches" and "commits" although I guess they are things that have been approved for the main program it'd be nice to know definitely.
Thanks again for all the newbie help!
Re: New emoncms get data request method and data viewer
See this link for branches & commits.
As far as I'm aware, the current master version is 8.4, the 8.5 version is, well 8.5.0, and the Extended version is 8.5.1 XT.
In all honesty, I've found the Extended version much less buggy that the 8.5 version (mainly due to the efforts of Chaveiro), and it appears very stable and well supported in the forum.
Paul
Re: New emoncms get data request method and data viewer
Thanks again Paul.
However, I guess I'm never going to be on the cutting edge, as after I figured out about the branches and downloaded the Extended Version 8.5.1 XT, there were two errors with the basic graph, the original error and a new one.
So I just switched back to the regular 8.5.0.
I don't think it is anything I've done wrong, and it seem I have a basic setup with Window 8.0, but I don't know if anybody wants to investigate further.
Re: New emoncms get data request method and data viewer
When you updated to the extended version, did you clear your browser cache?
I'm running the extended branch out of the box, and don't see any errors.
Paul