New emoncms get data request method and data viewer

While developing the new application specific dashboards recently, I found that I needed to be able to request data from the server for multiple feeds with the exact same timestamps so that you could directly run calculations such as the energy balance or 'virtual store' calculation using multiple feeds.

When a request was made to the emoncms data request api which looks like this:

http://emoncms.org/feed/data.json?id=71667&start=1426031940000&end=1426064520000&interval=30

The datapoints returned where not guaranteed to be aligned to the request start time and interval making this cross feed comparison difficult.

Yesterday's update makes it much easier to do this kind of request, if you call

http://emoncms.org/feed/data.json?id=71667&start=1426031940000&end=1426064520000&interval=30&skipmissing=0&limitinterval=0

for multiple feeds you will be guaranteed to get a json output with the same number of datapoints and the same timestamps.

I've added a new default data viewing graph that loads when you click on the eye on the feeds page which makes it easier to explore this new data request API. Its possible

The new paramaters are skipmissing and limitinterval.

skipmissing = 0 will return null if there is no value at the requested timestamp, skipmissing = 1 will skip the datapoint entirely returning an output similar to the older data request api.

limitinterval =1 will limit the minimum request interval to the interval of the feed being used so that if you have a 10s phpfina feed it wont return data at less than 10s.

CSV Output, theres also a new CSV output box in the data viewer graph which makes export to a spreadsheet easier to do.

There are still a few teething issues with the new request method that need ironing out, but hopefully the result will be a more useful data request method that makes multiple feed post processing easier.

kWh zoomer bug fix - as part of this update I have also fixed the kwh zoomer visualisation which wasnt displaying the years correctly.

Testing, if you wish to test this on your own emoncms install you can find these features in the newgetdatamethod branch here to be merged into master once all related bugs are sorted https://github.com/emoncms/emoncms/tree/newgetdatamethod you will also need this to run the latest application specific dashboards module https://github.com/emoncms/development/tree/master/Modules/app

jmessenger's picture

Re: New emoncms get data request method and data viewer

I'd like to try this out. Would you mind posting a pointer to some data backup/restore instructions please?

Thanks,  John.

Paul Reed's picture

Re: New emoncms get data request method and data viewer

Have you checked the user guide? - https://github.com/emoncms/emoncms/blob/master/docs/Backup.md Paul

hercules71035's picture

Re: New emoncms get data request method and data viewer

Hi there,

I downloaded newgetdatamethod with git clone (using --branch on master git), then I downloaded development branch and then I copied Module directory from development into emoncms_new directory. (I followed this tutorial https://github.com/emoncms/emoncms/blob/Extended/docs/Upgrading.md with some adjustments), but it doesn't work. The Apps menu is appeared, but if I click on Solar PV there is a blank screen.

Why?

Thanks.

[Please notice the message that tells you when your post is held for moderation, and do not post repeatedly. You only waste everyone's time. - Moderator (RW)]

Paul Reed's picture

Re: New emoncms get data request method and data viewer

Last time I tried switching Git branches, I made a complete mess of it!

Could you give the Git commands please to switch to the 'newgetdatamethod' branch and also the reverse - to switch back to origin/master.

If I run $ git branch -a I can't even see the 'newgetdatamethod' branch...

pb66's picture

Re: New emoncms get data request method and data viewer

Hi Paul

You will need to pull in the changes before switching

cd /var/www/emoncms

git pull

git checkout newgetdatamethod

cd -

and to switch back it should just be

cd /var/www/emoncms

git checkout master

cd -

although since git can be .....well, a bit of a git sometimes I would do a back-up first, just in case !

Paul

jmessenger's picture

Re: New emoncms get data request method and data viewer

Thank you!

Paul Reed's picture

Re: New emoncms get data request method and data viewer

Thanks Paul, the problem was that the 'newgetdatamethod' branch was not visible from $ git branch -a

A bit of reading suggested $ git fetch origin which updated my local git database, and now I can see 'newgetdatamethod' and have checkout to it.

Unfortunately, the changes made between v1 & v2 of 'My Solar' have introduced an bug stopping it being displayed over https as it requests an insecure XMLHttpRequest endpoint and therefore the page is blocked as a security weakness.
v1 worked OK.

I've logged it as an issue on git 

 

Paul

EDIT 14/3 - Also get an error message when viewing default graphs, but only on mobile device!
Notice: Undefined offset: 2 in /var/www/emoncms/Modules/vis/vis_controller.php on line 74
Strangely, I don't get an error message via laptop

PS - I have cleared the browser cache and cookies.

Paul Reed's picture

Re: New emoncms get data request method and data viewer

Here is a screenshot from Chrome's 'developer tools' showing the error message which appears to point back to the use of jquery-1.9.0.min.js although the first version of 'My Solar' still used that library, and yet worked fine.

Eric_AMANN's picture

Re: New emoncms get data request method and data viewer

Hi,

Great improvement ! 

This should allow people to download many feeds into the same CSV file in one click with all timestamps aligned and no missing point. Today, I'm wasting my time to do that in open office ... 

According to me, such a multidownload tool is a must-have feature !

Eric

Paul Reed's picture

Re: New emoncms get data request method and data viewer

The Modules/app files within development branch were updated today by Trystan, which has resolved the 2 issues that I raised above.

Works great now using https.

Paul

sumnerboy's picture

Re: New emoncms get data request method and data viewer

FYI - in the My Heatpump app the notes down the bottom state the room temp should be a feed named 'room_temp', which was how it originally worked. But the latest version is expecting 'heatpump_room_temp'.

Spent a while wondering why that one feed on the dashboard wasn't working after I upgraded!!

 

 

sub3marathonman's picture

Re: New emoncms get data request method and data viewer

OK, I admit I'm a bit behind the times, but I'm hoping somebody can expand a bit further, as I'm trying to setup v8.5 and having this exact issue mentioned by Paul Reed, but on a regular laptop computer:

Paul

EDIT 14/3 - Also get an error message when viewing default graphs, but only on mobile device!
Notice: Undefined offset: 2 in /var/www/emoncms/Modules/vis/vis_controller.php on line 74
Strangely, I don't get an error message via laptop

PS - I have cleared the browser cache and cookies.

The setup I have is probably as basic as possible, as I've just followed the instructions ( https://github.com/emoncms/emoncms/blob/master/docs/WindowsInstall.md) to set up the "Test" database and used the method in Step 10: http://localhost/emoncms/input/post.json?node=1&csv=100,200,300

And then I get that same error message when I try to look at the graph using the eye choice.  But the graph itself seems to be correct.

Any help is greatly appreciated.

 

Paul Reed's picture

Re: New emoncms get data request method and data viewer

If you switch to the 8.5.1 Extended branch, the error is not present.
I don't know what has changed between 8.5 & 8.5.1 Extended - maybe Chaveiro fixed it.

Paul

sub3marathonman's picture

Re: New emoncms get data request method and data viewer

Thank you very much Paul!

I did the one that was suggested, as opposed to the cutting edge one.  And you'll understand why when I ask the ridiculously basic question, is the emoncms-8.5.zip file from https://github.com/emoncms/emoncms the right one?  It says 8.5, not 8.5.1 and actually the apparently emoncms version 8.5.0 is listed as emoncms-master.zip, so rather than just possibly wreck what I've already done I thought I'd ask.

Also, while asking ridiculously basic questions, what are the "branches" and "commits" although I guess they are things that have been approved for the main program it'd be nice to know definitely.

Thanks again for all the newbie help!

 

Paul Reed's picture

Re: New emoncms get data request method and data viewer

See this link for branches & commits.

As far as I'm aware, the current master version is 8.4, the 8.5 version is, well 8.5.0, and the Extended version is 8.5.1 XT.

In all honesty, I've found the Extended version much less buggy that the 8.5 version (mainly due to the efforts of Chaveiro), and it appears very stable and well supported in the forum.

Paul

sub3marathonman's picture

Re: New emoncms get data request method and data viewer

Thanks again Paul.

However, I guess I'm never going to be on the cutting edge, as after I figured out about the branches and downloaded the Extended Version 8.5.1 XT, there were two errors with the basic graph, the original error and a new one.

So I just switched back to the regular 8.5.0.

I don't think it is anything I've done wrong, and it seem I have a basic setup with Window 8.0, but I don't know if anybody wants to investigate further.

 

 

Paul Reed's picture

Re: New emoncms get data request method and data viewer

When you updated to the extended version, did you clear your browser cache?
I'm running the extended branch out of the box, and don't see any errors.

Paul

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.