Importing data with curl

After editing my post several times there just came up a possible stumbling block: Does the data have to come sorted by date?

 

Throughout the day I've setup script to import data from XML's like this:

<?xml version="1.0" encoding="utf-8"?>
<RMCU>
<TP ID="1P3">
<VALUES><DATETIME>2014-06-09T18:14:00Z</DATETIME><EE>13.846</EE><STATUS>0</STATUS></VALUES><VALUES><DATETIME>2014-06-09T18:29:00Z</DATETIME><EE>13.850</EE><STATUS>0</STATUS></VALUES>
</TP>
<TP ID="4P2">
<VALUES><DATETIME>2014-06-09T18:14:01Z</DATETIME><EE>1469.310</EE><STATUS>0</STATUS></VALUES><VALUES><DATETIME>2014-06-09T18:29:01Z</DATETIME><EE>1469.595</EE><STATUS>0</STATUS></VALUES>
</TP>
</RMCU>

the access.log states that the requests were fine:

my-ip - - [23/Jun/2014:00:40:06 +0200] "POST /input/post?apikey=my-api-key2&node=12&time=1401723858&json=kWh:353.577 HTTP/1.1" 200 12 "-" "curl/7.26.0"
my-ip - - [23/Jun/2014:00:40:06 +0200] "POST /input/post?apikey=my-api-key2&node=12&time=1401724758&json=kWh:353.975 HTTP/1.1" 200 12 "-" "curl/7.26.0"
my-ip - - [23/Jun/2014:00:40:06 +0200] "POST /input/post?apikey=my-api-key2&node=13&time=1401696860&json=kWh:46.848 HTTP/1.1" 200 12 "-" "curl/7.26.0"

But the feed's didn't update as expected, some data got posted but it didn't continue... I've imported other data from CSV's before, the CSV's contained only the changerate's and all went fine... 

The process chain is: log, rate, 34, kwhpwr, hist

The terminal Screenshot show's that feeds seem to be empty. But browser show's something else... And the last one simply proof's that in the beginning there did some data arrive.

https://www.dropbox.com/s/0rb46wlrohnet8k/Screenshot%202014-06-23%2000.54.34.png
https://www.dropbox.com/s/jksdeucn4a9o083/Screenshot%202014-06-23%2001.18.35.png
https://www.dropbox.com/s/xp494lqx0x7dxyg/Screenshot%202014-06-23%2000.58.51.png

And BTW: I import data in kWh's but from what get's plotted it look's like emoncms interpret's that data as Wh's. On the input/api Page I can only see "{power:200}" as an example, but I use kwh... Can someone shad some light on that too please?

hoping for help...

Robert Wall's picture

Re: Importing data with curl

"And BTW: I import data in kWh's but from what get's plotted it look's like emoncms interpret's that data as Wh's, can someone shad some light on that too?"

I can shed some light on this one: Until recently, emonCMS only accepted signed integer values as input. In order to preserve decimal fractions, it was normal to multiply values such as temperature by 10 or 100 before sending to emonCMS, and to multiply by 0.1 or 0.01 inside emonCMS to restore the correct value. Power was always sent in Watts and the conversion to kW applied automatically.

Sorry, but I can't help with your other problems.

mircsicz's picture

Re: Importing data with curl

Thx for your reply robert. Especially the integer thing is very good to know! so I need just a little gnu-bc magic to happen before importing...

The not so good news is importing kwh is not supported as you say, but that leave's me confused: Because there's that process's in emoncms called "kwhpwr" & "kwhkwhd", from those I assumed that kwh input is possible!!!

Hopefully there will be a way to come around that restriction... Because I plan a longterm usage with a datalogger sending those XML's and this device simply reads MBus and ModBus meter's and handsover their value's in the file... So there's basicly no calculation on that logger. I sure can send the power if the meter holds that data, but not all meter's out there do have a register for that, especially the cheap ones.

According to: http://www.vis.openenergymonitor.org/emoncms/site/docs/inputprocessing I still expect kWh or kwh to be possible:

kWh to kWh/d

Description: Some datalogging equipment may just provide total kWh used or generated data, this can be used to convert this ever accumulating kWh variable to a kWh per day feed. (Note: in a pinch you can use this after an accumulator process to log the daily increase in the accumulator to a feed).

Argument: The feed name that you wish to write to. a feed can be created if it doesn't already exist.

could someone please approve that my expectation in what I read there meets the possibilities?

pb66's picture

Re: Importing data with curl

EmonCMS will use either kwh or wh, the difference is scaling. If you have an input in kwh and want to pass through the wh accumulator use a *1000 process first.

"The process chain is: log, rate, 34, kwhpwr, hist"

I think you could just use something like this if you need to retain kwh values in case of input device reseting to zero 

log to feed, *1000, wh accumulator, *0.001, kWh to Power 

or if the kwh value is unlikely to zero the you can use something like

log to feed, rate of change, accumulator, kWh to Power.

Paul

mircsicz's picture

Re: Importing data with curl

Hi Paul,

is my way telling emoncms that the VALUE is in kWh, by passing json=kwh:353.577 on the request, correct?

I had nearly the same chain, changed to your suggestion. The data I used for testing was older CSV's, those where not transmitted as the total meter value but contained the consumption or production ("rate of change") since last read...

Am I right that in that case the chain should be: log, 34, kwhpwr, hist I'll give that a shot tomorow during business hours...

I did not fully get your last two hints. What I understood was that in case the transferred value already is a "rate of change" since last read I use (in case of Wh's transfered) "log to feed, *1000, wh accumulator, *0.001, kWh to Power" and in case of kWh's: "log to feed, accumulator, kWh to Power"

And in case of a total permanently increasing value (like what you can read on the meter's display) I use (in case of kWh's transfered): "log to feed, rate of change, accumulator, kWh to Power"

And if I want to add a kwhkwhd feed is that the right chain (those are meter's with transformerratio's): "x,log,rate,accumulate,kwhpwr,=0,x,kwhkwhd?

 

THX for your patience

pb66's picture

Re: Importing data with curl

The emonCMS inputs are able to accept almost any value type or size. Only the processes that are specific to a value type, size or measurement dictate how the variables are handled and generally, since it is assumed that you will be measuring power or energy, many screens display kw w kw/h wh as unit identifiers etc

For example I also keep track of my energy cost and feed-in tariffs in monetary terms. emonCMS is not aware the value I passed in is a "pence per unit" value and when i multiply that value with my "importing in kw" feed I get a "current running cost" which is scaled and saved to both a "power to kwh" feed and a "kwh to kwhd" feed but it actually it is logging "pence per hour" and "pence per day" but it doesn't know or care the only time it is apparent the feeds are not being used as expected it if a graph is produced the data and values are correct but the unit is declared as kwh not pence per hour.

Most or the processes can be used for other units in this way but caution to attention to scaling is required, using my pence per hour example above since the power to kwh process has built in scaling from w to kw this has to be factored in and adjusted when using for pence to pence per hour or pence to pounds per hour.

With regard to the kwhkwhd, I believe that process passes the input to the output without any change only recording its value to a feed that is reset daily, so you should be able to just slot it into the existing chain, so either 

log to feed, *1000, wh accumulator, *0.001, kwh to kwhd, kWh to Power 

or

log to feed, rate of change, accumulator, kwh to kwhd, kWh to Power

Paul

pb66's picture

Re: Importing data with curl

 I believe "json=kwh:353.577" would be telling emoncms the value of the input called "kwh" is 353.577, maybe....

Paul

mircsicz's picture

Re: Importing data with curl

Hi Paul,

thx for the detailed answer's. I've already changed the process chain...

If anything is simply treated as integer and the string before the value in the json call is just a name so no one except some specific processes except anything from the value's how can I plot a graph of my temperature's next to my energy consumption with axis that have a correct naming in a multigraph?

What I mean is I still miss or didn't find the ability to asign several axes to a multigraph. What I'ld like to do in the near feature is plot a multigraph from my home in- and outside temp, energy consumption of my Desktop and power of my Desktop in a multigraph... 

I've already done a multigraph on my installation, will send the URL via PM.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.