I am a noob, so please be gentile. I wrote a python script to gather the previous weeks data from each of my emoncms.org feeds. It gathers the data and then creates a csv file for each feed. Hope someone else can find it useful. criticisms are welcomed.
import csv
import time
import urllib
sec_per_day = 60 * 60 * 24 # 60 seconds/min * 60 min/hr * 24 hr/day
# make a dictionary of feeds to create filenames
feeds ={"name1":"feed1 id","name2":"feed2 id","name3":"feed3 id"... etc} #name is what you want the file called
# feed id is the emoncms feed id
# create a list of keys to use as filenames
feed_list = feeds.keys()
now_tstamp = int(time.time()) # grab unix timestamp for right now
wk_ago_tstamp = now_tstamp - (7 * sec_per_day) # calculate the timestamp for 7 days ago
now_tstamp *= 1000 # format for emoncms
wk_ago_tstamp *= 1000 # format for emoncms
k=0
for z in feeds:
# create a string for the url of the data
data_url = "http://emoncms.org/feed/data.json?id=" + feeds[feed_list[k]] + "&start=" + str(wk_ago_tstamp) + "&end=" + str(now_tstamp) + "&dp="
# read in the data from emoncms
sock = urllib.urlopen(data_url)
data_str = sock.read()
sock.close
# data is output as a string so we convert it to a list or lists
data_list = eval(data_str)
# open a file to save the data to
j=0
myfile = open(feed_list[k] + ".csv",'wb')
wr=csv.writer(myfile,quoting=csv.QUOTE_ALL)
# loop though the list to format the data so libraOffice or excel can import it
for i in data_list:
wr.writerow(data_list[j])
j += 1
myfile.close()
k += 1
Re: script to download data
That is useful, thanks for sharing. I'd also add the API key in the data_url , now it will only work if you are logged in emoncms.
Re: script to download data
Just changed the script to convert emoncms timestamp into readable date and time. I also added the API key to the URL so you won't have to be logged in.
Re: script to download data
Hi there, thanks for the script !
Am having various troubles runnig it .. what version of python are you using ?
Pat
Re: script to download data
Sorry to hear you are having trouble. I am running python 2.7.3. I have not run into any trouble so far. Let me know if I can help.
Re: script to download data
Ah. I am on 3.3.2 as I suspect that is causing a heap of trouble for me I think :(
And I'm not Python savvy enough to figure it out (yet).
File "emoncms.py", line 44, in main data_url = "http://emoncms.org/feed/data.json?id=" + feeds[feed_list[k]] + " &apikey=337ec3bdc07321c9890ffc6c9f00XXXX&start=" + str(wk_ago_tstamp) + "&end=" + str(now_tstamp) + "&dp=" TypeError: 'dict_keys' object does not support indexingFeeds is only one feed right now
feeds = {
"House Power":"26845"
}
The only thing I can think that it does not like is ...
There *is* a change from 2 to 3 which is referenced in various places ... eg
"py33: 'dict_keys' object does not support indexing In python 3, dict.keys() is a class of 'dict_keys', it does not support indexing. Cast the dict to list to get the "first" value of dict, which is compatible with python 2&3. "But I'm afraid I still don't know what to do !
Pat
(ps will install python 2.7, on the basis I can have both versions!)
Re: script to download data
I'll see if I can re-write this to not use dictionary indexing. Thanks for bringing it to my attention.
Re: script to download data
Hopefully this fixes it.
Changed:
feed_list = feeds.keys()
To:
feed_list = list(feeds.keys())
This should change feed list from being an object to being a list, which can be indexed.
This rev will ask you how many days you would like to download.
Re: script to download data
Great, will give it a shot !
Thanks ever so.
Pat
Re: script to download data
So, it appears there are more relatively significant changes from 2 to 3 as well!
"raw_input" does not work, but "input" does .. but then later on I get ..
How many days data would you like? 14 Traceback (most recent call last): File "emoncms.py", line 71, in <module> main() File "emoncms.py", line 47, in main sock = urllib.urlopen(data_url) AttributeError: 'module' object has no attribute 'urlopen'Re: script to download data
Incidentally, works brilliantly well for me in python 2.7.6 !
Pat
Re: script to download data
There are indeed important changes from python 2.7 to python 3.
http://docs.python.org/3/howto/pyporting.html
http://python3porting.com/bookindex.html
Note the 2to3 translation tool.
Re: script to download data
I've run the script through the 2to3 translator. Hopefully this solves all the issues.
Re: script to download data
Hi, this was exactly what I was looking for. Thanks v much :)
However when I tried to run it in Python 3.3. I was still getting some errors (problems with binary file writing among them).
Attached is the version I got working for my build. Treat this with caution lol as I'm really rusty on coding! :-P
There's still an odd error when emoncms seems to return null in response to the request. I've just captured it and the program exits for now. If you run the program again then it usually works fine with a non-null return. That makes me think it's maybe a server load issue or potentially something with the timing format that goes into the request string...
Thanks again :)
Michael
Re: script to download data
Hi
I have use the python 2.7 script, and for any reason, the amount of downloaded data from www.emoncms.org is limited,
I mean:
Only if i download one hour period I get the 5 sec interval measurements. It seams that the limit is in nearly over 1000 measurements per csv file.
Has emoncms.org any kind of download limitation or is something about the script?
Re: script to download data
Using the json feed data call outputs 1000 data points, regardless of time span.