Archived Forum

Paul (pb66) will probably come to your rescue regarding importing into emonCMS.

On the general question of programming, you don't need to be an expert to survive here, but the basic skills will help. There are 3 main languages we use: C++ for the sketches in the emonTx/Arduino, PHP for emonCMS, possibly Perl, and then there's the BASH script you referred to.

My bible for C is of course "Kernigan & Ritchie" http://www.amazon.co.uk/C-Programming-Language-2nd/dp/0131103628/ref=sr_.... This is the standard text book. The normal place I point people at who want to move up to C++ is http://www.relisoft.com/book/index.htm and that assumes you know C. Because the Arduino environment normally uses a very small subset of the language, neither are the best place for a beginner to start, nor are many of the other on-line tutorials. I'd still suggest you have both of those, and maybe Bruce Eckel's "Thinking in C++" http://www.planetpdf.com/developer/article.asp?ContentID=6634 available for reference.

However, there is this: http://www.me.umn.edu/courses/me2011/arduino/arduinoGuide.pdf which does look to be a good starting place for a beginner. It does not go as far as classes and methods that are used here, though.

For PHP, the on-line manual is good, though as you're probably not going to actually do much with PHP, all you need to do is skim through it so that you know just enough to recognise it.

For scripts, think about the old DOS batch files, BASH is (very) roughly the Linux equivalent. I admit I have to look those up as I've touched on so many that I lose track of which one I'm using, and need to check. But essentially they are a means of stringing together a sequence of commands that you could type in yourself. Most can check the result that comes back and make a decision and branch based on that.

Perl I don't know at all.

For advice is what to do: The first question is do you have any control over the source of the data? If you do, the easy and best way is to format the data at source in the way that emonCMS expects it. Otherwise, how do the csv files get sent? - it can't be manually (can it?) so you'd need to intercept the data, read the csv file and reformat the data. All you're doing here is replicating externally what you would have done internally if only you could have altered the output format. The third way, if you have your own emonCMS, is to write a 'front end' that will accept the csv files that you already have. But to do that, if Paul cannot help, you'd have a very steep learning curve for both PHP and emonCMS, and I wouldn't recommend it as the first choice.

Wow! Thank you for your fast reply :D

I can have control over the data source. I didn't programmed it, but I can reach the one who did and ask to change things in my favor (if it doesn't implicate to many changes).

The files are sent automatically through FTP (don't really know exactly how this is being doing).

For now I only want to use emonCMS to visualize data, use some dashboards and, if possible, generate some alerts via email. Needing this, I won't need C++ right? Only a little of PHP?

Never eared about Perl...

If I'm not wrong, I'll need those bash files to send data to emonCMS is it?

Let's wait and see if Paul see this thread and helps :)

Thanks!

Just a quick question!

The data sent to the current software I'm using is the cumulative data, not the energy consumed in the last 15 minutes.

Does emonCMS receive cumulative data and transform it in 15 minutes (or any other interval) consumption?

Quick example of what it's being done:

Hour | Kitchen Electrical Energy

12:00 | 1354 kW

12:15 | 1359 kW

12:30 | 1370 kW

12:45 | 1382 kW

13:00 | 1389 kW

Then the software process the data and returns the following energy comsumption:

12:00 | 0 kW (no previous data available)

12:15 | 5 kWh

12:30 | 11 kWh

12:45 | 12 kWh

13:00 | 7 kWh

Hello Bruno, yes you can do this using the bargraph visualisation tool, you can select the interval and then request the difference. There a little documentation on how to do this at the bottom of this blog post: http://openenergymonitor.blogspot.co.uk/2014/08/monitoring-solarpv-heatp...

in the section: "Creating a daily electricity consumption graph in a dashboard using watt hour feeds (instead of the old power_to_kwhd method)."

What I think your next move should be is to check the formats (there are several) that emonCMS accepts, and see how easy (or difficult) it will be for your programmer to give you the data in one of those formats. What happens next will depend upon the answer you get.

The best place to collate or format the data is usually at source if you can, However you can probably just run a script to rearrange the csv and submit to emoncms quite easily.

Currently is the csv file replaced or overwritten every 15mins or is the data appended ? Is the new csv file named differently each time or does it always have the same name?

Also what is the current format of the data within the csv file? can you provide a bit of raw csv?

There are several options but I'm guessing you are not running any other OEM stuff and therefore are not using emonHub. That being the case you will probably use the emoncms input api directly so its probally just a case of looping through each entry converting the time to a unix timestamp adding a node id of choice and formatting the data(s) so we have a string of "unixtimestamp nodeid val1 val2 val3...."and then post all of those strings in one hit using "bulk input"

This will give you access to all the normal processing etc including calculating power from energy or vice versa

Paul

On my old piece of software i was doing the same.
I got Values fetched by a WAGO installation and send the data every 10 Minutes by a *.csv file to my monitoring system. Data ware dropped by ftp on the server. On the server there was a spooler process running, waiting for new packages. Inside the packages ware the description of the data. the spooler process took the new package and moved it in the right place (destination folder), so the sender could see over ftp that the package is moved and set it to recived.

No the spooler process starts the import process for the new package.

the import process reads the csv and push data to my monitoring script.
At this destination i think it would be either good to process data by emonHUB or by emonCMS.

if you like to, i lay down the source code to github and help optimizing the import process of .csv data.
I am at the same difficulty in the moment and have to do this for our old data fetchers to.

The processes are named ecccombie and you find it under
https://github.com/mpruessmeier/eccombie

the project is dead since round a year, but now i will get it alive

Greetings Marc

Hi, Bruno.

I have the same things to do on myself.
I have another datafetching system running on some WAGO Hardware. In my old monitoring system i am sending the data packages as csv via ftp to my server. My old process picks up the data and moves it to the right destination (importfolder) so the sender can see via ftp that the package is gone (taken) and sign it as sended.

the pickup process starts then the import process on the importfolder.

I started the process a year ago and the essence is put on github. you can find it here.
https://github.com/mpruessmeier/eccombie

I changed now the README.md so it is clear what should happen.
I will start on my own next days with coding, cause I am in the moment in different projects.

The questions to @TristanLea and @pb66 is,
- whether it is better to use emonhub or emoncms to insert the data readed by the csv file
- what happens, when resend the same time-value (maybe over bulk) to emoncms

Greetings Marc
 

Hi guys!

Thank you again for all the answers, really appreciate that!

The current .csv file is named after the date and hour it is created, so every 15 minutes it has a different name. The name for the 09h30 data exported was 24-03-2015_09-30-20.csv

The .csv has 2 lines and several columns, each column is one different meter, with the exception of the first 2 columns. The first column is the Date and the 2nd is the Hour. 

This is what my file looks like (columns are separated by |):

Date | Hour | Temperature | KitchenGaz | DHW | General | Kitchen | ExternalPool | Lighting

24-03-2015 | 09:30:00 | 11,7 | 11142,3 | 9406,7 | 1334631,8 | 337914,9 | 68146,9 |  200092,6

All data is cumulative with the exception of the temperature for obvious reasons.

DHW and Gaz are in m3. The current software then calculates the kWh of Gaz (the formula I use is 1m3 = 11,72 kWh).

So I just knocked up a bash script to get you started. In its current form it just looks in a preset folder for A file ending ".csv" I don't know the logistics of your file so have left it there to test and decide how it can be integrated.

It currently assumes there will only ever be one file there and it needs to be run manually. If the file count is ever increasing some sort of "last updated" check will be needed.

It can probally be run from cron every 15mins but if the csv files are regularly moved or deleted it may be better to run the script more frequently with a test like above so as not to miss any files.

I have commented throughout so you can alter it as required.

There is no error checking or logging etc

All the values are passed to emoncms as floats except the date/time which is converted to a unix timestamp, so after the first use you will have 7 inputs in the same order for the node id you set.

Paul

I got all confused with your script LOL

First I read BATCH instead of BASH, so I was trying to run it as a .bat file but, of course, without good results. I must have spent an hour until I noticed I was working with a bash file :D

Now, for some minutes now, I was trying to run the .sh file on my Windows 7 OS. After some digging in google, I've discovered windows don't run .sh files :P

Isnt it possible to run the script under Powershell.exe under WIN7?
​Powershell should be standard under WIN7.
Press Window Start Button, enter Powershell and ENTER.

Greetings Marc

PS: Paul thanks for the script, i will use it under ecccombie

Bruno - I wasn't aware you were on windows, the previous undisputed mention of bash just led be to think linux. I have never done any dev work in DOS the only dev stuff I do on windows is in an arduino or python IDE so I'm not sure how I will be able to help you.

I have not tried any, but there are apparently, ways to run bash on a windows PC (other than a linux VM) eg git for windows has an in-built bash interpreter.

My initial choice would of been to write it in python for use with emonHub but went the route of bash as I didn't expect you to have emonHub and possibly not python either. I am a bit tied up right now but if I get a chance before you find an alternative way, I will port it to Python.

Marc - You are welcome to use what you can from the script, I think everyone's "csv" will differ so it was more of an example than a solution. to answer your previous questions, My own preference is that all data should come via emonHub, but that is far from mandatory, some users don't even use emonHub. emonHub (to me) is an independent interface or input stage for emoncms. If I point everything at emonhub then I can manage where my data goes from one place (emonhub.conf) rather than individual scripts or hardware settings, plus if you want to post to multiple locations etc

In this instance it's less of an issue but with larger csv uploads, as long as the buffer size is larger than the number of csv entries you need'nt worry about network connection, server load or receipt confirmation as if you send via emonHub it will manage this by sending the data in manageable "bulk" batches, only delete the copy in memory after receipt is acknowledged and persevere until all data is posted regardless of how poor the network connection is (provided the hub isn't restarted or the frame count pushed over the buffer size)

On emoncms.org and full local versions of emoncms any new "duplicated" data will overwrite old data as a rule, but the feed engines in the buffered-write version (and probably emonView) are append only so you cannot insert or edit anything earlier than the last entry.So duplicate data posted is just extra work for client, network and server but shouldn't cause a lasting problem ie no corrupt or duplicate data saved. I'm not entirely sure myself of the impact of fresh historical data for fixed interval feeds,, timeseries will edit any existing datapoints of the same timestamp or insert new if the timestamp is unique.

Paul

Hi all!

I'm not posting here but I'm working on EmonCMS :)

I've started learning Python and already made a script to upload my data to EmonCMS. So this thread could be treated as solved (hoorray!) 

Thank you all for your help :D

I'll need more of it to set proper feeds and visualizations, but first I'll have a look in forum's topics to see if I find the help I need!

What did you change for importing data?

I opened a own issue, cause in standard it's not possible to import data older then 2 hours. 

Look at at my open post.

if you reimport your data, don't you have any problems?

http://openenergymonitor.org/emon/node/10386

I'm having troubles generating feeds, so I don't know if there's a problem importing data or no. 

I'm generating a link like this to send data every 5 minutes - http://.../..../input/post.json?time=1428308951&node=1&csv=100,200,300

I opened another thread to see if someone can help me - http://openenergymonitor.org/emon/node/10427

Hi Bruno,

Care to share the script? I am looking for a script to upload 2 years of Emoncms data from an older installation to a new one running PHPtimeseries. My scripting skills are next to 0, so I need a basic script to work upon...

The variables names and comments are in Portuguese (sorry for that!). Google translate should be enough for you to understand the comments and variables since I didn't used any abreviations, but if you encounter some translation issue just ask here and I'll translate as soon as I see it.

My next step (after makings emon fully operational with data I already have) will be to upload a bunch of historical data so I'll be interested in that script you want to make :p

Obrigado! So based on the comments in the scripts (I have a CSV with a column epoch and a column with the value) I need to:

- remove the epoch time conversion

- Change the emoncms link (and reduce the amount of variables inside the link)

And then I should be good to go, or am I missing something?

Yes, that should be enough. There's more "garbage" code there for some tests I made (some prints) that you can also remove if you don't want them.