Archived Forum

Interesting project,

Originally emonHub was planned to have a persistent "buffer" option using mySQL, but was "shelved" to keep emonHub "SD friendly" as a hdd would be required due to increased disk writes, It's return hasn't been entirely ruled out yet but once the step to installing a database and hdd is made, a full local emoncms install is not far off which negates the need for persistent storage. here is a link to the removed code it may be of use https://github.com/emonhub/emonhub/commit/3666d45acc5eedaa1ba66efd10c5190ef6c35df6 although technically just a buffer if the data is never sent on, then it will just accumulate.

Visualising the data would be easier with emoncms and I can't help but wonder if the gains over a full local emoncms install justify a bespoke solution even though it may not meet your requirements exactly.

There is also an emoncms profile module by Eric_AMANN that compares periods of use for patterns etc aimed at recognising energy saving opportunities but the concept could fit your purpose too, it adds the ability to mask abnormal readings which could help.

What you propose is do'able but may not be the only route, to aid with some debugging sometime ago I put together a "csv" reporter which was quite easy and the resulting file could be opened directly in excel which may be better for pattern recognition by number crunching. This too was write intensive so not a permanent solution either.

My programming skills are "minimal" but if I can help navigate the emonHub architecture I will. 

Paul

Hi Paul and thanks for replying!

Eric's profile module looks very interesting and admittedly is not that dissimilar to where my mind was wandering!  In my head, my raw data visualisation tool was Excel with powerpivot and powerview, with analytics to come later.

I have read with interest the attempts to reduce the writing to SD cards and do understand why the components are built the way they are built.

I have a HDD connected to my pi and am less worried about writes (it's actually an old 20GB ssd).  I documented my complete process of getting the pi and emonhub to work reliably with an external hard drive and it is very straightforward.  Do you think this is of value to anyone? if so, any suggestions as to where I should post it?

I think I understand *just* enough about your referenced python code with the mysql in-memory buffer to hack it into place for my intent, so I'll look into that tonight.  Ideally structure wise and for re-usability by others I was mentally approaching it as a reporter module.  

Do you think I should attempt to go with changes to the buffer module with your previous or do you think making it a reporter is the best approach?

I'd also like to see your csv "reporter" that you wrote if that's ok, is it on github?

I have a HDD connected to my pi and am less worried about writes (it's actually an old 20GB ssd). 

I was led to believe ssd's can suffer too to a lesser degree as do usb sticks, I thought it was just the spinning disks that escaped the "finite number of writes" issues.

The reporter maybe the best way to go, I was just throwing ideas out there and mentioned the "buffer option" as a source of some code for ideas and to explain emonHub's current take on persistent storage.

I don't think the "csv reporter" made it to git,(heres a discussion but no code I'm afraid https://github.com/emonhub/emonhub/issues/41) it was in the very early days,of emonHub but I will have a look to see if I still have a local copy tomorrow. The concept was very simple so should be easy to replicate if not.

Paul

Hi Paul!

Thanks for your comment on the SSD, I *assumed* that linux would "sort out* the trim, I found some references to a aapt-get hrparam and have now configured trim for the SSD!!!

How did you go with the CSV reporter code?  Did you manage to find it?

Actually I couldn't find very much at all, one of the pit falls of using git is not all versions of all files are readily available for searching etc. I did find a text file with a basic implementation, it's a bit messy and probally not of use but I've attached it anyway. maybe you can pick the bones and find something useful.

The basic theme is to "write to file" rather than "send url request" the biggest difference is probably that the data frames retrieved  from the buffer must each be formatted into its own line rather than grouped into one "bulk" string.

I've attached a couple of links about ssd as they are not cheap and can be ruined quite easily unless extremely careful, For me the risk & price doesn't justify the minimal gain (for our application), spinning discs are so cheap and once since they are spinning 24/7 the number of writes is almost irrelevant and the life doesn't deteriorate so much as most wear is done at spin up/down.

Increase Performance and lifespan of SSDs & SD Cards 

Two questions: boot from SSD, run program on startup.

I am actually very much in favor of ssd's generally, I have them on all my pc's and laptop, but stick with spinning disk on the Pi's because it is better suited to the constant stream of frequent writes, if it was being used as a workstation or a public server with heavy random read activity I would definitely reconsider.  

Paul