Archived Forum

Good work Trystan.  I've almost finished the initial version of my more secure sensor nodes which I'll push up to github soon.

Emoncms.org service, pricing, billing

Although probably not good news to many readers, it's a step that's probably long overdue, and if it funds an admin & development role, then all the better.
Nice write up, it's good to know your future development plans, and which direction emoncms is taking.

Paul

Thanks Paul, Stuart. appreciate your support and time reading through

Progress and recent changes to emoncms.org:

• Emoncms.org has now got a 3rd storage server adding 1TB of SSD disk space to the 0.5TB there at the moment!
• The password reset issue has now been rectified.
   
• **PHPFiwa: The option to create new PHPFiwa feeds has been removed, replaced with PHPFina as the standard engine. I have an initial implementation of post processing averaged layers that runs on the PHPFina feed engine that will provide much of the same functionality with the averaged layer being a cached property that can be easily rebuilt and updated on request to the average API. The next step for emoncms.org is to convert all existing PHPFiwa feeds to PHPFina feeds and then at a later stage introduce this post processed averaging once its been refined more  thoroughly.
   
• **Histograms: The input processing based histogram feature has been removed and the updating of existing histograms via input processing have also been stopped. This input processing implementation has now been replaced with the on the fly post processing option available using the graph module. See section on the histogram feature in the graph module here. I will start archiving existing histogram data towards the start of May. For anyone who wishes to export this data its possible to do this with the backup tool in usefulscripts or by using the export api directly: (https://emoncms.org/feed/export.json?id=HISTOGRAMFEEDID&start=0)
   
• Clearer input processing and feed engine options: I have tried to improve the grouping of the input processors and added to their descriptions including a link to the forum post on Creating kWh per day bar graphs from Accumulating kWh
  
• The feed engine descriptions have been amended to Fixed interval (PHPFina Recommended) and Variable interval (PHPTimeseries).

Next steps for emoncms.org

1. Fix timezone support using cumulative kwh data for: simplezoom, stacked solar stacked and zoom visualisations
2. Review timezone support for phptimeseries
3. Conversion of emoncms.org PHPFiwa feeds to PHPFina
4. Addition of the post processor module so that the feeds required for the new mysolar app can be calculated from existing data.
5. Launch new mysolar app which uses cumulative kWh feeds
6. Complete the new solar pv documentation to reflect the new mysolar app.
7. Create publicly accessible API documentation for emoncms.org

Later stage

1. Review datapoint editing + daily and realtime editor visualisations
2. Add the ability to save graphs created with the new graph interface replacing the need for the realtime, rawdata, bargraph and multigraph visualisations as all of these could be built from the new more powerful graph interface - simplifying the overall experience of using and configuring emoncms graphs.
3. Post processed cached average layers for PHPFina engine.

The result of all of this will be a much more streamlined, simpler to use and easier to maintain emoncms.org. Moving as much as possible away from a reliance on input processing to post processing reduces the reliance on getting everything right at the setup stage and makes processed data recoverable, as its easy to rebuild in the event of a bug, server issue or incorrect initial implementation of a feature as well as providing more flexibility: the ability to modify and reprocess data at a later stage as requirements change.

The data core of emoncms consisting of the remaining input processes and core feed engines will be a more robust and simpler core that is easier to maintain as emoncms.org develops especially with the way its is distributed across several servers.

I've fixed these now on emoncms.org:

• Fix timezone support using cumulative kwh data for: simplezoom, stacked solar stacked and zoom visualisations
• Review timezone support for phptimeseries

If your using cumulative kWh data created with the input process power_to_kwh on emoncms.org the visualisations bargraph, simplezoom, stacked solar, stacked and zoom + myelectric app will all return daily kWh data that is aligned to the timezone selected on the user profile page - which should complete timezone support across emoncms.org for cumulative kWh feeds.

A full page refresh might be needed to load the latest javascript.

If you have daily kWh data using the phptimeseries feed engine on emoncms.org the data is saved and reset at midnight UTC which has been a long running bug. For those on BST the changes to the visualisations required for power_to_kwh feeds mean that at least the day is correctly now for kWh/d phptimeseries data and the missing sunday 27th is back, although the kWh shown will be for the period defined by midnight UTC.

The post processing module will make it possible to re-process these feeds from their underlying power data feeds to a timezone correct form very soon.

The latest code on emoncmsorg is up here https://github.com/emoncms/emoncmsorg/commits/master. Im going to work out next how to copy the main changes around timezone support over to emoncms v9.

Changes to bargraph, simplezoom, stacked solar stacked and zoom visualisations to support timezone correct daily data for phpfina and phptimeseries have now been implemented on the emoncms v9 master branch for testing.

I've started with converting emoncms.org phpfiwa feeds to phpfina feeds today, its quite a simple operation but one thing I have noticed in a number of initial feeds is that if the posting to a particular feed is very sporadic or if the interval has been selected to be significantly less than the actual post interval the graph can show that there is no data present - the data is actually there its just that there are a lot of null values surrounding it. For example lets say we are posting data every 60s but the feed has been created at 10s feed interval. The underlying feed will look like this:

If the phpfina request method is not aligned to the data correctly, lets say the data was actually recorded 20s past the minute and the request is pulling data out every 60s, then the data returned will be empty even though its actually there.

I've ran a check on 15000 phpfina feeds on one of the emoncms.org storage servers to see how many feeds have more than 60% of the data as nan values based on the last 5000 datapoints in each feed and it's 32% which could mean that there are quite a large number of graphs not showing the underlying data that is present.

In order to make it possible to at least see the data recorded in phpfina feeds where the feed interval is less than the data post interval I've fast forwarded the introduction of the on the fly averaging (mean) I mentioned above. Its not the full implementation but it provides averaging for up to 1 week of data recorded at 10s.

So if you looking at your data in emoncms and it appears as thought there is none present this is how its possible to show the data:

1. Click on the eye icon for the feed you wish to view in the emoncms feed list to open the feed in the graph viewer (default feed viewer).
2. Perhaps there will be no data visible or only some of it will be visible, if you click on show missing it may go blank, or look like this:
   
3. With one week or less of data in the view, tick the PHPFina Average checkbox and then click on reload, the data should now appear more complete and as a plus the result is and accurate average:
4. Beyond 1 week of data it reverts to the default data request method without averaging, the next step is to precompile and cache the layers required for longer term averages.

I've pushed this all up to the emoncms master branch and graph module for other systems.

The longer term solution for feeds where the feed interval is shorter than the post rate is to recompile the feed to the correct interval but this is a bit more involved, perhaps a feature for the post processor module that's currently in development.

Hi guys,

Loving this project. But wondering about all the work you've been doing on database and dashboards. Have you guys looked into Grafana for presenting time series data? And what about using an existing time series database, such as graphite or influxdb? At my work we use send 1.3million metrics per minute to graphite. It's very robust.

I'm adding various sensors in my house and intend to log an assortment of metrics. emonPi is just one of them. So I was looking at using raintank and grafana.net. They are are hosting a graphite database and use grafana for dashboards. All very slick.

I'm going to start working on some emonHub interfaces shortly. I'll share them when I get something working.

Thanks!

Thanks hmmm01i, to be honest Im very happy with the current solution. Its very lightweight, I understand the code so can modify it to requirements, its scaling very well, the timeseries engine part occupies a very small fraction of the overall server load. The changes above are more a refinement and focus on a version of the timeseries engine I wrote that has now been stable and effective for quite some time. I did look a bit into graphite and I think its very good, I particularly enjoyed reading Chris Davis's contribution to the book architecture of open source book here http://aosabook.org/en/graphite.html

Would part of the solution be to add a check to the 'log to feeds process', to prevent, or at least warn users that they are about to create a process where the interval is less than the actual post interval. It wouldn't cure existing feeds, but may prevent future feeds from presenting this issue.

Paul

Yes something like that would be useful - emoncms would need to know somehow what the post interval is, perhaps the input module could record this as the nodes appear and then suggest the interval on feed creation?

I have now converted all PHPFiwa feeds on the first emoncms.org server to PHPFina which covers about half of the total.

All emoncms.org PHPFiwa feeds have now been converted to PHPFina which is one big step forwards in terms of simplifying and making emoncms.org more maintainable. It also means the same feed format is both supported on emoncms.org and the emonpi.

I've also made quite a bit of progress on the implementation of the on-the-fly averaging using PHPFina feeds mentioned above which will provide much of the functionality of PHPFiwa in a way that is more write efficient and with the benefit of the averaged layers being easily rebuildable from the base data. Its now possible to extract fully calculated averages covering up to 480 days with 10s data on any PHPFina feed on emoncms.org using the graph modules 'PHPFina average' tickbox as shown in the screenshot above.

Thanks for the reply Trystan. I wasn't familiar with that book. It is very good! I'm not much of a programmer myself so I'm prone to looking for existing solutions. However, I've started learning a bit recently and just had PR accepted to emonhub. :D I look forward to contributing more.

- Jacob

Emoncms.org has now got the input and feed api publicly accessible for reference:

https://emoncms.org/site/api#input
https://emoncms.org/site/api#feed

Id like to add to this a page on core concepts (i.e inputs, input processing and feeds), a page on generating kWh per day, week, month data and generating average data for temperature & humidity data.

Hi Trystan - making the api pages publicly available is a nice touch, the point of access may actually be better placed on the emoncms.org home screen with a short explanation for new users like the docs, github and install rather than on the navbar since it doesn't need to be on the navbar once logged in as there are links on the feeds page and inputs page for the pages prepopulated with an apikey and a new user without an account will only see the "logged out" navbar..

The "API" link looks a bit random when using just the readkey to view the apps, it does work and reveals the api docs when clicked but has very little relevance to the single target link. It's not really in keeping with the nice clean view you have only just achieved by removing the settings button for the readapi views.

Paul

Good point. I will do that, perhaps its worth having a link under setup, especially once I add a section on generating day, week, month, year kWh and temperature, humidity averages.

I've moved the api link under the setup section when logged in, at the moment it also shows in the menu bar when on the home page and logged out.

There is now a documentation page on creating daily kWh data available from the emoncms api pages: https://emoncms.org/site/api#dailykwh

Moving this to the new guide site: https://guide.openenergymonitor.org/setup/daily-kwh/