Archived Forum

Hello guys, long time no see, a new family member and a job switch has kept me away from (most of) the fun stuff.

I have been recently given the opportunity to connect what I do for a living (design software,network and hardware systems) and what I do when I want to have fun (play with cool stuff like arduino and emoncms): specifically I have been asked to evaluate costs and benefit of building a complete Software as a Service infrastructure with the specific goal of monitoring fleets of photovoltaic installations.

The current number of installations is about 6K, all of which would be retrofitted with a monitoring system that is able to post sensors data to an url, initially:

• PV production  kWh counter
• HOUSE consumption kWh counter
• PV inverter status
• temperatures
• local energy diversion system state
• possibly wheather data

Since all of this data will probably have to be transmitted through mobile/gprs, sampling frequency will be between 1-15 min depending on gsm SIM data projected costs (no pseudo real time power reading, that is)

The system should be able to:

• gather sensors data
• display aggregated data at various levels
• react to events like inverter failure and trigger notifications
• integrate power production and consumption data with local weather conditions
• (in the future) act as the base for more detailed analitycs reasonings

I believe that, with the correct approach, a modicum of new modules, and a lot of forward thinking and coding, emoncms could be one of the basic building blocks for such an infrastructure.

Specifically, I would like to try and use emoncms as the part that:

• collects data with single photovoltaic install granularity :
  • one user per site (and relative write apikey)
  • nodes preconfigured based on templates (the monitoring hardware will be uniform through the sites, but will undergo different revisions and could log more/less data depending on connectivity availability)
• displays such data to at least three levels of users (with different granularity):
  • fleet owners 
    • ​ global data visualization (read only)
    •  partners data breakdown and visualization (read only)
    •  sites data breakdown and visualization (read only)
    • new partners/sites management
    • device/node management 
  • partners/installers
    • partners data visualization
    • sites data breakdown (read only)
    • device/node setup/assignment to specific sites
  • site users/end clients
    • data visualization (read only)
• reacts to events and forwards alerts

I know it would probably mean a lot of work, but in the eventuality that it moves from the thinking and prototipying stage to reality I would like to do it in  such a way that benefits all users of the emoncms platform.

Things that I think would be necessary to handle all the listed functionality:

Emoncms:

• superuser module: a module that allows an admin/partner to
  • create new users and assign them to the appropriate (fleet/partner/site) category
  • associate specific hardware nodes (see below) to sites
  • handle hardware node maintenance: an admin should be able to enable/disable posting of sensor data from a data node depending on pre-set conditions
• supernode module: a module that allows the definition of specific nodes in terms of inputs and related processes:
  • hardware node: should reflect different versions/configurations of the hrdware doing the actual monitoring
  • collector node: should define the logic with which to pre-aggregate sensors reading at the partner/fleet level
• custom frontend app module: should take care of correctly displaying data to various levels of users, would probably start from the mysolarpv view and expand on that

At the moment I think emoncms could handle all the requirements except the visual analytics part (displaying data aggregated on variables like site installs, weather condition, pools of sites/partners ) for which emoncms would still be the provider of pre-computed data.

Data collection would be performed through either emonhub or a custom mqtt interface to emoncms, still undecided on that and will depend on the result of stress tests if/when we come to that

I am using this post as a general brainstorming session, mainly for myself ... I have been thinking/tinkering on how to shape it up using emoncms for a week now, and will probably start to play with basic versions of the superuser and supernode modules in the next week.

I have read other posts that discussed strategies for adding hierarchies of users to emoncms, but I honestly don't see how a hierarchical model could work using the actual interface, specifically I see no way of easily aggregating data if the single users have control on feeds and processes, and see no way (other that with an external app) of displaying in a secure way limited amount of information

I also have been thinking a lot about system requirements, both in terms of storage and in terms of connectivity, will probably post my reasonings in another message

All modules developed for this purpose would naturally be shared with  the community, and this would apply to the frontend, if it turns out to be feasible to keep everything in sync with emoncms development.

Please keep in mind that as of now everything is still at the design level, and all of this exists only in my mind (and on a couple of test installations on AWS :) )

Wil keep updating as soon as I have more specifics.