Archived Forum

The elephant in the room seems to be the lack of MQTT support from emoncms.

3 months ago, Trystan had a working model which appeared to work well, albeit there was no automatic daemon for it to function independently.

If MQTT was implemented, it would open the door for much greater integration with other data sources as it is a commonly accepted standard, and would help focus emoncms on it's core functionality, instead of being all things to all people.

My personal opinion is that if Trystan & Glyn could clearly define what emoncms 'is' and 'isn't', then development could concentrate in key areas, to make it a clearly defined, rock solid solution.

Paul

Hi Paul, I can't really disagree with anything you've said, although I do think the lack of MQTT is holding things back a bit it's not the whole picture, ideally emonhub and emoncms would use a similar standard and the bigger elephant is control, to which MQTT will play a large part. certainly emonhub needs considerable reworking to implement control properly (and maybe the same for emoncms).

​The reason I'm finding it difficult to get the control side going on emonHub is because it was built to collect data full stop. now I find myself trying to adapt it, the same could be said for other elements of the project so rather than implement MQTT optimised for collecting data and then try and bend it to fit control I would like to get a good grasp of what we need for control first. 

What is clear is the detail needs to be discussed openly with several perspectives so that time and effort isn't wasted on parallel and overlapping but incompatible ideas.

This is important but not directly linked to the point I was raising here though. the idea behind this is to provide a simple drop in facility that does away with thinking about MQTT, file structures and protocols ( as best we can) so that if someone has an idea or some weird sensor or website data etc they can take a blank file add a import statement to make it "emonhub compatible" and write anything that results in a string of values, no complexity, no conforming, no configuration files just get the idea to work, any sensor connected to the gpio or any website, cron, internal files, weird and wonderful calculations, the end result is passed to emonhub as a simple string of values via a socket and processed, which yes could mean publishing it to MQTT (in time), 

I was hoping this could speed up development of more exotic interfaces as users could create scripts without needing a knowledge of emonhub and the scripts (or modules) would be single files that can be passed around the forum like sketches until perfected and then maybe become a in-built emonhub_interfacer by adjusting it to fit.

Paul

Paul (pb66) has asked for members thoughts on this suggestion.

Out of courtesy to Paul, can we have a little feedback please? otherwise he may take the view that nobody wants it and therefore not worth his time developing...

Paul

Quote: "This is important but not directly linked to the point I was raising here though. the idea behind this is to provide a simple drop in facility that does away with thinking about MQTT, file structures and protocols ( as best we can) so that if someone has an idea or some weird sensor or website data etc they can take a blank file add a import statement to make it "emonhub compatible" and write anything that results in a string of values, no complexity, no conforming, no configuration files just get the idea to work, any sensor connected to the gpio or any website, cron, internal files, weird and wonderful calculations, the end result is passed to emonhub as a simple string of values via a socket and processed, which yes could mean publishing it to MQTT (in time), "

The way I would consider (I have not as yet had a look at the code, I have been lurking only for a while), would be similar to what you describe, but I would not have it write to a socket, there is a risk that a process might kill/hang the emoncms daemon.
What I have seen in the past, would be a daemon running the main emon get process. A second daemon would run, periodically spawning scripts from the 'plugin' dirt. These would import the main module, and then 'register' a name, and their outputs, descriptions and types. This would back off to a database store. The user would then do their stuff, and have two main functions, a get("module",item/all) which would retrieve an item (latest) from the db, where "module" is emoncms or plugin.name, and item is the variable.
Conversely, a put function, which DOESN'T take the module name, just the variable and data.
To display, a template could be stored in the plugin/html directory that the webserver parses and inserts the variables.
It is possible that this could also be powerful enough to publish variables across the network.
Personally, I would have a third daemon which is almost identical, but performs control functions.

Thanks Paul for asking about this! Paul Reed what do you think is the ambiguity about what emoncms is and isnt (before I try and second guess) I need to do some more work on communicating my recent developments and ideas. Been doing a lot, https://github.com/emonhub/emonhub/issues/134 but need to write more about it

(this is probably a bit off-topic for the discussion topic of modules sandbox though)

I like the idea of making it possible to add/remove modules

Creating a sustainable world is a aspiration, and which oem admirably contributes to, but not something which can be achieved by oem alone.

But it's unclear where the boundaries are (as pb66 has mentioned above), because emoncms started as data logging/display software, yet it has evolved into a multi-faceted system which is becoming ever-increasingly hard to develop, to maintain, and not reaching full potential in many areas.

I hoped (being the 10th initial member of this forum) that emoncms would eventually become a fully fledged graphical interface to energy data, and thereby educate users how to reduce wasted energy, but the focus appears to have shifted to energy models, add-ons, etc. which to the majority of users are disregarded.

Many of the additional functions could have been more admirably be performed by other software which is being developed within their own confines & expertise, but they need the conduit to communicate - hence my drive towards a common standard - MQTT.

​Paul

​



Thanks for the "bump" Paul.

@Trystan - Its always been a goal to add/remove modules one day, as in properly coded interfacers. The emphasis here is on the sandbox  idea. to try and make a simple way of trying stuff with hardcoded variables rather than a config file etc.

@coldpenguin - thanks for your input, this is targeting emonhub rather than emoncms, the use of sockets was to keep it simple as in a "testmodule" type interfacer could be used, so quite literally a test script would consist of an "from emonhub import modules" for example at the top and a "send_to_emonhub(string_of_values)" at the bottom what happens in the middle can be anything, as long as it doesn't crash the host machine it shouldn't effect emonhub or emoncms because of the strict(ish) conditioning/filtering/sanitation done by the "send_to_emonhub()" before dispatching the string to emonhub "modules" port (ModulesInterfacer).

If that's any clearer? probally not :-)

Paul

Feeding Hive data to emoncms

​'works with nest' functionality feasible?

How to simulate data sent from EmonTx to EmonHub in order to debug a reporter?

3 current examples that could use something like this, even if just during development until a working solution is found but one of scripts can remain in a prototype form perhaps.

Thanks @PaulReed, Il start another forum topic with my ideas for what I would like to include within emoncms. Where I see it going. I think its definitely going to get a lot better and yes making it more maintainable is crucial :)