Archived Forum

Ok so, learnt a bit of PHP; "necessity is the mother of invention";

I edited phpmqtt_input.php, removed the explode that assumes a CSV string and replaced it with json_decode.

//  $values = explode(",",$value);
//  $name = 0;
  $values = json_decode($value, true);
  foreach ($values as $k => $v) {
       $inputs[] = array("userid"=>$userid, "time"=>$time, "nodeid"=>$nodeid, "name"=>$k, "value"=>$v);
  }

So the mqtt feed now accepts valid JSON.  Tested with;

mosquitto_pub -t 'rx/21' -m '{"temperature": "13.2","humidity":69}'

Note the in the input page the Key becomes the first part of the JSON element.

I honestly have little idea what the rest of the script does but this change gets the JSON data into EmonCMS so it works for me but YMMV.  Unsurprisingly it also does not now accept CSV. (Need to remember to watch for changes on a git pull :) )

Cheers, Brian

Hi Brian, 

Having just read your latest post on the "emonHub mosquitto MQTT authentication" thread I am posting here to not go off-topic on that thread. I'm not an authority but I am interested in this too. I have seen your numerous other posts on this subject on several threads, each time I wonder what you mean and hope someone in the know will shed some light, but unfortunately not as yet.

My comments are based on my own understanding and if not accurate will hopefully draw out some correcting comments and maybe we can confirm, identify or clarify if there is an issue to fix, and what if anything needs addressing.

Firstly I do not believe this is actually anything to do with MQTT itself, your comment "The current MQTT support does not comply with the MQTT standard (such as it is) as EmonCMS does not accept valid JSON." doesn't seem to be correct in that, the MQTT standard (currently v3.1.1) has no reference to JSON.

MQTT is intended to be a lightweight and universal protocol for transporting a variety of payloads without preference, The various command structures are documented in detail but the message payload is left open, the "standard" states

"The Payload contains the Application Message that is being published. The content and format of the data is application specific. The length of the payload can be calculated by subtracting the length of the variable header from the Remaining Length field that is in the Fixed Header. It is valid for a PUBLISH Packet to contain a zero length payload." 

MQTT is just the "transport" and should have no effect on it's content, if it will only "accept" a certain format then it is not universal and if it is altering the content then it is not being faithful.

Many of the results I found when including "JSON" and "MQTT" as search terms led me to various forms of this "How to Send data as JSON objects over to MQTT broker" answer on stackexchange make me believe the MQTT payload CAN be JSON in a string form and it will then be down to the publishing application to ensure the payload is in "Valid JSON" IF that is the intention AND that the subscribing application is able to accept or parse the payload as JSON if that is what is intended.

So I believe the real "MQTT JSON" issue is that either "Valid JSON" is not being enforced at both ends or that JSON is not the intended format, neither of which are a fault of MQTT or a violation of the standard so the MQTT label on this issue may actually be confusing the issue and that's why there are no answers on this.

If the OEM project (or any other) choose to use csv or any other format that is fine if both ends are able to handle that input. There are 2 lines of thought in transporting data in the OEM project one is "concise" and uses a string of values in a predetermined order to conserve space and time eg JeeLib and emoncms bulk-posting etc, the other is more "verbose" using JSON type formats to include names and settings. Both are used in various places and I believe the csv v json maybe different stains of development and/or incompatible parts of the project being tied together, possibly from different stages of dev.

Just to throw the cat among the pigeons, my personal opinion is "why use JSON with MQTT at all?" Using a well formed MQTT topic tree you can "place" any value or setting precisely where it is supposed to be without any need for any payload formatting (braces and colons etc) or If the overhead of multiple publishes are a concern then use csv, don't pad the payload out with unnecessary characters and labels etc. As MQTT is intended to be lightweight JSON should be used for the more weighty http requests it was designed for and not MQTT, 

There are a few posts on the format of posting (JSON) to emoncms that do not involve MQTT so I suspect the emoncms API's need clarifying and/or revising. Ultimately how (well?) the API's are defined in emoncms will determine how (or if) we can post to emoncms (not the standards themselves), although I very much agree it is better for everyone if it is uses recognized standards for both HTTP(inc JSON) and MQTT methods of posting.

I hope I've understood your concern correctly and apologize if anything I've said is misleading, I just felt including the MQTT reference in this issue may actually be distracting away from any potential JSON issue in emoncms and the use of JSON in MQTT is perhaps a separate discussion regardless of whether it's done correctly or not.

Paul

Hi Paul,

I had done a bit more digging and discovered my assumption (mother of all &^%$ ups) was that this link to the IOTF was also the MQTT std which obviously it isn't.

It seems it is a wider issue than just OEM and it is a shame there is less concern for the good old RFCs which are the bedrock of the internet as we know it (imagine 100's of mail protocols).

Why JSON, well it is widely used for many different applications and it has a standard and can be validated. 

For OEM, if I use a JSON string as the input, the nodes are named so I do not have to guess (or work out) which one is which (and when there are 20 that takes some tme)!  It will accept decimal values (which I think the CSV does not but may be wrong) and is a standard format.  If I generate a JSON payload from  one system, it can be used by multiple systems if they all take that format.  if not I have to create multiple outputs one for each receiving system.

I think the IOTF is a good format.  It includes a timestamped version as well.  If all IOTF 'things' used this format life would be much simpler :)

If PHP can validate a JSON string perhaps the MQTT listener could check for valid JSON and if it fails assume CSV?

Cheers