Archived Forum

Sounds very interesting.

I did have some similar thoughts about an automatic pairing of base and clients. My idea was to use either a cable or some sort of docking. It could be done with some pogo pins on the base station.

Another though was to implement the mesh network of the RFM69, but I don't know if the encryption is end-to-end or it need to decrypt/encrypt it at each node.

Regards

Kim

Thanks Kim, good to hear similar lines of thinking. Searching for mesh networking and RFM69 i came across this which looks interesting http://www.airspayce.com/mikem/arduino/RadioHead has anyone used this?

I've been looking at the control packet format item again, I think it would be good to allow control nodes to send data packets containing sensor data and receive a packet containing control parameters that have different payload structures.

At the moment following the emonhub config file approach decoders for receiving sensor data are defined like this:

[nodes]
    [[10]]
        nodename = livingroom
        names = temperature
        codes = h,

I've added in here the node and variable name properties. The corresponding structure on the sensor node looks like this:

    typedef struct {
        int temperature
    } Payload;
    Payload livingroom_sensor_values;

Lets say that livingroom node now has a relay attached to control a radiator actuator and that the control parameters that we need to set are: actuator on/off state and room setpoint, If it is sufficient to define a single type of control packet for a particular node that contains all the control parameters - which I think it is in probably most cases. We could have a second structure used for decoding packets received on the livingroom node

    typedef struct {
        byte state,
        int setpoint
    } PayloadCtrl;
    PayloadCtrl livingroom_control_params;

the emonhub config file would then need a separate section for describing packets that are outgoing, perhaps this would look like this:

[nodes_rx]
    [[10]]
        nodename = livingroom
        names = temperature
        codes = h,

[nodes_tx]
    [[10]]
        nodename = livingroom
        names = state, setpoint
        codes = B, h

When transmitting control data out from the basestation or any other node we could use the target node flag packet header as designed by jcw http://jeelabs.org/2011/01/14/nodes-addresses-and-interference/ or keep the header as the nodeid from which the packet was send and add another byte to the packet data itself including the target id.

interested to hear any preferences either way and other ideas for defining sensor node and control packets and whether to add the target id in the data part or follow jcw's recommendation of swapping the target with the sent from nodeid and then including the sent from in the packet data if needed

Hi Trystan.

As you know I'm working on "control" in emonhub currently. I too think there should be separate Tx and Rx settings for each node, There maybe some instances where the payloads can be the same eg passing current meter readings to a Wh counting node but I think these would be few and far between, in the majority of cases they will differ.

Within the sketch the "Payload" data type created by the typedef struct is often given a name ending in Tx eg emonTx, I have for a while been using "PayloadTx" and "PayloadRx" as the types and that seems to work well.

In the emonhub.conf however I was thinking more along the lines of splitting the node into [[[Rx]]] and [[[Tx]]] rather than listing the node twice so that thier could be some common data eg node name etc and the complete node could be cut and pasted in (or even just appended to the file by a script) as one "node" 

[nodes]
      [[10]]
           nodename = livingroom
           [[[Rx]]]
                 names = temperature
                 datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

and in the sketches

typedef struct {
      int temperature
} PayloadTx;
PayloadTx emonTx; 

typedef struct {
      byte state,
      int setpoint
} PayloadRx;
PayloadRx emonRx;

I think we should definitely use jcw's "flagged as source or target", as then the devices sketch only has to deal with relevant packets rather than every sketch needing the ability to weed out the packets destined for that device. Especially as it makes sending the targeted packets so easy by effectively done by just adding 64 to the node id at serial.write() time. 

I don't think this is useable other than across the rfm network as it will impose a 64 node (id) limit which granted is double the current rfm limit but a restriction all the same. Yes we could add a different value outside of the rfm network but that may just be confusing and it's not totally "human readable" either as a sum has to be done (albeit a simple one) My thoughts were to expand on the flag idea alittle and use a negative number so node 10 is from the node and -10 is to the node, the minus sign being the source or target flag (so it works with bytes, ints and longs)

Although I like the potentially easy set up of DCHP nodes, the reality is even with my home network I use fixed IP's to make networking easier and I suspect the same may apply to the node network, combined with a need to swap out sketches, calibrate, pass encryption keys etc I think it is probably more secure, reliable and simpler to connect the node to a pc at some point.

However I believe that process could be made easier, maybe a custom UI to abstract the user from the IDE, git conf files etc. Certainly interlinking the sketches and datacodes/node settings would be a start, Robert and I thought adding the datacodes to the 3 or 4 non-int-only sketches could be a first step, so users can copy and paste from the sketch to the conf.

I have since  thought this through a little more and would like to experiment with easily recognizable __VARIABLES__ used in sketches, so that a bash script could be initiated to configure a node device connected by FTDI. Something like a " config_node 10 " command would look at emonhub.conf for node 10

[nodes]
      [[10]]
          nodename = livingroom
          device = emonTx v3
          sketch = some_generic_sketch.ino
          [[[Rx]]]
                names = temperature
                datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

And would replace the __VARIABLES__ in the sketch file with the values given node id, device frequency, group and of course an encryption key whilst it is in text format. Then compile and upload it to the node from the same command. so once the user has the details in the conf one command does it all with some reassuring status messages.

There are many mays something like this could evolve and I think a webpage where a user could select a sketch to suit thier needs from a short list for the chosen device, select a node id, specify rfm settings, choose variables and data types etc and this would provide "cut and paste" or downloadable data for sketches and emonhub.conf or for a "makefile" to compile. I previously commented on using something like the packetgen webpage to generate a datacodes string, this could be extended to provide a complete nodes section entry with matching typedef structs.

As you are aware I've been experimenting with I²C also, the reason I mention this is with regard to your "polling" comments. The way my I²C works is emonhub sends out a "broadcast" to trigger the calculations in each node, it then collects a frame each node sequentially, as long as all the nodes are polled between the broadcasts the data packets are all for the same exact period in time despite being collected at different times, the nodes also continue to collect data towards the next period whilst waiting to have the last periods totals collected. This works well as the previous periods totals are not zero'd until they get an ack confirming emonhub has them, so if not collected the new intervals totals are added to last so you never lose data.

This could also work with rfm although you have the added power conservation complications with battery powered nodes, but it is possible to ensure they are the first to be polled so they can sleep and a wake up timer set by the original "broadcast" would wake the node moments before the next "broadcast" effectively resetting the devices time counting every interval so it should never get out of sync.

Hello Paul,

great!

emonhub conf format

[nodes]
      [[10]]
           nodename = livingroom
           [[[Rx]]]
                 names = temperature
                 datacodes = h,
          [[[Tx]]]
                names = state, setpoint
                codes = B, h

looks great.

happy to for jcw's "flagged as source or target". I agree more and more about DHCP as I think about it, it just becomes over complicated and that the better approach is making configuration easier. Really like your idea :

"I have since  thought this through a little more and would like to experiment with easily recognizable __VARIABLES__ used in sketches, so that a bash script could be initiated to configure a node device connected by FTDI. Something like a " config_node 10 " command would look at emonhub.conf for node 10"

that would be a very neat way of connecting the two, solving the nodeid, decoding, descriptions, potentially encryption key all in one.

brilliant!

I have experimented a bit with using "easily recognizable __VARIABLES__ "  to do a "sketch merge" and found it is quite easily done. There are many questions around best way to implement it.

However, the (basic) method I currently have working is to use 2 markers disguised as comments so as not to effect the normal use of the sketch. The markers are added before and after the "mergeable" variables such as the node id and typedef structs etc.When the sketch is "merged" the 2 markers are replaced with block comment markers so the existing variables cannot be seen and the new variables are added. This seems to work ok and is very simple.

This method would mean any existing sketches can easily be made into "mergeable" templates by adding 2 comments and slightly rearranging the variable positioning in the sketch to group the "mergable" vars away from the ones to be retained. There also needs to be some info made available about each participating sketch eg what variables are available etc so the user can choose from them and configure their own payloads.

I think this will mean that there will be a need for some mapping between the sketches variable names and the users preferred input names. The alternative is too use individual variable markers so that the users names are merged into the sketch, but this may prove confusing when diagnosing a sketch issue. I think retaining the generic variable names in the sketch would be better when discussing the sketch and they are not seen after compiling so names are not important.

The "prototype" version I have also creates a "filecopy" of the sketch using date and nodename as a filename (eg 20150120_kitchenNode.ino) and/or saves the file in a temp location in the required "inotool" structure and calls inotool "make" and "upload" and/or creates a "Makefile" in the sketche's sub-directory for "aduino_mk" to use. (I have had both these "outputs" working but have not actually compiled yet as I'm working on a windows PC)

Paul

Hello,

I have been reading through this post and i am finding it incredibly interesting and have some things to add.

Encryption
I say this first because it is related to other points below.

I was talking the other day to Adam and Michael about the DHCP service and how to implement encryption without having to plug the node to the computer to write the key (which makes the DHCP pointless). I have been researching an idea that Adam had and ended reading about Public Key encryption, and i think this might be the answer. If you read carefully the introduction in here , you will understand how it works. A brief summary: you use the public key to encrypt and the private key to decrypt, if you try to decrypt with the public key what has been encrypted with the public key it doesn't work. That is why it is secure. This is called asymetric encryption while the one with only one key to do everything is called symetric encryption.

Sitiutation: Node A wants to send something encrypted to node B: A asks B for its public key (via a non encrypted message) -> A gets B's public key (via a non encrypted message) -> A encrypts the message with B's public key and sends it (this message can only be decrypted with B's private key that only B knows)

Automatic ID assingment
Well, i still think it is quite a good idea. For me the reason for having DHCP isn't to avoid pluging the node and uploading something, the reason is to avoid having to mess up with the nodes ids and keep a map (if you have plenty of them) to ensure you don't put the wrong node id when updating the firmware. We may still get into this problem if we want to use any kind of unique identifier but a serial number feels different that the node.

The main problem for me was the encryption, but using the Public Key would solve the issue. And something else that this encryption would solve is how the base sends the node id back to the node. An option would be to put the serial number in the same message with the node id, so that the node gets it. Another option is that the base broadcasts to all the nodes: "Registration successful, your node id is klsddjfdskfjhdslkjghdfskjlgh" (node id encrypted with node public key sent when requesting the registration). The nodes which are in the registration process will listen to the message, decrypt the node id and if they get an integer it is for them, or just in case another registering node gets an integer after decrypting (i guess quite difficult to happen), the message can be "node:3" explode it and get the integer. This applies specially to RF networks. Using IP adresses makes it easier.

Automatic node detection and description
I am not completely clear about the aim of this one. I think it is a good idea that the node sends and "extended descriptor package" after registering and keep an up to date table with this information but i can only see it for information purposes. This applies to a sensing unit.

Polling powered nodes
I think there is a good reason in your first post Trystan to think it is a good idea: "The basestation could sequentially poll all the powered listening nodes avoiding packetloss", That could be enough.

Another one that Adam was saying is that for sensors on batteries it is better to be able to request the measurement when you want it or to be able to set the intervals easily (Not in the node).

Control module trigered by an input processor
This was not in the original list but i want to ask. The approach i see in the  "Open heating controller/thermostat scheduler" is to have something running continuously, and (sorry i think in therms of emonCMS and not emonView) i guess in emonCMS it would be a module with a cron task.

But, what if you want to do some control as a reply to some meassured data? When talking with Adam and Michael they said and i agree the scheduled tasks seems easier, every now and again the module checks all the feeds on its list and if there is change on the value it does something.

I had a look and found out that to add input processors you need to hack the input module, bad thing. But it would be a great feature to have a processor that triggers the controller of a module that analyses the input and then sends some control signals o request some other meassurements if needed.

This would be great to combine with Polling data from the nodes. Your "control module" in emonCMS requests the data from the node (trigered by the scheduler or even manually), the node replies and the input triggers the control module that analyzes the meassurement and decides if something needs to be done.

Graphical user interface for setting up nodes
What a great idea, i love it

Cointrol packets
I see this is a hot thing. I have nothing to say but i will follow what you do and tell Adam and Michael they should keep an eye on this. This is very important as they are developing control units so it would be great if they can use what will be the standard way for emonCMS to set up and send control signals. You are doing an amazing work.

All the best 

   Carlos

Looks good, but as per Twitter conversation of a few weeks ago.  It would be great to see compatability / integration with OpenTRV ( http://opentrv.org.uk ) built into the design.

The other thing is have you spoken to innovateuk ( https://interact.innovateuk.org ) they often have funding competitions which might be relevant.

Hello Carlos, great to hear your helping Adam and Michael at CAT, Adam mentioned you might be getting involved!

For people interested there is an ongoing project at CAT the center for alternative technology in Machynlleth Mid-Wales to develop monitoring and control technology for the renewables and heating systems on site, there was a blog post here http://openenergymonitor.blogspot.co.uk/2013/07/probably-best-student-pl... about the placement on which Michael from Aberystwyth University is on at the moment.

Thanks for sharing your thoughts on all this Carlos, Asymetric, public key encryption sounds like what we need, do you know if it would be possible to do this with the RFM69 using its built in AES encryption?

For the control module triggering, we where discussing here perhaps there is a need for being able to modularise input processing https://github.com/emoncms/emoncms/pull/280, perhaps we could implement some kind of hooks system that calls all module implementations of the hook function name when an event happens?

Hi,

Yesterday i spent a while doing a bit of research about the encryption. I  have started a new thread specific to this: http://openenergymonitor.org/emon/node/10164

AES is symmetric encryption, so it cannot do the work for us as we want. Anyway we have to keep it in mind as asymmetric encryption normally takes around 2 seconds to encrypt, that might be too much for our battery powered nodes. So maybe we can use the asymmetric encryption to retrieve the key to use with AES, but first we need to make asymmetric one work.

Cheers,

   Carlos

Further to the idea of Auto-configuring node's sketches from the data held in in emonhub.conf, which I've had some success with, I would like to throw a related idea out there.

Basically it's to extend the "emonhub.conf" to use an additional "hardware.conf" and "firmware.conf", not unlike the "boards.txt" and "programmers.txt" used by avrdude and the arduino IDE.

These two files will be easily read/edited and hold profiles for "emon" hardware devices and firmware sketches. The files would only come into play when loading sketches and configuring hardware, again much like the avrdude files. They would not be needed for normal running and could be installed as a full library or just cut and paste in the profiles you need.

This would allow a unique device name and sketch name to be held in the emonhub.conf along with the custom node data to allow a automatic "plug and upload" system to be used to ease the task, eliminating (or at least reducing) errors to boot.

Paul

I'm currently looking at the above issues (in isolation so far), I currently have working AES128 encryption (done in software) along with a protocol for the sensors to describe their data loads/packets to emonCMS.

I have an aim to get the sensor data encrypted, sensors nodes to automatically be discovered/added/configured and fairly "plug and play"

Still in experimental stages at the moment though - and I also have the problem of the initial "seed" key being placed into each node.

This is mainly driven by the hundreds of IoT devices out in the wild which are completely insecure, and I'm fed up of reading about them :-(

PS: Also looking at how difficult it would be to do OTA updates from emonBase to each sensor

Hello Stuart, that sounds very interesting, Glyn and I met up with Damon from OpenTRV last week where he discussed what sounds like a similar approach. Keen to learn more about what your doing.

Sure, I've taken some inspiration from the "MySensors" project, although they use NRF24L01 radio chips

https://github.com/mysensors/Arduino/blob/master/libraries/MySensors/MyH...

I'll try and describe what I'm up to, but its still "playing about" at the moment!