Archived Forum

I just tried this, and the topic in settings.php is now "emon" whereas the topic in "emonhub.conf" is "nodes"

Doing a web update of EMONCMS didn't update emonhub.conf for some reason so I ended up with a non working system until I changed the topic in one of them. Anyone doing this may want to lay aside some time as some of the inputs may need another setup afterwards if they appear as new topics.

The new topic structure makes much more sense when interfacing with other things

Sorry Kevin, I missed a step from the post above re updating emonhub, so the emonhub.conf shares the same basetopic. - it's added now.

Paul

Hi Paul

I must have missed something. I have followed update instructions. I find I am publishing but not subscribing. I see no reference to base topic in settings.php other than 

// The 'subscriber' topic format is rx/* - where * is the emoncms input node number. 

see my post http://openenergymonitor.org/emon/node/12143

In case there have been further up dates I tried cd /var/www/emoncms && git pull

This produced this error.

Auto-merging docs/RaspberryPi/MQTT.md
CONFLICT (content): Merge conflict in docs/RaspberryPi/MQTT.md
Automatic merge failed; fix conflicts and then commit the result.

Regards

Ian

Hello Ian, when was the last time you did a git pull/update before this? We had an issue with branch merging yesterday.

Hi Trystan

Some time this morning.  Just tried again with a new error:-

pi@emon2:~ $ cd /var/www/emoncms && git pull

U       docs/RaspberryPi/MQTT.md
Pull is not possible because you have unmerged files.
Please, fix them up in the work tree, and then use 'git add/rm <file>'
as appropriate to mark resolution, or use 'git commit -a'.

Now I am totally baffled!

Regards

Ian

But I believe that you did a git pull yesterday, during the time we had an issue with branch merging. -http://openenergymonitor.org/emon/node/12143

Paul

Hi Paul

Yes, I did do a git pull yesterday. As I had the MQTT subscribing problem that I thought may be related to not being up to date I attempted another git pull this morning with the result:-

Auto-merging docs/RaspberryPi/MQTT.md
CONFLICT (content): Merge conflict in docs/RaspberryPi/MQTT.md
Automatic merge failed; fix conflicts and then commit the result.

I therefore made a later attempt today with the result:-

U       docs/RaspberryPi/MQTT.md
Pull is not possible because you have unmerged files.
Please, fix them up in the work tree, and then use 'git add/rm <file>'
as appropriate to mark resolution, or use 'git commit -a'.

I googled the error which resulted in the advice to run a git status:-

 cd /var/www/emoncms && git status
On branch stable
Your branch and 'origin/stable' have diverged,
and have 3 and 1 different commit each, respectively.
  (use "git pull" to merge the remote branch into yours)
You have unmerged paths.
  (fix conflicts and run "git commit")

Unmerged paths:
  (use "git add <file>..." to mark resolution)

        both modified:   docs/RaspberryPi/MQTT.md

no changes added to commit (use "git add" and/or "git commit -a")

I reasoned that my docs/RaspberryPi/MQTT.md must be different so I deleted it but I still have the same error. Is there a git command that will bring me up to date regardless of the files my end?

Regards

Ian

There will be a way to fix this via git commands, but if you want to take the easy route, rename your emoncms folder to oldemoncms, then git clone emoncms, so you have a completely clean download.
Don't forget to create and edit your settings.php file.
Once you are back working, delete the oldemoncms folder.

Paul

Thanks, I will take the easy way out!

Ian

Hi Paul

Git issues sorted and I am up to date and now with the latest settings.php.

 // Used with scripts/phpmqtt_input.php
                          'userid'=>1,
                          'basetopic'=> 'emon'
                          );

I am publishing from emoncms to mqtt with no problem at all. However I still see no subscribed inputs. I know the topic emon/11/dcahnew is being published from NodeRed as I can see it on a third party app.

I checked services with sudo service --status-all.

 [ - ]  mountnfs.sh
 [ ? ]  mqtt_input
 [ + ]  mysql
 [ + ]  networking

mqtt_input appears and I understand that the fact it has [?] is not important.

The only thing I am not sure of is userid which I understand means the first user. I have 2 users the second one being a test. Can you check the curent user id number in emoncms? 

Ian

Hello Ian, You can check the user id with emoncms/user/get.json. It should  really be printed on the user page.

For anyone else experiencing the "Pull is not possible because you have unmerged files." error, Pauls advice is probably the safest solution. If you dont have any branches or changes locally this will also  work:

git fetch --all
git reset --hard origin/master

This issue will only affect systems updated in the last week or so I think, I just updated my system from an earlier point and the update went smoothly.
 

Hi Trystan

Thanks. That confirmed I am user 1 so there must be some other reason I am not subscribing.

I just noticed this line in  phpmqtt_input.php   

$mqtt_client->connect("localhost", 1883, 5);

Is it supposed to get the mosquitto server address from here or settings.php.? I did try putting my server url in but subscribe is still not working.

My server is not localhost as it is on a different Pi.

EDIT later today

I managed to stop mqtt_input so I could start phpmqtt_input.php manually. This is the result:-

pi@emon2:/var/www/emoncms/scripts $ php phpmqtt_input.php
Subscribing to: emon/#
I got code 0 and message Connection Accepted.
Subscribed to a topic
emon/11 438
set_timevalue
emon/11 438
emon/11 438
emon/11 438

Now I see the input so the problem is in the mqtt_input boot startup. I must have messed up somewhere.

How do I reverse this instruction sudo update-rc.d mqtt_input defaults?

Ian

Hello Ian

I think just re-running the mqtt_input installation steps should do it:

cd /etc/init.d && sudo ln -s /var/www/emoncms/scripts/mqtt_input
sudo chown root:root /var/www/emoncms/scripts/mqtt_input
sudo chmod 755 /var/www/emoncms/scripts/mqtt_input
sudo update-rc.d mqtt_input defaults
 

Good point about the mqtt server and port settings, Il look into changing that.

Hi Trystan

I think I have found the problem but not how to completely solve it. Beyond my Linux skills. The instructions in mqtt_input diifer from the documents here:-

https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

mqtt_input has these lines:-

Installation
# 1) Move this to /etc/init.d/ folder
# 2) sudo chown root:root /etc/init.d/mqtt_input
# 3) sudo chmod +x /etc/init.d/mqtt_input
# 4) sudo update-rc.d mqtt_input defaults
#

MQTT.md has these lines:-

cd /etc/init.d && sudo ln -s /var/www/emoncms/scripts/mqtt_input
sudo chown root:root /var/www/emoncms/scripts/mqtt_input
sudo chmod 755 /var/www/emoncms/scripts/mqtt_input
sudo update-rc.d mqtt_input defaults

I am not sure which is correct but having deleted mqtt_input in /etc/init.d/ folder and trying both I have ended up with a working system if I run sudo service mqtt_input stop followed by sudo service mqtt_input start from SSH into server after a reboot. So it still needs a little tweak.

Ian

Both versions have the same effect and should work equally well, but...

When I wrote the MQTT.md guide, I decided to symlink the mqtt_input script instead of moving it to init.d because if we later made improvements to the script, then a 'emoncms git pull' will automatically update it, otherwise, if we actually moved out of the emoncms directory it would not be updated.

I'll suggest that both guides are aligned.

Paul

Hi Paul

That makes sense. What are the commands to stop and start the service when using the symlink?  Although I am now working I seem to have to start mqtt_input manually to get it working.

What is the best way of reverting to the symlink start as I have probably made a mess of init.d ?

Ian

I have the same problem as Ian.

For some reason during the boot process the mqtt_input process exits and I have to manually restart it. I assumed on my system that it was a conflict between Openhab, Owserver and Samba. A workaround is to create an Openhab rule that restarts the service after a delay of 120 seconds from the first entry in the Syslog. Restarting it earlier doesn't work.

Is there any way of turning on logging in the MQTT Input process so I can see where it is failing?

Ian, simply delete the mqtt_input script from the /etc/init.d/ directory, then follow the MQTT.md guide to set up the symlink.

Paul

Kev, can you remind me please;

1. Which operating system you're using
2. How you built your system - from an image or from scratch
3. Are you full write or low write

Paul

Paul,

Raspberry Pi version of Jessie
Built from Scratch following https://github.com/openenergymonitor/emonpi/blob/master/docs/SD-card-build.md with the EMONPI LCD Screen left out to allow me to use all my GPIO Pins and Openhab running in a folder on /home/pi/data. No Node Red
I'm running the Low Write version

I'm also using the latest MQTT arrangement

I also have a spare Pi to test with

Cheers

Kevin

That installation guide is being actively prepared by Glyn, so he's probably best placed to answer this.

Paul

OK, will he see this post or should I PM / email him?

Glyn is pretty active in the forum at this time (it's too cold to climb mountains!!) so I'd be surprised if he didn't pick this up.

Paul

OK, thanks

I'm here! It is indeed too cold to climb mountains (outside at least!). I've just pushed an update to improve the logging for MQTT input script.

https://github.com/emoncms/emoncms/commit/b970b0c26ca53c17a96ed9b88293a6...

The script now logs to the main emoncms logger which can be viewed Admin panel in Emoncms. If you have the log level set to WARN which is the default for emonPi you should see a log entry from MQTT input to notify that it's connected to your MQTT server. If you change loglevel to 'info' in settings.php (then restart the MQTT input service) you should see every MQTT message that the MQTT input script subscribes to posted to the log. 

You will need to git pull then restart the service to pull in latest update. 

BTW: the emonPi build guide is still a work in progress as stated at the beginning. It's very much in a state of flux at the moment. E.g the current migrate / backup system is about to be replaced by the Emncms backup module: https://github.com/emoncms/backup  

Thanks Glyn, I'll pull in the update and report back here. If you have a chance can you take a look at my other current issue at http://openenergymonitor.org/emon/node/12146

Update added, log level set to 1 and server rebooted. I can see it start, then the messages stop with no reason, then they start again when openhab restarts the service. I can't see any clues why it exits, just an empty time between line 117 & 118

The log seems to show the script working great. I'm not sure if this is making a difference but I would recommend reducing the number of decimal places you have the temperature measured to e.g.  ( 26.66667). When does the crash happen? It seems to be working until the end of the log

It's the around 90ish second gap that's the issue, around the time Openhab starts, 70% of the time, MQTT updates stop, the updates and log entries only start again after Openhab issues "sudo service mqtt_input start" from a rule.

The incoming temperatures are from Owserver via Openhab. I only need them to 1 decimal place so happy to turn them down if you can tell me how! 

Hi

I have reverted to the MQTT.md guide setup.

Something very strange on my setup.

I rebooted and there was no input updating so it was not working as expected. I looked in the log and saw:-

2016-02-09 09:37:40.556|WARN|phpmqtt_input.php|Starting MQTT Input script

So the script was starting but no subscribing!  So nothing to lose I did a:- 

pi@emon2:~ $ sudo service mqtt_input stop
pi@emon2:~ $ sudo service mqtt_input start

This time the input did start updating!  This is exactly what I found earlier, I have to do a stop and restart after a boot for php_input to work. Could this be a timing issue?

Log attached

@sheppy, you could use something like nodeRED to reduce the number of decimal places then post to a fresh MQTT topic. 

@Ian that's very strange

Could you try and reset the MQTT input script service:

sudo rm /etc/init.d/mqtt_input

cd /etc/init.d && sudo ln -s /var/www/emoncms/scripts/mqtt_input
sudo chown root:root /var/www/emoncms/scripts/mqtt_input
sudo chmod 755 /var/www/emoncms/scripts/mqtt_input
sudo update-rc.d mqtt_input defaults

Hi Glyn

I had already tried that but have just done it again.

Exactly the same result. No mqtt inputs until I stopped and started the service. Latest log attached.

I could overcome it with a delayed script stopping and starting the service I guess.

Ian 

Could it be that the mqtt_input service is starting before the MQTT broker ?

Try adding "mosquitto" as a start dependency in /etc/init.d/mqtt_input

# Required-Start: $remote_fs $syslog mysql mosquitto

Sorry I haven't been able try it myself as I don't have a setup at a stage suitable at the moment.

Paul

Hi Paul

Broker is on a different Pi and already running. I did wonder if the network was taking a while to connect.

Could I add a delay or a requirement for network to be connected in Paul's (Reed) mqtt_input ?

Ian

This issue appears new, and I wonder if it's only surfaced since changing from the bluerhinos extension to the Mosquitto-PHP extension.

To confirm this, are you able to identify when you last had a fully working system, (what release/date of emoncms worked) and what you did that appears to have broken it?

If we can establish the timeframe of when it worked and when it didn't, we can look more closely of what changes were made between the two.

Paul

You can add " $network " instead of mosquitto, but that may not be a fail safe solution as it will set the order of the processes starting so that the mqtt_input service is started after the network services but may trip up if those services haven't established a working connection, eg wifi up and running but no network connection available. 

However the "delay" offers no guarantees there will be a connection either, ideally the mqtt_input service should start the php script and that script should handle both the connection, ie it should try and connect if it cannot it should log an error message once, wait and try again in a loop until it connects or gets restarted. The alternative is to add a "service mqtt_input restart" to the ifup routine somewhere but that starts to get messy IMO.

Perhaps the mqtt_input could ping the mqtt broker ip and loop until it gets a response before continuing to connect, a single log message "waiting for response from broker" before the loop could indicate a connection failure if not followed by a "successful connection" message.

Either would work, confirm before connect or retry after failure, I have to say I'm surprised the php mqtt library doesn't handle this automatically, I hope it does reconnections !

Try the $network dependency first though as the lib may well be able to handle the connection if the services are started in the right order perhaps.

Paul  

Ah yes, good point we have not tested on external MQTT servers. Trystan is looking to add more checking into the script to try and reconnect periodically if connection to MQTT server in unavailable  

Hi Paul (R).

Can't help identify when the problem started. I have a main server running 8.4 and that has never had a problem with mqtt. I have set up a new server with 9.3 basically to test everything before upgrading. As it happens I only started implementing mqtt on the 7th, the day after you announced the changes so I have never had any version of 9 running mqtt.

Ian

I have this issue on startup using a local mosquitto instance. One maybe 2 updates get through from EMONHUB before the MQTT_input usually, but not always, terminates so everything is maybe working at the beginning. Taking a read through syslog, it seems that the network does a lot of changing as it comes up related to DHCP leases, I wonder if this is to blame? I think the way DHCP works was changed in Jessie as static ip addresses are no longer set in /etc/network/interfaces.

EDIT: looking into this further, I've just set a static ip and that has got rid of the syslog ip changes. Mosquitto is started before mqtt_input. Unfortunately it still terminates with no logging after a few updates get through with no obvious reasons, on this test image I have mosquitto bridged to the mosquitto instance on my live system so the ethernet network has to be working to receive any updates. Both my Pi's are hard wired.

Could mqtt_input have some periodic if service failed then restart logic added? As I get the odd feed update pass through could the network be present for a short time before the script fails, or is it messages that are stuck in mosquitto from the last reboot that I'm seeing initially?

I've updated the script to have an auto reconnect if the connection is closed. Available in the latest commit to master:

https://github.com/emoncms/emoncms/commit/2a69df8ca34fb94d6b20e131270d49...

2 successful reboots in on the test system and it looks like you nailed it. Will update my live system shortly and update this post.

Sadly the early success didn't make the transition to my live system. Looking at the log it seems to have reconnected once after a second and then not tried again. My experience with using Openhab to restart the service suggests it needs to be around 120 seconds after the initial booting syslog entry.

2016-02-09 23:03:16.268|WARN|phpmqtt_input.php|Starting MQTT Input script
2016-02-09 23:03:17.559|WARN|phpmqtt_input.php|Subscribing to: emon/#
2016-02-09 23:03:17.560|WARN|phpmqtt_input.php|Not connected, retrying connection
2016-02-09 23:03:17.561|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2016-02-09 23:03:17.561|WARN|phpmqtt_input.php|Subscribed to topic: emon/#

Hi

Updated the script this morning but still the same problem.

Have stop and start mqtt_input to subscribe.

Ian

Kevin: it should keep trying to reconnect, every 5 seconds. But it looks like it did manage to connect but perhaps not receiving anything? Once running, if your stop/start mosquitto does it disconnect and reconnect ok for you?

Hi Trystan

To stop and start mosquitto I actually rebooted the Pi running mosquitto.

Version 9.3 emoncms did NOT reconnect. Just out of interest I am running version 8.4 on a 3rd Pi that is using the same mosquitto server and that reconnected without any trouble.

As previously reported if I stop and start the mqtt service it works fine. 

I am running Jessie on the Pi with 9.3 and Wheezy on the Pi with 8.4.

Also I believe the 8.4 is using bluerhino's library as I got that running in May 2015.

Ian

Is it possible to subscribe to an additional base topic either now or in the future with the new setup? I currently use Openhab to translate the topic and have discovered its double posting.

EDIT: I got around it by adding a bridge into mosquitto.conf to translate the topic.

@TrystanLee yes, stopping and restarting mosquitto does cause it to reconnect when all is working, so this seems to be a start up problem.

In the attached log:

the first set at 21:57 is as it reboots
the second set at 21:59 is after openhab issues "sudo service mqtt_input restart" from a rule
the last set around 22:40 is when I manually issued "sudo service mosquitto stop" followed by "sudo service mosquitto start" about a minute later

Hi

Pleased to report that after pulling in latest changes, including an update to phpmqtt_input.php, that if the remote mosquitto server is shut down and restarted version  9.31   2016.02.13 reconnects to php automatically.

Thanks for all the effort.

Regards

Ian

Hi there,

Does someone have a guide or some info on how to get the mqtt service to run under windows?

Kind Regards,

Stephan Rossouw