Archived Forum

Did you update your 'emoncms' database when you updated to v9?

The way that emoncms name's it's feeds changed in v9 and I'm wondering if the script failed because it couldn't find the source feed?

If you did update your database, I'm surprised that you are able to roll-back....

Paul

Hi Paul,

Yes I updated the database but I'm using a virtual machine which I can roll back using a snapshot. Before upgrading to 9.x I upgraded php from 5.2 to 5.6.  Following your previous suggestion I tried creating a new feed but receive the following error:

ERROR: Feed could not be created, undefined

Which I'm sure is very related.  Interestingly the feed is created although it is never updated. I'll find the cause of that issue and I'm sure the upgrade will work correctly after that.

Thanks for your help and I'm very sorry this it appears to be caused by an external factor.

Kev

btw as i prep to install emoncms i read the guide.... did i miss something or where do i free up the serial port?

G'day team -

I just wanted to raise two bugs that I have found with this version (they may have been in previous versions two)... they both specifically relate to the mysolarpv app, but may be applicable to the other apps as well..

1. If you try to link to the mysolarpv app like this:- http://emoncms.org/app?apikey=<apikey>#mysolarpv, it only works if you use the 'write' apikey - the read only apikey doesn't work.
2. If you try to link to the mysolarpv app like this  (by embedding):- http://emoncms.org/app?apikey=<apikey>&embed=1#mysolarpv, the responsive layout does not work for mobile devices - if you don't embed, it works just fine.

Cheers,

M

decided to stay on wheezy and followed this guide: https://github.com/emoncms/emoncms/tree/9.0/docs/RaspberryPi

stuck at this step:

git clone https://github.com/emonhub/dev-emonhub.git ~/dev-emonhub && ~/dev-emonhub/install
 

even tried to put sudo in front of it but i get this on the screen:

usage: emonhub.py [-h] [--config-file CONFIG_FILE] [--logfile LOGFILE]
                  [--show-settings] [--version]
emonhub.py: error: argument --logfile: can't open '/var/log/emonhub/emonhub.log': [Errno 13] Permission denied: '/var/log/emonhub/emonhub.log'

is that normal?

before this i got: 

sudo git clone https://github.com/emonhub/dev-emonhub.git ~/dev-emonhub && ~/dev-emonhub/install
Cloning into '/home/pi/dev-emonhub'...
remote: Counting objects: 98, done.
remote: Total 98 (delta 0), reused 0 (delta 0), pack-reused 98
Unpacking objects: 100% (98/98), done.
Cloning into '/home/pi/emonhub'...
remote: Counting objects: 1042, done.
remote: Total 1042 (delta 0), reused 0 (delta 0), pack-reused 1042
Receiving objects: 100% (1042/1042), 254.11 KiB | 244 KiB/s, done.
Resolving deltas: 100% (636/636), done.
Already on 'development'
update-rc.d: using dependency based boot sequencing
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-configobj is already the newest version.
python-serial is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 81 not upgraded.
useradd: user 'emonhub' already exists
[ ok ] Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.

so i guess the install was ok

but why that error?

last of log said:

2015-11-20 11:05:20.917|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:06:10.513|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:14:10.169|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:18:10.329|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:22:10.170|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:26:10.300|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:30:16.876|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:32:10.119|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:34:10.239|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:38:10.480|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:46:10.313|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:47:10.279|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:50:20.879|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:52:10.206|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:54:10.601|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 11:58:10.272|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:02:10.291|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:06:10.590|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:10:21.133|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:14:10.280|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:18:10.312|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:22:10.347|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:26:10.522|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:27:10.196|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a' 2015-11-20 12:30:22.616|ERROR|index.php|Invalid API key '4b8af356245e368cb44fc5727db0736a'
 

but i have just edited the hub conf file so it might be the last time that one shows up

nope it keeps going on... hmmm why does it say that the key is invalid? i copy&pasted it from the admin interface

just checked.... that api key is not even the one that is in the config and admin..... very odd

Try reloading the emonhub config:

sudo service emonhub restart

Paul

did that a few times...

loads and then a few sec after the same error pops up

will recheck but moving root fs to hdd as there is not enough room for my old data

the invalid api key was data comming in from a remote location i forgot... once i added the user back from backup it was ok

got stuff moved.... checked the config once more...

pi@GPS-raspi ~ $ sudo service emonhub restart
[ ok ] Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
pi@GPS-raspi ~ $ usage: emonhub.py [-h] [--config-file CONFIG_FILE] [--logfile LOGFILE]
                  [--show-settings] [--version]
emonhub.py: error: argument --logfile: can't open '/var/log/emonhub/emonhub.log': [Errno 13] Permission denied: '/var/log/emonhub/emonhub.log'

are there any files i could post copy off that might help solve the thing?

but emonhub is not running... i guess that its because it fails access on that log file...

the log file is not mentioned in the guide so have no clue what access permissions should be set

EDIT: when thinking of it next version of emoncms should have an config section for Emonhub... so almost every setting is done from emoncms and also something that checks the permissions on the emonhub log file like there is for the emoncms log file

EDIT2.... since emonhub is not running i guess that also the reason for API writes dont get through

EDIT3.. for some reason i cant edit my old dashboard the i imported via mysql... also dials do not show up, but graphs does and also show the old data. if i create a new dashboard i cant insert anything... 

i wonder what permissions there should be on /var/log/emonhub/emonhub.log ?

EDIT: chmod 777 did the trick... now emonhub can run and data comes in again

but dashboard is still not good

EDIT2: emonhub.log shows:

2015-11-20 22:21:11,796 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:21:11,801 INFO Opening hub...

but data comes through, thou it might sometimes take a restart or 2

if i use the api method to feed stuff like weather data that fails... 

EDIT3: i have changed the max node id setting from 32 to 3200... my weather and solar import uses id 100 and 200

EDIT2: emonhub.log shows:

2015-11-20 22:21:11,796 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:21:11,801 INFO Opening hub...

What did you expect it to display?

Paul

i forgot to incude what was above those lines:

2015-11-20 22:06:07,205 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:06:07,226 INFO Opening hub...
2015-11-20 22:06:13,270 WARNING Device communication error - check settings
2015-11-20 22:07:39,377 WARNING emonCMS couldn't send to server, Exception: Traceback (most recent call last):
  File "/home/pi/emonhub/src/emonhub_reporter.py", line 227, in _send_post
    response = urllib2.urlopen(request, timeout=60)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 401, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 419, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1034, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
timeout: timed out

2015-11-20 22:07:39,390 WARNING emonCMS send failure: wanted 'ok' but got ''
2015-11-20 22:08:39,712 WARNING emonCMS couldn't send to server, Exception: Traceback (most recent call last):
  File "/home/pi/emonhub/src/emonhub_reporter.py", line 227, in _send_post
    response = urllib2.urlopen(request, timeout=60)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 401, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 419, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1034, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
timeout: timed out

2015-11-20 22:08:39,728 WARNING emonCMS send failure: wanted 'ok' but got ''
2015-11-20 22:14:29,561 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:14:29,565 INFO Opening hub...
2015-11-20 22:14:33,696 WARNING Device communication error - check settings
2015-11-20 22:16:48,908 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:16:48,923 INFO Opening hub...
2015-11-20 22:16:53,097 WARNING Device communication error - check settings
2015-11-20 22:21:11,796 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-11-20 22:21:11,801 INFO Opening hub...

EDIT: has the api format changed since v8 ? this is what i do to import weather data:

$url = 'http://127.0.0.1/emoncms/api/post?apikey=*******&node=10&json={humidity:' . $relative_humidity . ',pressure:' . $pressure_mb . ',wind:' . $wind_kph . ',temp:' . $temp_c .'}';

EDIT2: and where did the dials etc go on the dashboard? i even created a new one and could not even insert text.... database is updated... 

If I'm not completely mistaken, yes this api will not be used anymore. I think I read it somewhere here in the forum. 

The way how the dials are getting their data has changed in 9.0+. So you need to relate them again to the feeds. I had to do that on 89 dashboards. Was a little bit annoying :-) And not all have been changed so far so you can expect to do that for the others too when they have been updated. Using the feedid instead of the feedname is far better from my point of view. It's worth changing that.

edit: I scrolled through the first page of this thread and found an entry submitted by dockarl on Fri, 23/10/2015 - 20:49.

It sounded similar to your issue and it was somehow still in my mind. Maybe this helps?

hmm... if you have a link to the replacement of the API it would be nice

about the dials... my problem is that i cant even make a new dashboard and place text

will look at the other post after breakfast

my import scripts fixed.... just replace /api/ with /input/

Can you use Chrome and then use the F12 button to show up the developer tools? 

Try creating a dashboard and look on the console log on the right bottom. Maybe it shows you an error message which can help to diagnose the issue. 

For sure I know it needs to add some columns to the existing dashboard table to store the dashboard background color etc. Maybe something is missing but the update & check mechanism does not recognize that something is missing.

Try create a new db and a new copy of emon 9.x. If this is working you can copy over the data (not structure etc.) to the new db and check if it is still working. I also had some issues not be able to create feeds. This step solved it for me.

i can create feeds, no issue there

also i can create a new dashboard, but i cant put anything on it... and i have hit update DB a lot without fixing it.

will try F12... lucky chrome is my religion :-D

http://imgur.com/mv3oJ2o

so an error i assume, but no idea whats wrong

what i did for dashboards was importing the mysql table from the old database, just so i did not had to redraw the whole thing again

strange thing is that the graphs i used show up, but not the dials, cant remember which type it was, most likely the once that can go from green to red etc

EDIT: phpmyadmin is making a copy of the database right now... takes some time as each table is about 50MB

The graphs have not been updated till now, only the dials ;-) That's why they are working.

One thing I see: Is there a specific reason why you are using 9.0RC2? Chaveiro has put up a 9.0x which is now 9.1 at the moment. Maybe you are missing quite some bugfixes because of that "older" version of it. I suggest to uses the actual one from here: https://github.com/chaveiro/emoncms/archive/9.x.zip

Maybe this solves some issues too as I can remember there were quiet some bugs in the dashboard area in the older versions.

9.0 is the official release version, 9.x is the development branch which may not be suitable for all users.

Paul

"9.0 is the official release version, 9.x is the development branch which may not be suitable for all users."

So that is why the dials are not working in my case? everything else works

except emonhub... sometimes it starts ok, other times it starts with the errors i showed a few steps up, in which case i have to do a restart until it starts ok

but can i go back and forth between the version and see how it works for me? ie without touching the database

"The graphs have not been updated till now, only the dials ;-) That's why they are working."

in my case its the other way round... graphs are working... NOT the dials

about the versions.... can it be true that i cant even make a new empty dashboard and put a simple text on it?

tried to empty the database and start over.... same thing... cant create a dashboard

Bo

"2015-11-20 22:06:13,270 WARNING Device communication error - check settings"

Have you defined the baud of the RFM2Pi in emonhub.conf? the intermittent starting of emonhub may be due to the serial baud auto-detection, it is known to be a bit hit and miss with certain rfm2pi firmwares.

"emonhub.py: error: argument --logfile: can't open '/var/log/emonhub/emonhub.log': [Errno 13] Permission denied: "

The install log for emonhub you provided clearly shows it was not the first time you run it, I have no idea what the consequence could be of running the script multiple times, But I suspect the reason for the emonhub.log permission issues are due to trying at least once to run the install as root using sudo, that will have cloned the repo with different permissions and that has caused an issue.

"WARNING emonCMS couldn't send to server, Exception: Traceback (most recent call last):........."

I have no idea what the root cause of these messages is, other than it is the response from emoncms seen by emonhub, for emonhub to be able to deliver data, emoncms must be error free unless you set $display_errors = false; in emoncms settings.php.

Also you can IF emonhub is functioning correctly and trying to post data correctly by setting "loglevel = DEBUG" in emonhub.conf.

Paul

how is baud defined??

problem with the log file for emonhub is that nowhere in the install guide it says what permissions it should have... once i set 777 it runs fine

as it is now i only have 2 issues

1. setting baud
2. create a dashboard

No, the graphs are using still the old way on how to get the feed data thats why they work with the old existing dashboards. The Dials are already using the new way and are therefore not working with the existing dashboards as there the feedname is expected and not an ID. :-)

I would give 9.x a try. i'm using it too and its working for me.

If a newly installed emoncms and newly created schema in mysql will not allow you to create a dashboard we have to analyze your environment maybe :-)

as it is now i only have 3 issues

1. setting baud
2. create a dashboard
3. test 9.x without change settings etc... ie can i switch a git command?

"how is baud defined??"

In emonhub.conf [interfacers] [[RFM2Pi]] [[[init_settings]]] com_baud = ???, the value will depend on the firmware installed on your rfm2pi, it will be 57600, 39400 or 9600.

"problem with the log file for emonhub is that nowhere in the install guide it says what permissions it should have... once i set 777 it runs fine"

There has never been a need for it before. Tthe log file and it's folder are (usually) created by the emonhub init script if they do not exist (this is done at every start to work with read-only installs to), I assume the problems you have are due to not using the installer as intended and since the file already exists but the permissions are not checked, the init script does not work as intended.

Paul

as for the log - just followed the install guide

will change emonhub settings to inlude baud

as for my dashboard... my idea was to click the dials and edit them and give then the feedname instead of the id

but it seems that i have to change to 9.X to be able to even create a dashboard in the first place

can i via git change version so i dont have to do setup from scratch?

Edit: got all my feeds sorted vis some witchcraft in phpmyadmin, so now i just need to do the dashboard

i want to give the 9.x branch a go... i assume there is no trouble with 2 dirs with emoncms

but how would i force this command to install in to a certain dir?

cd /var/www && git clone https://github.com/emoncms/emoncms.git
 

if not much mistaken this is the same command for both 9.0 and 9.x ?

i looked at both install guides and i cant see any difference.... 

my thought was to put 9.x in a dir called emoncms2 hence able to keep them as much seperated as i can

If you install v9.x it will make changes to your MYSQL database, which may cause issues if you decide to go back to 9.0.
There are discussions taking place between the OEM team about changing the branches and it appears at this time that 9.0 will remain as the default installation version, whilst a new branch (as yet unnamed will concentrate on development).
v9.x may of course be merged into 9.0 beforehand, but T&G are still looking at options.

Also, especially as you have struggled with so many problems so far, I would stay on 9.0.

But it's your choice!!

Paul

If you want to test it then also create a new db schema for it. Test a clean install and when everything is working you can copy over your data from the other db. 

That's how I did it with my move from 8.4 to 9.1. If you want to be extra save also make a copy of your data folder. If everything is working you can simply rename the folder name of emoncms and everything is as normally.

Sill, if you also have problems with 9.1 then you have definitely something going wrong on your environement.

@ Paul
Are you sure with that DB changes? I also used 9.0rc2 and can remember that it also changed a lot on the DB side. Of course I was using an older version (8.4) and this could be a reason. Is it not more likely that there are no changes when coming from 8.5? 

hehe

the problem is i dont have a dashbaoard.... nor can i create one...

a small downer as we call it here :-(

EDIT: i downloaded the zip with 9.x in to a seperate dir and now doing a DB backup.... 

in worst case i can just delete the new dir and revert any changes to the DB

@Andi, Bo is already on v9.0 having updated from 8.5 and is considering changing again to v9.x
I believe that the below screenshot is the database difference between v9.0 & v9.x
The difference between the two branches is this.

Just tried on my test Pi.

Paul

i tried to load 9.x in a different dir and point my browser to /emoncms2

i get an error 500

i would have stayed on 9.0 if the dashboard was working...

EDIT, the 500 error is also there if i move 9.x to /emoncms and 9.0 to another dir

EDIT2.... hmm... is the structure for the dashboard table the same from 8.5 to 9.0 to 9.x ?

are the reasons for me not being able to draw a dashboard because a fault at my end? or is V9 not ready for that yet?

There are a few glitches with v9 but many others, including, me, have installed/updated emoncms to v9 with minimal issues - check the forum.

However, I find it frustrating that you seem to compound issues by making too many changes, without building upon a stable working system. Look back upon your post's, where do you have a fully working system, before making random changes, importing data, moving to a external drive, switching branches, and so on. Firefox7518 has said the same above

Bumping, making repeated posts, and editing post's is frustrating the forum members, and perhaps why you are not getting the forum response that you hoped. Maybe instead of ' creating a diary of events' in the forum, you exhaust your own expertise, explore what the issue is and try and gather evidence before making a post?

Paul

my recent system is in fact made on a stable system

and i got emocms running... the first thing i noted was that i could not create a dashboard but did not give it much thought.. i tried to import my old dashboard and still stuck the same place

and i followed the install guide for v9 excatly as it was written... then i could not start emoncms, first due to a file permission issue and next to the fact i forgot to include coms speed

and this morning i discover that somebody has hacked my install due to my own stupidity of having port 22 open, but i look from the bright side... gives me another option to start from scratch... thou i expect no new outcome

The goal for you should be to install your operating system, and then follow the emoncms installation guide, with no deviations whatsoever.
At this stage, before you import data, before adding an external drive, or do anything else, make absolutely sure that everything works OK. If it doesn't, then search the forum to see if anyone else has had the same problem, and if you can't resolve it, then start a new forum thread describing in detail what the issue is.
 

Paul
 

that was the plan :-D

Thread closed!
If you have any issues around v9 - please start a new thread.

Paul