Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

I'm running the SD image from many years ago using usb HD for everything other than boot.  

Linux emoncms 3.6.11+ #538 PREEMPT Fri Aug 30 20:42:08 BST 2013 armv6l GNU/Linux

Walked thru the upgrade guide for v9 but have hit a few issues.

Inputs exist, Feeds are stuck on loading - clearning cache doesn't correct this

Inputs - indicated feeds are missing or deleted, yet if I look in 8.3.6 the feeds exist

installed emonhub which removed rfm12piphp so I can't roll back and I'm not getting any updated info - how can check this?

redis-server is installed

Whatever you need , let me know and I will dig it up.

Thanks

 

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Which upgrade guide did you use?
A common issue is forgetting to set the timezone in Setup > My Account.
This was my experience of upgrading - http://openenergymonitor.org/emon/node/11636

Paul

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hi Paul,

Thank you for the response.

I followed:https://github.com/emoncms/emoncms/blob/stable/docs/RaspberryPi/install_Wheezy.md

I had no issues until:

Edit the cmdline.txt file:

sudo nano /boot/cmdline.txt

by changing the line to - dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

I have no cmdline.txt file in the boot partition

I have now also followed your additional post. No change

emoncms.log:

2016-01-19 13:38:22.680|ERROR|feed_model.php|EngineClass() Engine id '1' is not supported.

Feeds just sit at loading and if I look at a feed under the input it indicates they have been deleted or missing.

mysqldb:

mysql> show tables;
+----------------------+
| Tables_in_emoncms    |
+----------------------+
| app_config           |
| applist              |
| converttotimestore   |
| dashboard            |
| device               |
| energy               |
| event                |
| event_settings       |
| feed_12              |
| feed_13              |
| feed_15              |
| feed_16              |
| feed_30              |
| feed_4               |
| feed_46              |
| feed_5               |
| feed_6               |
| feed_7               |
| feed_8               |
| feed_9               |
| feeds                |
| input                |
| mqtt                 |
| multigraph           |
| myelectric           |
| node                 |
| notify               |
| openbem              |
| openbem2             |
| openbem_projects_v3  |
| openbem_scenarios_v3 |
| packetgen            |
| raspberrypi          |
| rememberme           |
| report               |
| schedule             |
| users                |
+----------------------+

Thanks!!  Look forward to how I fix this :)

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Additional info:

Server Information

Emoncms Version 9.3 | 2015.12.30
Server OS Linux 3.6.11+
Host emoncms emoncms (127.0.1.1)
Date 2016-01-19 09:17:22 MST
Uptime 09:17:22 up 2:59, 1 user, load average: 0.11, 0.25, 0.47
HTTP Server Apache/2.2.22 (Debian) HTTP/1.1 CGI/1.1 80
Database Version MySQL 5.5.46-0+deb7u1
Host localhost (127.0.0.1)
Date 2016-01-19 09:17:22 (UTC -07:00‌​)
Stats Uptime: 10736 Threads: 1 Questions: 33376 Slow queries: 0 Opens: 85 Flush tables: 1 Open tables: 78 Queries per second avg: 3.108
Redis Version 2.4.14
Host localhost:6379 (127.0.0.1)
Size 402 keys (630.93K)Flush
Uptime 0 days
PHP Version 5.4.45-0+deb7u2 (Zend Version 2.4.0)
Modules Core   date   ereg   libxml   openssl   pcre   zlib   bcmath   bz2   calendar   ctype   dba   dom   hash   fileinfo   filter   ftp   gettext   SPL   iconv   json   mbstring   session   posix   Reflection   standard   shmop   SimpleXML   soap   sockets   Phar   exif   sysvmsg   sysvsem   sysvshm   tokenizer   wddx   xml   xmlreader   xmlwriter   zip   apache2handler   PDO   curl   dio   mcrypt   mysql   mysqli   pdo_mysql   redis   mhash  

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Have been able to get rfm12 on the pi receiving data correctly now...

needed to add the com_baud = 9600 to the setup on emonhub.conf

Port /dev/ttyAMA0

Press CTRL-A Z for help on special keys

 10 141 1 41 1 33 0 34 13
 9 16 0 16 0 26 0 138 0 0 0 0 0
 20 245 5
 10 142 1 38 1 34 0 34 13
 10 148 1 38 1 33 0 34 13
 20 245 5
 9 16 0 16 0 26 0 136 0 0 0 0 0
 10 149 1 38 1 33 0 34 13
 28 176 0 0 0 41 1 27 0
 20 245 5
 10 140 1 41 1 33 0 34 13
 9 16 0 16 0 26 0 134 0 0 0 0 0
 26 174 0 0 0 79 1 27 0
 10 138 1 41 1 33 0 34 13
 30 176 0 0 0 13 1 27 0
 20 245 5
 10 135 1 38 1 33 0 34 13
 9 16 0 16 0 26 0 131 0 0 0 0 0
 27 162 0 0 0 56 1 27 0
 10 141 1 41 1 33 0 34 13
 20 251 5
 10 144 1 37 1 34 0 34 13
 9 16 0 16 0 26 0 132 0 0 0 0 0
 20 251 5
 10 145 1 37 1 34 0 34 13
 29 27 0 0 0 113 2 27 0
 10 145 1 39 1 33 0 34 13

 

no change however within emoncms, these values are not getting processed

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

additional info:

 

2016-01-19 10:56:46,428 WARNING emonCMS send failure: wanted 'ok' but got '<br />
<b>Fatal error</b>:  Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms_new/Modules/feed/feed_model.php:71
Stack trace:
#0 /var/www/emoncms_new/Modules/feed/feed_model.php(532): Feed-&gt;EngineClass('1')
#1 /var/www/emoncms_new/Modules/process/process_processlist.php(215): Feed-&gt;insert_data('14', 1453224885, 1453224885, 15.25)
#2 [internal function]: Process_ProcessList-&gt;log_to_feed('14', 1453224885, 15.25, Array, Object(Process))
#3 /var/www/emoncms_new/Modules/process/process_model.php(48): call_user_func_array(Array, Array)
#4 /var/www/emoncms_new/Modules/process/process_model.php(88): Process-&gt;__call('log_to_feed', Array)
#5 /var/www/emoncms_new/Modules/process/process_model.php(88): Process-&gt;log_to_feed('14', 1453224885, 15.25, Array)
#6 /var/www/emoncms_new/Modules/input/input_controller.php(165): Process-&gt;input(1453224885, 1525, '2:0.01,1:14,16:...', Array)
#7 /var/www/emoncms_new/core.php(64): input_controller()
#8 /var/www/emoncms_new/index.php(146): controller('input' in <b>/var/www/emoncms_new/Modules/feed/feed_model.php</b> on line <b>71</b><br />

 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

confirmed in emonhub.conf that the API key is correct? but where is this coming from --apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y (below)

Also debug level to emonhub logging:

2016-01-19 12:00:34,563 INFO emonCMS sending: http://localhost/emoncms_new/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1453230021.91,10,312,318,33,3362],[1453230022.41,9,16,16,26,18,0,0],[1453230023.2,26,194,0,249,27],[1453230023.44,20,1600],[1453230024.17,10,302,316,33,3362],[1453230029.93,10,303,316,33,3362],[1453230031.54,9,16,16,26,18,0,0],[1453230032.47,20,1600]]&sentat=1453230034
2016-01-19 12:00:35,461 WARNING emonCMS send failure: wanted 'ok' but got '<br />
<b>Fatal error</b>:  Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms_new/Modules/feed/feed_model.php:71
Stack trace:
#0 /var/www/emoncms_new/Modules/feed/feed_model.php(532): Feed-&gt;EngineClass('1')
#1 /var/www/emoncms_new/Modules/process/process_processlist.php(215): Feed-&gt;insert_data('1', 1453230022, 1453230022, 312)
#2 [internal function]: Process_ProcessList-&gt;log_to_feed('1', 1453230022, 312, Array, Object(Process))
#3 /var/www/emoncms_new/Modules/process/process_model.php(48): call_user_func_array(Array, Array)
#4 /var/www/emoncms_new/Modules/process/process_model.php(88): Process-&gt;__call('log_to_feed', Array)
#5 /var/www/emoncms_new/Modules/process/process_model.php(88): Process-&gt;log_to_feed('1', 1453230022, 312, Array)
#6 /var/www/emoncms_new/Modules/input/input_controller.php(165): Process-&gt;input(1453230022, 312, '1:1,5:4,16:5,11...', Array)
#7 /var/www/emoncms_new/core.php(64): input_controller()
#8 /var/www/emoncms_new/index.php(146): controller('input')
#9 {main in <b>/var/www/emoncms_new/Modules/feed/feed_model.php</b> on line <b>71</b><br />

 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Reinstalled rfm12piphp, stopped emonhub and old install is working, inputs are reporting correctly.

--------------------------------------------------------------------------------------------------------------------------

Install rfm12piphp gateway service:

sudo cp /var/www/emoncms/Modules/raspberrypi/rfm12piphp /etc/init.d/ sudo chmod 755 /etc/init.d/rfm12piphp sudo update-rc.d rfm12piphp defaults

Start the service with:

sudo /etc/init.d/rfm12piphp start

The following commands can also be used to control the service:

sudo service rfm12piphp status sudo service rfm12piphp start sudo service rfm12piphp stop sudo service rfm12piphp restart

Important! You will need to re-run this installation procedure in full every time you pull new changes from github. If you do not re-copy the service script and restart it, your installation may suffer from impaired functionality.

--------------------------------------------------------------------------------------------------------------------------------

Not sure what next?   Thanks

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

1) You cannot run both rfm12piphp & emonhub on the same system, as the two are not compatible.
v9 emoncms only requires emonhub & which replaces rfm12piphp..

2) Look at your original installation - did you have any Timestore feeds? If so, they are not compatible with v9 and need to be converted (or deleted).

3) In your v9 settings.php change;
$display_errors = true;     to
$display_errors = false;

Paul

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hi Paul,

I know they can't run together but I hat losing time/data since I can't easily overwrite the lost time with a previous similar day.

looks like most of my feeds are engine 1.

mysql> select * from feeds;
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+
| id | name                   | userid | tag  | time | value | datatype | public | size    | engine | processList |
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+
|  1 | ct1-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3883744 |      1 |             |
|  2 | ct2-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3241104 |      1 |             |
|  3 | ct3-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3238912 |      1 |             |
|  4 | ct1-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  5 | ct1-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   32244 |      0 |             |
|  6 | ct2-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  7 | ct2-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   34958 |      0 |             |
|  8 | ct3-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  9 | ct3-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   20988 |      0 |             |
| 11 | Power                  |      1 | NULL | NULL |  NULL |        1 |      0 | 3882976 |      1 |             |
| 12 | Power-kwhd             |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
| 13 | Power-histogram        |      1 | NULL | NULL |  NULL |        3 |      0 |   35283 |      0 |             |
| 14 | monitor-temp           |      1 | NULL | NULL |  NULL |        1 |      0 |  307872 |      1 |             |
| 15 | monitor-temp-hist      |      1 | NULL | NULL |  NULL |        3 |      0 |    2841 |      0 |             |
| 16 | Power-average          |      1 | NULL | NULL |  NULL |        3 |      0 |    2698 |      0 |             |
| 19 | TH-30-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      1 |             |
| 18 | TH-29-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      1 |             |
| 20 | House-Temp             |      1 | NULL | NULL |  NULL |        1 |      0 |  216428 |      1 |             |
| 21 | Garage-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  216432 |      1 |             |
| 22 | Garage-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |  216424 |      1 |             |
| 23 | House-Humidity         |      1 | NULL | NULL |  NULL |        1 |      0 |  216420 |      1 |             |
| 24 | TH-28-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      1 |             |
| 25 | House-2F-Temp          |      1 | NULL | NULL |  NULL |        1 |      0 |  216356 |      1 |             |
| 26 | House-2F-Humidity      |      1 | NULL | NULL |  NULL |        1 |      0 |  216352 |      1 |             |
| 27 | Rover1-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  208204 |      1 |             |
| 28 | TH-26-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3752 |      1 |             |
| 29 | Rover1-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |    3752 |      1 |             |
| 30 | House-Temp-Hist        |      1 | NULL | NULL |  NULL |        3 |      0 |    2503 |      0 |             |
| 33 | edm-temp               |      1 | NULL | NULL |  NULL |        1 |      0 |    3576 |      1 |             |
| 32 | edm-windchill          |      1 | NULL | NULL |  NULL |        1 |      0 |    3576 |      1 |             |
| 34 | edm-feelslike          |      1 | NULL | NULL |  NULL |        1 |      0 |    3560 |      1 |             |
| 35 | edm-pressure           |      1 | NULL | NULL |  NULL |        1 |      0 |    3452 |      1 |             |
| 36 | edm-rain-day           |      1 | NULL | NULL |  NULL |        1 |      0 |     424 |      1 |             |
| 37 | edm-rain-hr            |      1 | NULL | NULL |  NULL |        1 |      0 |    3452 |      1 |             |
| 38 | Rover2-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  162088 |      1 |             |
| 39 | TH-27-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    2984 |      1 |             |
| 40 | Rover2-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |  162084 |      1 |             |
| 41 | ct1-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      1 |             |
| 42 | ct2-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      1 |             |
| 43 | ct3-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      1 |             |
| 44 | ct4-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      6 |             |
| 45 | ct2-emontxv3-kwhd      |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      5 |             |
| 46 | ct2-emontxv3-histogram |      1 | NULL | NULL |  NULL |        3 |      0 |    NULL |      5 |             |
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+

 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hi Paul,

1 - whats the easiest way to flip between rfm12piphp and emonhub, can I not stop and start the other?

2 - how do I convert the Timestore feeds?

3 - change - done :)

Thanks

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

1) I don't think that you can 'flip', probably need to remove rfm12piphp & reboot

sudo service rfm12piphp stop
sudo rm /etc/init.d/rfm12piphp
sudo update-rc.d -f rfm12piphp remove

2) See the usefulscripts repo in emoncms git

 

Paul

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Thanks Paul,

I used this post: http://openenergymonitor.net/emon/node/10380

Converted the timestore to phpfina and the v8 sees the change successfully

v9 still doesn't show feeds, I guess I should reenable the emonhub and restart to see if anything changes.

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Ok restored emonhub and now I get:

2016-01-19 17:56:21,168 WARNING emonCMS couldn't send to server, HTTPError: 500 2016-01-19 17:56:21,171 WARNING emonCMS send failure: wanted 'ok' but got '' 2016-01-19 17:56:21,301 INFO emonCMS sending: http://localhost/emoncms_new/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1453251323.38,20,1925],[1453251323.65,10,726,616,220,3362],[1453251324.41,26,193,0,316,27],[1453251325.84,10,708,608,220,3362],[1453251352.33,20,1925],[1453251353.11,9,16,16,355,15,0,0],[1453251354.34,10,653,617,132,3362],[1453251355.05,28,195,0,373,27],[1453251360.27,10,560,614,34,3362],[1453251362.14,20,1925],[1453251364.88,9,16,16,357,15,0,0],[1453251366.03,10,561,610,33,3362],[1453251369.46,30,204,0,291,27],[1453251372.02,10,564,615,33,3362],[1453251372.3,20,1925],[1453251376.66,9,16,16,357,15,0,0],[1453251377.92,10,583,617,34,3362]]&sentat=1453251381 2016-01-19 17:56:22,154 WARNING emonCMS couldn't send to server, HTTPError: 500 2016-01-19 17:56:22,158 WARNING emonCMS send failure: wanted 'ok' but got '' 2016-01-19 17:56:22,262 INFO emonCMS sending: http://localhost/emoncms_new/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1453251323.38,20,1925],[1453251323.65,10,726,616,220,3362],[1453251324.41,26,193,0,316,27],[1453251325.84,10,708,608,220,3362],[1453251352.33,20,1925],[1453251353.11,9,16,16,355,15,0,0],[1453251354.34,10,653,617,132,3362],[1453251355.05,28,195,0,373,27],[1453251360.27,10,560,614,34,3362],[1453251362.14,20,1925],[1453251364.88,9,16,16,357,15,0,0],[1453251366.03,10,561,610,33,3362],[1453251369.46,30,204,0,291,27],[1453251372.02,10,564,615,33,3362],[1453251372.3,20,1925],[1453251376.66,9,16,16,357,15,0,0],[1453251377.92,10,583,617,34,3362]]&sentat=1453251382 2016-01-19 17:56:22,348 DEBUG 18 NEW FRAME : 1453251382.35 20 133 7 2016-01-19 17:56:22,351 DEBUG 18 Timestamp : 1453251382.35 2016-01-19 17:56:22,353 DEBUG 18 Node : 20 2016-01-19 17:56:22,355 DEBUG 18 Values : [1925] 2016-01-19 17:56:22,834 WARNING emonCMS couldn't send to server, HTTPError: 500 2016-01-19 17:56:22,838 WARNING emonCMS send failure: wanted 'ok' but got '' 2016-01-19 17:56:22,841 DEBUG 18 Append to 'emonCMS' buffer => time: 1453251382.35, data: [20, 1925], ref: 18 2016-01-19 17:56:22,945 INFO emonCMS sending: http://localhost/emoncms_new/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1453251323.38,20,1925],[1453251323.65,10,726,616,220,3362],[1453251324.41,26,193,0,316,27],[1453251325.84,10,708,608,220,3362],[1453251352.33,20,1925],[1453251353.11,9,16,16,355,15,0,0],[1453251354.34,10,653,617,132,3362],[1453251355.05,28,195,0,373,27],[1453251360.27,10,560,614,34,3362],[1453251362.14,20,1925],[1453251364.88,9,16,16,357,15,0,0],[1453251366.03,10,561,610,33,3362],[1453251369.46,30,204,0,291,27],[1453251372.02,10,564,615,33,3362],[1453251372.3,20,1925],[1453251376.66,9,16,16,357,15,0,0],[1453251377.92,10,583,617,34,3362],[1453251382.35,20,1925]]&sentat=1453251382 2016-01-19 17:56:23,444 WARNING emonCMS couldn't send to server, HTTPError: 500 2016-01-19 17:56:23,448 WARNING emonCMS send failure: wanted 'ok' but got ''
 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Is there anyway to step thru the database changes to ensure they have been completed or is the schema some wherre I can reference to ensure they match?

I know under administrator database check indicates everything is OK but for each node it indicates wait... and if I go to edit it indicates that feed## is missing or deleted?

Thanks

 

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Have you removed the rfm12piphp module?
If so, I would go back to the missing cmdline.txt and find out why it's missing.

Paul

pb66's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

The cmdline.txt will not need editing on a previously working system. The 2 reasons we tend no nedd to edit the cmdline.txt for are moving the file system or enabling serial port access to "install" a RFM2Pi/emonPi etc.

For what I read your filesystem was previouly moved and you are mot moving it againg and the RFM2Pi was previouly working fine so should not need "installing".

If cmdline.txt did not exist the system simply would not boot up as it wouldn't know where to find the filesystem. So the cmdline.txt is probally present and correct, just not where you are looking. You can confirm this by popping the SDcard in a cardreader to look at the file it if want to be sure.

There was some confusion with the /boot partition and /boot directory linking on that "separate" image so that may be another but seperate issue you need to investigate. But it shouldn't be related to this issue.

The root issue is the feed types

The emonhub install seems to have worked correctly, and it's the response from emoncms that is preventing the data from being accepted.

if you have reinstalled the raspberrypi module you need to remove that again, it will only cause problems. emonhub is working correctly. The difference being it is unable to post the data because emoncms is objecting. the raspberrypi module is not geared up to care and just keeps throwing data at emoncms whether it uses it or not. emonhub has informed you of the objection emoncms has raised and continues to buffer the data in hope you resolve the issue and it can resume posting.

 When emoncms is able to see and load hose feeds emonhub should be able to resume, that needs to be the focus. Are the feeds properly converted and recogised etc? I'm not familiar with the conversion so cannot offer any pointers there I'm afraid.

Paul 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

I have removed the rfm12piphp module and reinstated the emonhub.

feeds conversion to phpfina  - complete

mysql> select * from feeds;
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+
| id | name                   | userid | tag  | time | value | datatype | public | size    | engine | processList |
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+
|  1 | ct1-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3883744 |      5 |             |
|  2 | ct2-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3241104 |      5 |             |
|  3 | ct3-main               |      1 | NULL | NULL |  NULL |        1 |      0 | 3238912 |      5 |             |
|  4 | ct1-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  5 | ct1-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   32244 |      0 |             |
|  6 | ct2-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  7 | ct2-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   34958 |      0 |             |
|  8 | ct3-kwhd               |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
|  9 | ct3-histogram          |      1 | NULL | NULL |  NULL |        3 |      0 |   20988 |      0 |             |
| 11 | Power                  |      1 | NULL | NULL |  NULL |        1 |      0 | 3882976 |      5 |             |
| 12 | Power-kwhd             |      1 | NULL | NULL |  NULL |        2 |      0 |    2534 |      0 |             |
| 13 | Power-histogram        |      1 | NULL | NULL |  NULL |        3 |      0 |   35283 |      0 |             |
| 14 | monitor-temp           |      1 | NULL | NULL |  NULL |        1 |      0 |  307872 |      5 |             |
| 15 | monitor-temp-hist      |      1 | NULL | NULL |  NULL |        3 |      0 |    2841 |      0 |             |
| 16 | Power-average          |      1 | NULL | NULL |  NULL |        3 |      0 |    2698 |      0 |             |
| 19 | TH-30-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      5 |             |
| 18 | TH-29-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      5 |             |
| 20 | House-Temp             |      1 | NULL | NULL |  NULL |        1 |      0 |  216428 |      5 |             |
| 21 | Garage-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  216432 |      5 |             |
| 22 | Garage-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |  216424 |      5 |             |
| 23 | House-Humidity         |      1 | NULL | NULL |  NULL |        1 |      0 |  216420 |      5 |             |
| 24 | TH-28-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3896 |      5 |             |
| 25 | House-2F-Temp          |      1 | NULL | NULL |  NULL |        1 |      0 |  216356 |      5 |             |
| 26 | House-2F-Humidity      |      1 | NULL | NULL |  NULL |        1 |      0 |  216352 |      5 |             |
| 27 | Rover1-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  208204 |      5 |             |
| 28 | TH-26-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    3752 |      5 |             |
| 29 | Rover1-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |    3752 |      5 |             |
| 30 | House-Temp-Hist        |      1 | NULL | NULL |  NULL |        3 |      0 |    2503 |      0 |             |
| 33 | edm-temp               |      1 | NULL | NULL |  NULL |        1 |      0 |    3576 |      5 |             |
| 32 | edm-windchill          |      1 | NULL | NULL |  NULL |        1 |      0 |    3576 |      5 |             |
| 34 | edm-feelslike          |      1 | NULL | NULL |  NULL |        1 |      0 |    3560 |      5 |             |
| 35 | edm-pressure           |      1 | NULL | NULL |  NULL |        1 |      0 |    3452 |      5 |             |
| 36 | edm-rain-day           |      1 | NULL | NULL |  NULL |        1 |      0 |     424 |      5 |             |
| 37 | edm-rain-hr            |      1 | NULL | NULL |  NULL |        1 |      0 |    3452 |      5 |             |
| 38 | Rover2-Temp            |      1 | NULL | NULL |  NULL |        1 |      0 |  162088 |      5 |             |
| 39 | TH-27-Voltage          |      1 | NULL | NULL |  NULL |        1 |      0 |    2984 |      5 |             |
| 40 | Rover2-Humidity        |      1 | NULL | NULL |  NULL |        1 |      0 |  162084 |      5 |             |
| 41 | ct1-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      5 |             |
| 42 | ct2-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      5 |             |
| 43 | ct3-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      5 |             |
| 44 | ct4-emontxv3           |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      6 |             |
| 45 | ct2-emontxv3-kwhd      |      1 | NULL | NULL |  NULL |        1 |      0 |    NULL |      5 |             |
| 46 | ct2-emontxv3-histogram |      1 | NULL | NULL |  NULL |        3 |      0 |    NULL |      5 |             |
+----+------------------------+--------+------+------+-------+----------+--------+---------+--------+-------------+

 

emoncms v8 sees these changes correct

emoncmsv9 does not see the feeds?

The old DB didn't have all these tables feed_##, but not all of my feed ID were created into these separate tables if that is what suppose to happen during one of the database updates?

mysql> show tables;
+----------------------+
| Tables_in_emoncms    |
+----------------------+
| app_config           |
| applist              |
| converttotimestore   |
| dashboard            |
| device               |
| energy               |
| event                |
| event_settings       |
| feed_12              |
| feed_13              |
| feed_15              |
| feed_16              |
| feed_30              |
| feed_4               |
| feed_46              |
| feed_5               |
| feed_6               |
| feed_7               |
| feed_8               |
| feed_9               |
| feeds                |
| input                |
| mqtt                 |
| multigraph           |
| myelectric           |
| node                 |
| notify               |
| openbem              |
| openbem2             |
| openbem_projects_v3  |
| openbem_scenarios_v3 |
| packetgen            |
| raspberrypi          |
| rememberme           |
| report               |
| schedule             |
| users                |
+----------------------+
37 rows in set (0.01 sec)

Currently I'm getting the WARNING emonCMS couldn't send to server, HTTPError: 500

 

Thanks

 

pb66's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

I believe the "Feed_nn" entries are pre-existing "SQL" based data feeds which (here I'm scratching around in the foggy depths of my memory of stuff I didn't really feel the need to retain at the time) have been renamed with the "Feed_" prefix for v9. These will probably be your "Daily Totals" type feeds, do you have any "power to kWh/d" processes?

The newly created phpfina feeds will have should have been saved to your phpfina feed directory usually "/var/lib/phpfina" can you check they are there and the correct permissions/ownerships are set? Using 

ls -la /var/lib/phpfina

(​adjust if you have them (intentionally) located somewhere else.

Paul

EDIT - Actually the "feed_" feeds look like your histgrams too.

Do you have SQL(0), phpfina(5) and phpfiwa(6) data feed engines all enabled in settings.php?

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hi Paul,

the Feed_nn - weird, don't exist in the bckup of DB so it happenend during upgrade to v9 but none the less....

I do have daily totals and histograms :)

I have all the engines enalbed in settings.php and can choose them if I go add a new feed

//4 #### Engine settings
    $feed_settings = array(
        // Supported engines. Uncommented engines will not be available for user to create a new feed using it. Existing feeds with a hidden engine still work.
        // Place a ',' as the first character on all uncommented engines lines but first.
        // If using emoncms in low-write mode, ensure that PHPFIWA is disabled by removing the leading //, from the PHPFIWA entry
        'engines_hidden'=>array(
            //Engine::MYSQL         // 0  Mysql traditional
            //Engine::MYSQLMEMORY   // 8  Mysql with MEMORY tables on RAM. All data is lost on shutdown
            //,Engine::PHPTIMESERIES // 2
            //,Engine::PHPFINA      // 5
            //,Engine::PHPFIWA      // 6

 

Permissions look good:

 ls -lah /var/lib/phpfina/
total 280M
drwxr-xr-x  2 www-data root 4.0K Jan 19 15:31 .
drwxr-xr-x 50 root     root 4.0K Jan 18 13:05 ..
-rw-r--r--  1 www-data root  50M Jan 20 11:31 11.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:29 11.meta
-rw-r--r--  1 www-data root 4.2M Jan 20 11:31 14.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 14.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 18.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 18.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 19.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 19.meta
-rw-r--r--  1 www-data root  50M Jan 20 11:31 1.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:28 1.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:30 20.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 20.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:30 21.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 21.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:30 22.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 22.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:30 23.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 23.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:31 24.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 24.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:31 25.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 25.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:31 26.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 26.meta
-rw-r--r--  1 www-data root 4.1M Jan 20 11:30 27.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 27.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 28.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 28.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 29.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 29.meta
-rw-r--r--  1 www-data root  26M Jan 20 11:31 2.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:28 2.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 32.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 32.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 33.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 33.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 34.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 34.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 35.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 35.meta
-rw-r--r--  1 www-data root 2.9K Jan 20 11:30 36.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 36.meta
-rw-r--r--  1 www-data root  69K Jan 20 11:30 37.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 37.meta
-rw-r--r--  1 www-data root 4.0M Jan 20 11:30 38.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:30 38.meta
-rw-r--r--  1 www-data root  68K Jan 20 11:30 39.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:31 39.meta
-rw-r--r--  1 www-data root  26M Jan 20 11:31 3.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:29 3.meta
-rw-r--r--  1 www-data root 4.0M Jan 20 11:30 40.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:31 40.meta
-rw-r--r--  1 www-data root  23M Jan 20 11:31 41.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:31 41.meta
-rw-r--r--  1 www-data root  23M Jan 20 11:31 42.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:31 42.meta
-rw-r--r--  1 www-data root  23M Jan 20 11:31 43.dat
-rw-r--r--  1 www-data root   16 Jan 19 15:31 43.meta
-rw-r--r--  1 www-data root  21M Jan 20 11:31 45.dat
-rw-r--r--  1 www-data root   16 Dec 11  2014 45.meta
-rw-r--r--  1 www-data root    0 Jan 18 21:16 47.dat
-rw-r--r--  1 www-data root   16 Jan 18 21:16 47.meta

 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

does this help?

[Wed Jan 20 14:49:43 2016] [error] [client 192.168.1.78] PHP Fatal error:  Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms_new/Modules/feed/feed_model.php:71\nStack trace:\n#0 /var/www/emoncms_new/Modules/feed/feed_model.php(396): Feed->EngineClass('1')\n#1 /var/www/emoncms_new/Modules/feed/feed_model.php(279): Feed->get_timevalue('1')\n#2 /var/www/emoncms_new/Modules/feed/feed_model.php(253): Feed->redis_get_user_feeds(1)\n#3 /var/www/emoncms_new/Modules/feed/feed_controller.php(38): Feed->get_user_feeds('1')\n#4 /var/www/emoncms_new/core.php(64): feed_controller()\n#5 /var/www/emoncms_new/index.php(146): controller('feed')\n#6 {main}\n  thrown in /var/www/emoncms_new/Modules/feed/feed_model.php on line 71, referer: http://192.168.1.15/emoncms_new/feed/list
[Wed Jan 20 14:49:48 2016] [error] [client 192.168.1.78] PHP Fatal error:  Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms_new/Modules/feed/feed_model.php:71\nStack trace:\n#0 /var/www/emoncms_new/Modules/feed/feed_model.php(396): Feed->EngineClass('1')\n#1 /var/www/emoncms_new/Modules/feed/feed_model.php(279): Feed->get_timevalue('1')\n#2 /var/www/emoncms_new/Modules/feed/feed_model.php(253): Feed->redis_get_user_feeds(1)\n#3 /var/www/emoncms_new/Modules/feed/feed_controller.php(38): Feed->get_user_feeds('1')\n#4 /var/www/emoncms_new/core.php(64): feed_controller()\n#5 /var/www/emoncms_new/index.php(146): controller('feed')\n#6 {main}\n  thrown in /var/www/emoncms_new/Modules/feed/feed_model.php on line 71, referer: http://192.168.1.15/emoncms_new/feed/list

 

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hello,

It would seem if I go to create a new feed it seems to be successful however when I go back into the feed to which I created the new entry, it indicates that the feed doesn't exist or was deleted.

Thoughts?

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

After trying to create the feed with no success, going and doing:

redis-cli "flushall" 

all the feeds appeared and emonhub no longer complains

Cheers!

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Hello,

How would I go about getting this old SD image to upgrade to the latest kernel?

apt-get update, apt-get upgrade work successfully however uname shows the newer kernel never loads?

Is there a setting somewhere that has locked the kernel that I can undo to get up to date?

Thanks

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Redis-cli flushall...which was one of the instructions included in the link in my first post above!

Paul

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

really.....  thats what you have to contribute.  following the instructions in your first post didn't correct this

Paul Reed's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

No, because despite you knowing that you can't run emonhub & rfm12piphp on the same system, you continued to do so, despite pb66 and myself repeatedly advising you not to. This would have the effect of caching the errors, which continued to be written, until Redis was flushed.

Regardless, I'm pleased that you are back working eventually, and your question about updating the Kernel would be better placed in the raspberry Pi forum than here.
Now knowing that you also want to update the OS Kernel, the most logical and easiest route would have been;

  • Convert your 8.5 feeds to current standard
  • Create backups of your data
  • Format your drive
  • Install Jessie & emoncms
  • Import your backed up data

Paul

pb66's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

You should read "New emoncms raspberrypi harddrive image" as I believe you have the issues caused by the boot partition not being correctly mounted, this is evident by the missing cmdline.txt and that it reports it's running a very old kernel despite attempted upgrades.

I would be tempted to backup your converted data and start again with a fresh Jessie install, although that image (when fixed) remains one of my favorites it is a bit dated now and had some flaws. the way the hdd image is put together now is more robust in that it uses the hdd's "uuid" in the boot routine rather than just booting to "/dev/sda2" the latter method is a bit hit or miss if you leave another similar usb device connected and reboot the pi you have a 50/50 chance of a successful boot as the the 2 devices will be given the addresses "sda" and "sdb" the first to respond being "sda". The "boot" partition is not duplicated either which reduces the chance of confusion there too.

Even with a straight forward SDcard only setup, many discussions on the subject recommend a fresh "Jessie" install over a dist-upgrade so now would probably be the best time to make a clean break of it.

Paul

cbwonderboy's picture

Re: Upgrade 8.3.6 on Wheezy from Emoncms SD image to v9.3

Thank you Paul and Paul.

Makes sense, I hate loosing data and your likely correct in that me flipping between rfm12piphp and emonhub did add to the issue.

I will take a look at your suggestions for upgrading, I thought I only had to upgrade the boot on the SD card since everything else seems to have been upgraded but I will review your links and post any questions on teh pi forum.

Thanks again, appreciate the assistance.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.