@glyn - now on the new thread ...

I'm now going to try a migration - assuming that other with more experience will be checking functionality. So, here's what happens:

1 - Start with Jun15 image, added backup, then manually ran emoncms-export.sh (did not spend time getting backup to work in the GUI)

2 - On a new SD, run the 16feb16sd image. Check that input is working, etc.

3 - Restore from backup in #1

4 - Confirm that inputs look ok :-)

5 - Confirm that feeds look ok ... er ... :-(

The Feeds page says "loading" and sticks there.

Try to create a feed from the input - message "ERROR: Feed could not be created, undefined"

Try to read a feed using the API "http://192.168.16.31/emoncms/feed/list.json" - and get a PHP error:

Fatal error: Uncaught exception 'Exception' with message 'ABORTED: Engine id '1' is not supported.' in /var/www/emoncms/Modules/feed/feed_model.php:71 
Stack trace: 
#0 /var/www/emoncms/Modules/feed/feed_model.php(396): Feed->EngineClass('1') 
#1 /var/www/emoncms/Modules/feed/feed_model.php(279): Feed->get_timevalue('10') 
#2 /var/www/emoncms/Modules/feed/feed_model.php(253): Feed->redis_get_user_feeds(1) 
#3 /var/www/emoncms/Modules/feed/feed_controller.php(38): Feed->get_user_feeds('1') 
#4 /var/www/emoncms/core.php(64): feed_controller() 
#5 /var/www/emoncms/index.php(146): controller('feed') 
#6 {main} thrown in /var/www/emoncms/Modules/feed/feed_model.php on line 71

... tried a reboot, same situation.

Also looked at logs in "Admin->Logger" and saw:

2016-02-17 11:58:42.903|WARN|phpmqtt_input.php|Subscribing to: emon/# 
2016-02-17 11:58:42.904|WARN|phpmqtt_input.php|Not connected, retrying connection 
2016-02-17 11:58:42.905|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0 
2016-02-17 11:58:42.906|WARN|phpmqtt_input.php|Subscribed to topic: emon/# 
2016-02-17 11:59:18.463|ERROR|feed_model.php|EngineClass() Engine id '1' is not supported. 
2016-02-17 11:59:22.067|ERROR|feed_model.php|EngineClass() Engine id '1' is not supported.

I will try to look at this - but any pointers would be useful. 

@glyn - again on the new thread

Now trying "factory reset"

So, from message above, did a factory reset, which should put everything back in place. So tests ...

1 - run factory reset under a root shell - ok

2 - login - password now the "factory" one :-)

3 - Check inputs - working :-)

4 - Go to feeds page - all seems ok :-)

5 - Back to inputs, hit the spanner to create a feed 

6 - Hit "add" and ... :-(

When I hit "Add", I get, in a dialogue box:

ERROR: Feed could not be created, undefined

What to do now ?

@glyn

So, the image that I downloaded this morning emonSD-16feb16 ... has. er, redis 2.2.5-1 ...

Installed the new redis per the instructions from the alpha thread, and ended up in the same place - the feeds page says "loading ..." and the API is complaining about the wrong engine ID ...

EDIT: just realised that the apt cache will show the apt installed version number - installing the redis manually will not overwrite this versioning.

So, I'm stuck!

Yes, that's right. We're not running the version obtained via the packages. To check the version we can look at the header file before compiling 

cat ~/phpredis/php_redis.h | grep VERSION

I'm not sure if the engine ID is another separate issue of if it was a side effect of the Redis issue....

i too am getting this issue. If i restart apache2 then i can quickly add a feed/input...but it then dies soon after.

@glyn

Yes - when I rebuilt, I overwrote what was on the image with 2.2.7 ...

I guess we need a way of tracking versions for non-apt installed stuff?

The max backup file that should be able to be uploaded the the web interface should be 200Mb as set in /etc/php5/apache2/php.ini. I will investigate why errors are being reported when uploading smaller files and there is sufficient space in the ~/data partition. 

@jason I don't there there is a way to track version for non apt sutff. The only way I know is the check the version number before compiling and installing. See my post above, 

@gregl please could you provide more info regarding the issue you are experiancing 

Glyn - Concerning Backup / Import - for me it filled because of the uploaded tar.gz file in the upload directory (33 MB total) and the uncompressed files in the import directory (~99 MB total).  With both of those directories equals 132 MB and that filled the /home/pi/data area.

EDIT: added attachment - Listing of /data directory

I've taken a look at what happens for the "Engine ID" issue. Looking through feed_model.php and adding a pile of debug, I can see how the logic is supposed to work. Clearly there is a feed with an unknown "engine" type.

I looked at what is happening with redis - and using the MONITOR command, I noticed that something is creating 5 nodes:

... "EXISTS" "user:feeds:1"

... "SADD" "user:feeds:1" "1"

... "HMSET" "feed:1" "id" "1" "userid" "1" "name" "Node emonpi" "datatype" "0" "tag" "" "public" "0" "size" "" "engine" "1" "processList" ""

... "SADD" "user:feeds:1" "2"

... "HMSET" "feed:2" "id" "2" "userid" "1" "name" "Node emonpi" "datatype" "0" "tag" "" "public" "0" "size" "" "engine" "1" "processList" ""

... "SADD" "user:feeds:1" "3"

... "HMSET" "feed:3" "id" "3" "userid" "1" "name" "Node emonpi" "datatype" "0" "tag" "" "public" "0" "size" "" "engine" "1" "processList" ""

... "SADD" "user:feeds:1" "4"

... "HMSET" "feed:4" "id" "4" "userid" "1" "name" "Node emonpi" "datatype" "0" "tag" "" "public" "0" "size" "" "engine" "1" "processList" ""

... "SADD" "user:feeds:1" "5"

... "HMSET" "feed:5" "id" "5" "userid" "1" "name" "Node emonpi" "datatype" "0" "tag" "" "public" "0" "size" "" "engine" "1" "processList" ""

Looking at where this comes from, it seems to be from feed_controller.php:44 - and the 'engine' comes from "get('engine')" call ... 

I will continue tracing this through...

I spent a bit of time with this - from what I can see, by adding debug into feed_model.php and using redis-cli to monitor what is going on, there seems to be a problem right from feed_controller.php:44 where the result of "get('engine')" is passed into create() in feed_model.php.

I now need to trace where the "get('engine')" goes to ...

My guess is that there is something wrong with the intial set up - so I will re-image the SD card and try again.

@jon if you have flashed the image onto a card larger than 4gb you should expand the data partition to fill the remaining space on the card. There are many ways to do this. My preference would be insert that card into your computer and use gparted on Linux to expand the partition. I'm sure there are equivalent window tools and it's probably possible to do direct on the P

@jason thanks for debug output I will investigate

@sheppy Sorry for slow reply, I have been away. Thanks for the backup file. I'm not sure why it's failing to upload via php. The settings set in php.ini should allow up to 200Mb. However for large imports it's probably best do to manual import. I have added note to backup modules instructions:

https://github.com/emoncms/backup/

You will need to expand the data partition. Default emonPi image has a RW ~/data partition of 150Mb, size of uncompressed backup must be less. If using an SD card > 4GB (default emonPi is 8GB) the data partition can be expanded to fill the rest of the SD card 4GB+ using gparted tool or equivalent.

I have been having regular wifi dropouts on my emonPi for ages now and did not realise that there had been several major updates since I received the device, after backing it on Kickstarter. I have asked about this particular problem on the forum without much success, but did not have a lot of time to look through the forum properly. Only recently did I discover that a major update had been released last June which may have solved my problem.

After checking over the last few days, I see that there have recently been major update efforts, including moving to Minibian and then Raspbian Jessie Light (after the PiZero came out), as well as loads of functionality improvements.

I have spent as much time as I have been able to in catching up and have stumbled upon encouraging information and also a few problems during the process.

I discovered by looking through the filesystem, that my device has been running "emonpi-28may2015", which could have been the cause of my problems. I therefore decided that it would probably be better for me to have a go with the latest beta (rather than installing the June update) as I really wanted to have a look at all the new features I'd missed out on.

The major problem I have come upon has been with the Backup module:

The first issue I had was, although I followed the instructions in the Readme.md at https://github.com/emoncms/backup and rebooted, I could not find the Backup page in emonCMS, as the web interface was evidently a (pre-release) old-style one (which I intuited from the screenshot on GitHub). Thankfully, I had the idea of manually entering the URL: http://emonpi/emoncms/*backup* and so was then able to export my data, saving the resulting ".tar.gz" file to my PC.

(This problem should be documented, as, like myself, many people may have many months worth of useful data collecting on their emonPi's, which they will not want to sacrifice in order to upgrade.)

The second problem I have run into is that I have not been able to import this data into my new emonPi installation. This is the biggie...

(At this stage, I need to tell you how I have gone about the upgrade process and let you draw your own conclusions about my methodology.)

In order to prevent any major downtime; rather than simply backup my data, wipe my emonPi µSD card and DD the new OS image, I did this:

1. After trying to flash the new image on various 4GB cards and failing due to lack of available space (N.B. The card I got with my emonPi was ~3.75GB), I eventually selected a 16GB µSD card and successfully DD'd the image to it. I then expanded it's partitions using Gparted.

2. I inserted this card into a spare Raspberry Pi 2, booted and then browsed the new web interface from my PC.

3. I navigated to the *backup* page, selected the file "emoncms-backup-2016-02-**.tar.gz" from my home folder and clicked "Import Backup".

4. It failed!

   The result was a simple webpage displaying the message " There was an error uploading the file, only .tar.gz are allowed. "

   I tried all of this several times and in several different ways, but was unable to get the new system to accept the backup file at all.
   
   I then gave up on trying to minimise downtime of my emonPi, taking it apart and replacing the SD card with the new one, rebooting and failing again.

5. Not giving up, I installed *MC* on the emonPi, trawled the filesystem, then checked and noted the ownership and permissions of all of the relevant data files and folders.

6. Next, I extracted the files and folders from the backup archive on my desktop, scp'd them to the Pi, deleted the originals from the Pi and copied the new ones into place:
 

    Files:                                               User:           Group:
    /home/pi/data/emoncms.conf        pi                www-data
    /home/pi/data/emoncms.sql          pi                pi
    /home/pi/data/emonhub.conf         pi               www-data

    Folders:                                          User:           Group:
    /home/pi/data/phpfina/                  www-data    root
    (contained files)                            www.data    www-data
    /home/pi/data/phptimeseries/       www.data    root
    (contained files)                            N/A               N/A

7. It failed again!
   I have tried to fiddle with the permissions and ownership of the backed-up and restored data, restarted services, clicked on various buttons, updated emonCMS etc. etc. etc...
   
   Nothing so far has worked.
   What am I doing wrong here?

Please help me out,

Rich.

@ritch 

Sorry to hear you having trouble updating. See my comments in line below:

The major problem I have come upon has been with the Backup module:

The first issue I had was, although I followed the instructions in the Readme.md athttps://github.com/emoncms/backup and rebooted, I could not find the Backup page in emonCMS, as the web interface was evidently a (pre-release) old-style one (which I intuited from the screenshot on GitHub). Thankfully, I had the idea of manually entering the URL: http://emonpi/emoncms/*backup* and so was then able to export my data, saving the resulting ".tar.gz" file to my PC.

(This problem should be documented, as, like myself, many people may have many months worth of useful data collecting on their emonPi's, which they will not want to sacrifice in order to upgrade.)

GLYN: thanks, I've added note the the backup module 

The second problem I have run into is that I have not been able to import this data into my new emonPi installation. This is the biggie...

(At this stage, I need to tell you how I have gone about the upgrade process and let you draw your own conclusions about my methodology.)

In order to prevent any major downtime; rather than simply backup my data, wipe my emonPi µSD card and DD the new OS image, I did this:

1. After trying to flash the new image on various 4GB cards and failing due to lack of available space (N.B. The card I got with my emonPi was ~3.75GB), I eventually selected a 16GB µSD card and successfully DD'd the image to it. I then expanded it's partitions using Gparted.

GLYN: Great, good work. 

2. I inserted this card into a spare Raspberry Pi 2, booted and then browsed the new web interface from my PC.

3. I navigated to the *backup* page, selected the file "emoncms-backup-2016-02-**.tar.gz" from my home folder and clicked "Import Backup".

4. It failed!

   The result was a simple webpage displaying the message " There was an error uploading the file, only .tar.gz are allowed. "

   I tried all of this several times and in several different ways, but was unable to get the new system to accept the backup file at all.
   
   I then gave up on trying to minimise downtime of my emonPi, taking it apart and replacing the SD card with the new one, rebooting and failing again.

GLYN: if your backup file to be restored is large, whch id porobably is if you have been logging since May15 then I recommend you follow the manual import steps as detailed in the backup module readme. There is still an issue with uploading large files via PHP web interface. 

5. Not giving up, I installed *MC* on the emonPi, trawled the filesystem, then checked and noted the ownership and permissions of all of the relevant data files and folders.

6. Next, I extracted the files and folders from the backup archive on my desktop, scp'd them to the Pi, deleted the originals from the Pi and copied the new ones into place:
 

    Files:                                               User:           Group:
    /home/pi/data/emoncms.conf        pi                www-data
    /home/pi/data/emoncms.sql          pi                pi
    /home/pi/data/emonhub.conf         pi               www-data

    Folders:                                          User:           Group:
    /home/pi/data/phpfina/                  www-data    root
    (contained files)                            www.data    www-data
    /home/pi/data/phptimeseries/       www.data    root
    (contained files)                            N/A               N/A

7. It failed again!
   I have tried to fiddle with the permissions and ownership of the backed-up and restored data, restarted services, clicked on various buttons, updated emonCMS etc. etc. etc...
   
GLYN: You need to restore the Emoncms mysql database as well as restoring the files. I recommend running the emoncms-import.sh script  manually (see link above). I would recommend starting with a fresh image using the latest RC image posted this morning: http://openenergymonitor.org/emon/node/12212

Thanks for your help testing, I have just posted what I hope is a Release Candidate image emonSD-20Feb16. 

http://openenergymonitor.org/emon/node/12212

Please post issues for this image on the new RC thread.