Archived Forum

Great Schism, Happy to help out, I agree it would be best to upgrade using the debian package once its ready rather spend time doing this twice.

Would be great to get redis in there as default, It looks like php5-redis is maintained by the same poeple as the pecl redis package, there are quite a few different redis clients around for php the c based one being a fair bit faster. Is it possible though to add pecl based installation as its needed for serial dio package?

Whats the difference between the debian versioned or direct? apart from location of install?

(re: redis etc) - these things are separate to the debian packaging of emoncms itself- the way we install emoncms should be transparent to redis and vice-versa.  i.e. we could potentially package up the serial PHP code too, but it would be separate and you could either choose PECL or apt to set it up.

I'm having a little trouble just now getting the SMTP configuration added to the emoncms debian install process. Once that works, I'll update my apt-repo and make a post with an initial install guide. If you can correct it, that will help me understand the interaction of the different systems.

There's no code difference between the two versions, it's just that in the debian-packaged version I've put together, the directory structure isn't the same. I have debian/ www/ config/ etc. and only  www/ is copied out to be served by apache (while config/ contains things like the sites-available definition for serving emoncms, potentially db configuration, and so on).

When I merged in v8 to my working branch, all the files I'd moved updated without conflicts, but the new files (node module, for example) appeared under ./Modules/node rather than ./www/Modules/node/

It's not something that would take more than a few minutes to resolve, but just something to be aware of.

Rather than getting people to clone a git repo into /var/www, I suggest that the install steps for non-apt based emoncms would be something like "clone the repository. Copy the www/ directory to /usr/share/emoncms/, copy the config/emoncms file to /etc/apache2/sites-available/" and so on. Nesting the git repositories as we currently do is not great because some git commands on emoncms/ can mangle the module repos in subdirectories, so that's a straight benefit regardless of the debian packaging approach.

If the data-providers (be it PHP, Python or whatnot) are separated from the 'main' CMS, wouldn't it be possible to create a virtual package that provides the functionality?

So, if you don't need the pecl DIO module (because you use the python script for example) EmonCMS would depend on the virual redis package, which in turn provides one of the various packages (be it the C or the pecl version). If however you use the PHP script, you would get a hard dependency on the pecl verion of redis, and a blocking on the C version (with all kind of hints how to solve that if the C version is already installed).

It would be great if it where possible to make the EmonCMS package modular in that way.

Keep in mind, my Debian knowledge is rusty, as I'm more of a FreeBSD and Gentoo guy. I'll pop in a blank SD card in a spare pi to try out the packages sounds great!

I'm not really familiar with the engines within emoncms or how they integrate. Abstractly, the ideal would be if there was just one default that is always supported (mysql?) and additional engines that could be added or not as the admin pleases.

If this was the case, at least from a debian packaging point of view, we could make emoncms depend on a virtual package called 'emoncms-engine' and then package up things like 'emoncms-engine-redis' which are marked as providing "emoncms-engine". That way you could have as many or as few engines as you liked (and the respective debian package installations would set those engines up, freeing this worry from emoncms itself).

It would be cool just to get it to install the way it is however :P

emoncms v8 upgrade procedure (verified against Jan 2014 HDD image)

I've based this on the readme on github, but adapted things for an installation using debian-packaged modules which I've forked - see my own github for the source.

1. Backup

Back up your existing installation. I recommend approach 2.

Relocate your current emoncms filesystem to prevent any possible confusion with the debian installation:

sudo mv /var/www/emoncms /var/www/old-emoncms

2. Pre-requisites

The emoncms debian package ensures all dependencies (that are listed here) are installed automatically. 

However, it can't do anything about PECL. I've reproduced the configuration steps from the readme here for completeness:

Install pecl dependencies (serial, redis and swift mailer)

sudo pear channel-discover pear.swiftmailer.org
sudo pecl install channel://pecl.php.net/dio-0.0.6 redis swift/swift

Add pecl modules to php5 config

sudo sh -c 'echo "extension=dio.so" > /etc/php5/apache2/conf.d/20-dio.ini'
sudo sh -c 'echo "extension=dio.so" > /etc/php5/cli/conf.d/20-dio.ini'
sudo sh -c 'echo "extension=redis.so" > /etc/php5/apache2/conf.d/20-redis.ini'
sudo sh -c 'echo "extension=redis.so" > /etc/php5/cli/conf.d/20-redis.ini'

3. Install emoncms

Add a line to your /etc/apt/sources.list so that the Pi can access the new Openenergymonitor packages:

sudo echo "deb http://emon-repo.s3.amazonaws.com wheezy unstable" >> /etc/apt/sources.list

Now update apt:

sudo apt-get update

You should be able to see the core emoncms, raspberrypi and event modules coming in (these are the only components of emoncms which are currently supported using this install method). 

Install:

sudo apt-get install emoncms

The installer should pop up a bunch of questions about your emoncms DB settings, as well as SMTP options. You must supply the first four pieces of information: DB username and password, DB host, and database name (which will be created if it doesn't already exist).

Consult your existing settings.php file for a reminder - it's at /var/www/old-emoncms/settings.php

The debian installer will create a valid settings.php file at the end of the installation, and put it in the right place. If you don't get asked these questions, or if you need to change one of the answers, run this command:

sudo dpkg-reconfigure emoncms --force

This is the preferred way of updating the settings.php file because when the time comes to update to emoncms v9 you just need to confirm the answers again - even if the format of the v9 settings.php file is completely different. On the other hand, if you edit settings.php by hand, when v9 comes along your original answers to the questions would be used, rather than anything you edited in by hand.

4. Apache configuration

We need to tell apache to switch on emoncms:

sudo a2ensite emoncms

sudo /etc/init.d/apache2 reload

Now you should be able to load emoncms in your browser at http://your-pi-address/emoncms and see it working.

5. Install raspberrypi module

To add the raspberrypi module, run

sudo apt-get install emoncms-module-rfm12pi

sudo service rfm12piphp restart log

(I recommend running with logging until you're sure the system is stable - especially on an HDD!)

6.Install event module

Even simpler:

sudo apt-get install emoncms-module-event

Emoncms v8 (debian packaging) work items

Here's a list of things which should be addressed before we pull the "debianized" changes into master:

1. PECL prerequisites
2. Support all existing modules
3. Agree on a git policy
4. Upgrade plan for non-debian or shared hosting users
5. "help!" guide for people who git pull the debian source tree into a legacy installation
6. ???

In order:

1. PECL prerequisites

It would be great if we could eliminate the manual config above (redis, swiftmailer and serial).

Based on a quick google, we have libphp-swiftmailer right now, php5-redis (looks to be the same as the PECL module, but not available in wheezy) and no sign of dio at all. Could/should we debian package it (and host the php5-redis package on our own apt repo for the time being?

This is definitely a 'nice to have' rather than a prerequisite.

2. Support all existing modules

You can add one of the non-debian modules to the debian-installed version of emoncms just by checking out its git repo as usual (but into /usr/share/emoncms/www/Modules/ instead of /var/www/...) however, it would be better just to add debian support to these.

At least one (packetgen) depends on some weirdness that looks like it needs a refactor before it could be packaged (has to modify raspberrypi module file). Others unclear.

3. Agree on a git / debian policy

Not essential as committing things to master on git won't do anything to the apt repository, so in that sense we will already gain (extra) control over releases. However, we should agree on a tagging system to be used to mark where different versions in apt were taken from, and also create some more controlled git branches for contributors (rather than just having one 'rework' branch).

If we don't do this it will be hard to know what's in apt at any given time, since the git metadata is not part of debian packaging.

We should also start thinking about how we'll offer incremental upgrades. Just for an example: if something doesn't require any installation steps, could it be released as v8.1 next week (with a major version increment reserved for releasing something that needs manual intervention)?

4. Upgrade plan for non-debian or shared hosting users

Needless to say, if you aren't a debian admin then this packaging step isn't very exciting - in fact it makes installation of emoncms a little bit more challenging.

Instead of just cloning emoncms into /var/www, you'll now need to clone it into /usr/share/emoncms/ and symlink it into place.

It's not rocket science, but a solid guide to non-debian installation of the debian version is really a prerequisite to updating master.

5. "Help!" guide for legacy installations which are updated

If someone pulls the debianized master of emoncms into a /var/www/emoncms/ git repo then their installation will stop working. This will be an ongoing problem until everyone with an old install who is prone to upgrading it out of the blue has done so - a clear guide to give them would be preferable...

Anything else?

I'll have a look at dh-make-php, to see if that can help creating a .deb for the dio module.

This .deb of DIO seems to work for me: (dropbox link) https://db.tt/3lG3u811 (note: for Debian on Arm -> Raspberry Pi)

Created it with the dh-make-php, 'twas real easy too :)

I've pushed a new version to the apt repo: version 8.0-1  (that version number means: v8.0 of emoncms, debian packaging increment 1)

This is intended to help out with the can't create feed problem by making sure the feed directory structure/permissions are OK as part of the debian install.

Probably worth pointing out that I've added a news file as well as a debian changelog to emoncms as part of this effort, and direct your attention to apt-listchanges.

sudo apt-get install apt-listchanges

sudo dpkg-reconfigure apt-listchanges

I choose the following: 'text', 'yes' (remember seen changes), no email notification (empty), news, 'yes' (confirmation)

Now (assuming you're running yesterday's emoncms version 8.0):

sudo apt-get update

sudo apt-get install emoncms

You'll see this output:

Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be upgraded:
  emoncms
1 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Need to get 759 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  emoncms
Install these packages without verification [y/N]? y
Get:1 http://emon-repo.s3.amazonaws.com/ wheezy/unstable emoncms armhf 8.0-1 [759 kB]
Fetched 759 kB in 1s (745 kB/s) 
Reading changelogs...
emoncms (8.0-1) unstable; urgency=low

  This version contains incremental updates to the installation procedure:
  in this case, ensuring the filesystem for the various types of feed exists
  and has the correct ownership.

-- Dave McCraw <dave@mccraw.co.uk>  Sun, 16 Mar 2014 07:09:08 +0000

apt-listchanges: Do you want to continue? [Y/n]

I think this should be pretty useful when it comes to helping people upgrade without breaking their installations / making clear the scope of each change. It just requires whoever updates the apt repository to run dch --news and give a new entry for each version :)

Dave: Looks great!

Thinking about your point 3: Agree on a git / debian policy

Lets say there are 3 main types of update: Bug fixes, Refactoring, New features.

If in the future most people are testing via the debian package then we need to get bug fixes from git into the debian package quickly, refactoring and new features don't need such a high priority.

Am I right that every change to the debian package needs to be a sub version increment? or that it would be good for it to be?

I guess it makes sense if the git master branch is a mirror of the debian package state as its probably easier to check the git commit log to see what the exact code changes are than open the deb package. We could then potentially have some kind of automatic debianization from the master branch, a script checking if the master branch has changed once a minute and rebuilding on change.. would this work?

The current emoncms git workflow is similar to this: https://www.atlassian.com/git/workflows#!workflow-gitflow although not always strictly adhered too. What do you think of if we where to have a continuous develop branch with feature branches and every merge into the master branch could be tagged with a version number with if possible the master automatically linked to the apt package?

I've taken a look at that atlassian workflow and it looks good to me. But note that develop hangs around across multiple releases - you should try never to delete branches (not everyone can handle a complex rebase, especially after a long time has passed).

I wouldn't try to overthink the debian packaging. You wouldn't normally look at the deb for anything (always referring to git), and anyone can build and install their own .deb by checking out a fix branch (or the develop branch) from git and running a simple command.

When the master branch advances, incrementing the debian changelog and news files allows us to build a new .deb and get it onto the apt repo, where everyone else can upgrade to it.

I suggest that we simply move up 0.1 of a version every time master advances with new features, and 0.0.1 for bug fixes. This means that addressing the issues with visualisations last night puts us on emoncms 8.0.1 - in this way it's really obvious what people are running when we ask them, and they will have an expectation of what they're upgrading to based on consistent versioning?

You can number the develop branch differently: ideally it would keep up with master and you'd number it something like 8.0.1-develop so that it's obvious when someone is running unreleased code too.

We could have some script that monitored git and pushed new stuff to apt, but I'd be worried it might go off-piste for some reason. I'd think we probably just want to manually build and push .debs for new master increments for now, since the super keen can build themselves from source.

A little bit of progress here, although I had a titanic distraction due to feeds not updating simply because I hadn't restarted rfm12bpiphp...

As part of troubleshooting I've broken out the 'myelectric' module into its own debian package - emoncms-module-myelectric (it looked like it might have killed apache, I'm not sure, but figured it wouldn't hurt if it was installable separately?)

The notify module seems to be a subset of event in terms of functionality, so I'm thinking that we could take this opportunity to consolidate by only migrating event to a debian packaged version?

I've migrated openbem (emoncms-module-openbem) and sync (emoncms-module-sync , suffers from this issue) and pushed them up to the apt repo.

packetgen is the obvious outstanding one - I'm not sure what the difference is between the rfm12pi version of the file and the one contained in packetgen though, so I'm not sure I want to proceed (I don't think "copyto_raspberrpi_module" is a strategy that's going to work in a debian-based install).

I gave a little bit of thought to the two documentation scenarios described above too.

1. People pull the latest code from git without realising that it's now the source for a packaged application, not something for cloning into /var/www:

- we provide a tag on master called something like "last pre-debian version" and advise people who do this that they either need to switch to a debian-based install or run git reset --hard onto that tag.

2. People want to carry on upgrading their install but can't use (or prefer not to use) the debian packages.

The simple solution to this is to clone the repositories outside of /var/www and then symlink them in.

That is, you could check out emoncms (debian version) into /usr/share/emoncms and then replace the old /var/www/emoncms directory with a symlink to /usr/share/emoncms/www/ - modules would be much the same (you just symlink www/ to /var/www/emoncms/Modules/foomodule).

In theory I don't see any major obstacles to pulling these debian changes into master, so long as we're happy to potentially have a few support topics where people are like "wtf... I git pulled and it's not doing what I expected".

Worth the price for a nice consistant install IMO (but then I would say that!)

Bold suggestion: should we make the debian-packaged version of emoncms / modules available as completely new git repositories? We could do this by forking them internally within the emoncms github account and renaming, I think.

Pros:

• people who update by pulling the git repo into /var/www/ won't have their installs broken or need to take any special actions
• we can formalize the repo names to match the package names (i.e. you'll be able to tell by the repo name that something is an emoncms module, a gateway, or whatever)
• we can run side-by-side for a while and tease out any issues while maintaining the existing v8.0 repo for people who don't want to be guinea pigs
• If we don't support a particular module yet, it doesn't matter - people can stick to the old non-debianized version or do a manual install until someone submits a pull request that fixes it.
• In the worst case scenario, suppose we decide we hate the debian versioned procedure, we could just make sure all changes are in the original repos and then drop support.
• On the other hand, when we decide to stop officially supporting the 'clone into /var/www/' method, there won't be anything stopping other people from maintaining it themselves, if they like. They'll easily be able to track upstream changes from the new repos if they reconfigure git.

Cons:

• Even though github should support this as a fork and rename, because the file structure of the two repos would be different, pull requests might need a bit of manual merging to bring them across from one to the other.
• It might discourage contributions by confusing people as to WTF is going on?

Anything that simplifies the install process is to be encouraged, as it's clear from the forum posts that it is something that trips up quite a few people. However the people that get tripped up tend to be the less technical minded, these aren't necessarily the ones using Raspberry Pi's.

My concerns are:

• Currently Emocnms works on Operating systems that aren't going to support debian package, so changing underlying modules isn't a trivial task.
   
• Need to check that all supported OS's have some form of easy symlinking if the underlying directory structure is going to change.  Should we be making the install on non debian more complicated to make debian easier?   
   
• Need way of determining how you installed Emoncms ( the average person isn't going to remember) so need numbering system displayed in Emoncms that incorporates install method (e.g. v8.0.2DP) 
   
• Need version control parity between installation methods.  Currently we have V8.0.1 on debian package and v8.0 pre release on github master, (with Github emoncms releases showing latest release to be v7.0)

• Need clarity on what is the master.  Trystan's comment earlier in the thread "git master branch is a mirror of the debian package" scares me.  Surely a method of install for specific OS should be a mirror of a non-OS specific master?

Hi

I have been following this by installing on a spare Pi. As far as I can tell every thing has worked each time I have run a new update. I can log in without trouble. ( I have no inputs as yet so I can go no further at this stage).

I did a  rfm12piphp restart as suggested and got the following message. May not be significant to you but I thought I would let you know.

PHP Notice:  Undefined variable: timestore_adminkey in /usr/share/emoncms/www/Modules/raspberrypi/raspberrypi_run.php on line 36

Is this because I have to locate and enter the adminkey in raspberrypi_run.php?

Regards 

Ian

I saw that, Ian, and I'm not sure why it wouldn't use the timestore key from settings.php since it already shares some of the PHP classes. Something to look into.

I could potentially add a separate config file for the rfm12piphp module and get the debian installer to configure the key value as it does for main emoncms.

Thanks for trying this :)

Currently Emocnms works on Operating systems that aren't going to support debian package, so changing underlying modules isn't a trivial task.

Just to be clear, it's not so much that the modules have been made debian-specific, as I've restructured it to a more "normal" repo layout / install path *at the same time* as implementing debian packaging.

This made packaging much easier and more robust, but we could do it separately to Debian packaging if we liked (and if we want to add support for packaging in redhat, for instance, there would be no need to have another repo later).

Need to check that all supported OS's have some form of easy symlinking if the underlying directory structure is going to change.  Should we be making the install on non debian more complicated to make debian easier?  

The existing install is no good IMO because it nests git repositories on the filesystem. Try running git reset --hard in /var/www/emoncms and all modules will be deleted... (don't actually do this ;-)

Maybe what we need is to have a non-debian install script bundled into each repo to take care of creating symlinks. That way there are two install paths: clone the repo and run the install script (generic linux) or use apt.

Need way of determining how you installed Emoncms ( the average person isn't going to remember) so need numbering system displayed in Emoncms that incorporates install method (e.g. v8.0.2DP)

Of course, this is a great argument against running two git repos (one for debian and one for normal).

If we avoid that, we don't need to differentiate between debian and non-debian because we'll know what version 8.2.1 corresponds to in git (a tag "debian/8.2.1" for instance).

Need version control parity between installation methods.  Currently we have V8.0.1 on debian package and v8.0 pre release on github master, (with Github emoncms releases showing latest release to be v7.0)
Need clarity on what is the master.  Trystan's comment earlier in the thread "git master branch is a mirror of the debian package" scares me.  Surely a method of install for specific OS should be a mirror of a non-OS specific master?

Correct, and this is what the forked repos I've put together are - generic masters which also have debian packaging information that can otherwise be ignored (notwithstanding the lack of some kind of alternative install script for generic linux).

You can only put things from git into the apt repository, so don't worry that it might be the other way around.

We should simply tag the git master with the debian changelog version at the point when we generate that deb, then it will always be easy to see what's in apt, and what people have installed. This is how we manage a large (200+ system) apt repository under git control in the office.

Ideally there would be two tags in master on github right now: "8.0.1" and "debian/8.0.1" - if we had to tweak the debian installation script only, master would advance without adding "8.0.2" tag (because no functionality has been changed except the installation stuff for one distro) but we would add "debian/8.0.1-1" to denote the debian version built and deployed to apt.

The fact that github still says 8.0 (or even prerelease) is a good illustration of why debian packaging adds some needed discipline (IMO) because you have to increment to update through apt. You don't get the situation where there's doubt over which version of a particular version number people are running...

Excuse me for jumping into this thread, but I have a few questions that I am unsure of the answers to. I had installed EmonCMS on my NAS4Free / FreeBSD server last summer but recently had a hardware failure and lost all of my data :-(

I purchased new hardware and got NAS4Free running again, and followed my previous instructions to install EmonCMS. It appeared to work fine and the version I have running appears to be still MySQL based. I pursued this path as my initial search seemed to indicate that the Timestore variant was not fully setup for home installs yet. But reading through this thread seemed to indicate that it may be ready for that. Can someone bring me up to speed on whether Timestore and this branch is ready for a FreeBSD install attempt? The only mention of the two seems to be in a thread from last fall with no clear answers.

A quick search on the Timestore site doesn't seem to add much info as well, so any guidance would be appreciated. Thanks.

Dan

Actually, why don't I submit a pull request just to restructure the filesystem?

Forget all the debian packaging for just a moment. This would be like a bridging step, demonstrating that we can all still run emoncms with a more robust installation procedure, whatever the platform.

In practice, adding debian packaging to this just means adding a debian/ directory in the root folder, but perhaps by separating it into two logical steps, it will make it easier. We'd be separating upgrade concerns based on debian packaging and concerns based on not pointing apache directly at git (directory restructure)?

Dave: Thanks for all your work on this,

Im looking at:

https://github.com/Dave-McCraw/emoncms/tree/debian-packaging

and

https://github.com/Dave-McCraw/emoncms/tree/restructure

Would it be possible to have a debian "scaffold" repo which has  the folders: conf, debian, an empty www with a COPYHERE.txt readme, and the other copyright and install.sh files.

and then have the process of building the debian folder consist of copying over emoncms from the current repo into the debain scaffold repo?

Would that work?

it would be a separation of concerns in a way, what do you think?

Just to check that I understand the solution to the problem of nesting git repo's, are you saying that it is essentially the use of apt instead of git to install modules that solves the situation of what can happen if git reset is called? as in this instance git is no longer used and so git reset cant be called.

But under the apt install calling: sudo apt-get install event

still puts the event module in emoncms/www/Modules?

Someone could still install the module via git in the Modules folder if they really wanted to?

Here's the problem as I see it: we're not really 'installing' emoncms.

By that I mean that there's no separation between what we have running on servers and completely separate concerns like what version control system we use, what file and directory structure we want to organise our stuff, and how we choose to handle bug fixes and feature releases.

It's fantastic that emoncms already has a modular architecture, but all the teething issues with engines, visualisation etc. that people have been experiencing with v8 has me thinking that it might be even more awesome if we could split up the core of emoncms into smaller 'chunks'. That way we could roll out new engines without messing with visualisations, or add new visualisations without releasing a whole new major version of emoncms, and so on.

Literally having our git repos served by Apache poses major obstacles to this idea because you will get conflicts between repos when you restructure like this. What if a version of one repo which controlled certain files is mixed with a newer repo that claims that part of emoncms?

Another issue is that we can't include stuff like apache config files or schema definitions in the emoncms repository, otherwise they'll be public-facing and potentially exploitable because everything is under the apache docroot.

The alternative approach is to have the git repos (or alternative installers, like debian packaging) not be cloned under /var/www/ at all, but somewhere else, and then run an installer script that sets up apache config, any other configuration, moves files into a serveable structure (if necessary merging the files and directories, as with the contents of the Modules/ directory, and so on.

Although it may be slightly painful in the short term for existing installs, it will give us massively more flexibility going forward, as well as making it easier to upgrade and maintain emoncms for less experienced admins.

Specifically regarding the debian packaging: it works as you suspect (physically installing the files into an apache-served file structure) so it is possible to freely mix the git method with the debian method if the sysadmin prefers, apache can't tell if something was installed via git clone or apt.

The advantage of debian packaging is that we tie in with sophisticated versioning, installation and related toolsets, and it eliminates any requirement to use git as a prerequisite for an emoncms install. Everyone can use apt (who is running a Pi, anyway)... and unlike Git, it's possible to freely mix files in an arbitrary directory structure (some in /etc/emoncms, some in /usr/share/emoncms, some in /usr/docs/emoncs, whatever) and apt will keep track of which package owns which files automatically.

[edit] Just to add another point: although not everyone is installing emoncms on a Pi or other debian-derived system, the big advantage of setting up a debian installer is that we won't need to worry so much about manually documenting the install procedure for everyone else. All we really need to do is publish a short guide to help people understand any slightly cryptic aspects of the debian installation script, then say to people "if you're not installing on debian, open the debian install file and follow the steps in it manually". That way, there's less chance of forgetting a dependency, some configuration option etc. because we can confirm the functioning of the emoncms debian build programatically.

By that I mean that there's no separation between what we have running on servers and completely separate concerns like what version control system we use, what file and directory structure we want to organise our stuff, and how we choose to handle bug fixes and feature releases.

Just to be sure that Im following, your saying there is no separation with the current git approach while the debian aproach provides seperation. Or are you saying that there cannot be separation in general?

It's fantastic that emoncms already has a modular architecture, but all the teething issues with engines, visualisation etc. that people have been experiencing with v8 has me thinking that it might be even more awesome if we could split up the core of emoncms into smaller 'chunks'. That way we could roll out new engines without messing with visualisations, or add new visualisations without releasing a whole new major version of emoncms, and so on.

I think the teething issues are more a problem of not testing thoroughly enough on multiple platforms, the visualisation issue was because I had not tested those particular visualisations, which admittedly was a fair oversight. Maybe if we had a defined testing period where it was stated that the current state is likely to be buggy and that were looking for help to iron out most of the issues before the full release.

Literally having our git repos served by Apache poses major obstacles to this idea because you will get conflicts between repos when you restructure like this. What if a version of one repo which controlled certain files is mixed with a newer repo that claims that part of emoncms?

I get the use of debian apt vs git with the nesting git reset problem. but not sure if Im fully following you here.

Another issue is that we can't include stuff like apache config files or schema definitions in the emoncms repository, otherwise they'll be public-facing and potentially exploitable because everything is under the apache docroot.

Sure.

The alternative approach is to have the git repos (or alternative installers, like debian packaging) not be cloned under /var/www/ at all, but somewhere else, and then run an installer script that sets up apache config, any other configuration, moves files into a serveable structure (if necessary merging the files and directories, as with the contents of the Modules/ directory, and so on.

Yup got this too, and this is adhering to debian standards too as you mentioned. But then windows installs and shared server installs just need emoncms placed in public_html or www so we need both options I think.

Because debian is a specific installation case it seems to me that it would be simplest to keep the current emoncms git repo as it is (without the introduction of the emoncms app files within the www folder in the git repo) and have this additional repo with the debian specific stuff to which we copy emoncms when updating the debian package.

This allows us to seperate the emoncms app from the debian install while still having a debian package and all its benefits?

Just to be sure that Im following, your saying there is no separation with the current git approach while the debian aproach provides seperation. Or are you saying that there cannot be separation in general?

There should be separation which we can't get with git alone. We do get it through debian packaging (but that's just a special case of the general principal of "don't put the whole git repo under the docroot").

Indeed, although it's a positive thing that we are having this discussion, the reason we have to have it al all is only because we've welded together our source control and our deployment. Normally you would have all aspects of an application in source control and then install files into varied locations as required, so you can do whatever you like without any impact on people's installations.

We're now talking about having multiple git repositories to contain one source, to work around the fact that legacy installs involve cloning into the docroot... 

Maybe if we had a defined testing period where it was stated that the current state is likely to be buggy and that were looking for help to iron out most of the issues before the full release.

There are lots of possible improvements, the benefit of experience and hindsight :)

My point was more that the current git structure precludes us from doing any significant reorganisation - as this discussion underlines. I think we should grasp the nettle and break the link between git and the installation now, or we might have to do it later to support some kind of must-have feature, and it will be vastly more painful.

Yup got this too, and this is adhering to debian standards too as you mentioned. But then windows installs and shared server installs just need emoncms placed in public_html or www so we need both options I think.

Because debian is a specific installation case it seems to me that it would be simplest to keep the current emoncms git repo as it is (without the introduction of the emoncms app files within the www folder in the git repo) and have this additional repo with the debian specific stuff to which we copy emoncms when updating the debian package.

This allows us to seperate the emoncms app from the debian install while still having a debian package and all its benefits?

I get what you're saying and we could do that, technically. I'm sort of on the fence about it because I think we're storing up serious future pain by encouraging people to clone the repo directly and serve it.

I'd be interested to know what proportion of installs are onto a Pi / linux box and how many onto shared hosting, and where we want to position ourselves philosophically. After all, we introduced a Redis dependency in v7 which meant a large proportion of people couldn't even run v7 on shared hosting...

What I was attempting to do with my pull request on https://github.com/Dave-McCraw/emoncms/tree/restructure was to introduce a separate (shared hosting / non-debian) install script that would allow us to proceed by saying "either you install from apt, or you can install using this script, but in neither case do we support cloning emoncms under the Apache docroot any more" - the idea being that on a shared host you could just change the path in the install script to your shared hosting directory and it would still work.

I'm not actually sure that would work with the current pull, I guess it would preferably take an argument and use that as the directory for installation - but is this not a possible way forward?

I think we should be aiming to support, in order of platforms that generate the most forums posts/interest?

1) debian based systems: primarily ubuntu and rasbian on the raspberrypi but others should also work.

2) windows

3) linux shared servers

4) mac's

I originally though redis and timestore would be easy to get running on windows when v7 was released but after realising that it wasnt going to be the case I put a fair bit of work into making v8 compatible with windows which also luckily makes it compatible with linux shared servers where neither timestore nor redis could be installed either, but there was more call to support windows than shared servers hence the driver of these changes being the windows requirement rather than to get shared server compatibility.

Another question is what is best for developers? using git directly reduces the number of steps between developing, testing and uploading changes. We may still want to document the git install approach for developers as well as providing the debian package for easy installation? what do you think?

Emoncms documentation navigation could look like this

• Install on Debian Linux
• Install on Shared servers
• Install on Windows
• Install via git for development

On shared server's it may not be a possible to run the install script as you dont always have ssh access. A possibility is that the www folder in the new restructure is just copied to public_html...

We could start with two separate repo's, as the debianisation step only really concerns ourselves  and a minority who will want to build the debian package from scratch?

We then compile the debian package, serve it, update all the documentation to use it as the default install method, keep the current install procedure up there in parallel as another option for the time being. We could then re assess in a month or so, establish if its stopping us from doing anything?

Normally when developing I would symlink the relevant parts of the git repo into the deployed directory. But there are a lot of ways to do it, and it's true that building and installing a debian package every time you save your PHP isn't a quick way to develop.

However, there is a blindingly obvious middle way which only occurred to me just now. I was too focused on how to roll out the "proper" change and not thinking enough about the "minimum necessary" change.

We could just add the debian directory and config files to the existing git repo but not move anything into the www/ directory for web-served content. This might complicate the debian installer somewhat, but it could be worked around.

This way it would still be possible for people to clone the repo into /var/www/ or wherever else they liked, because we wouldn't be moving any files *at all* - only adding new stuff. And also cater for people to develop in-situ and then push their changes separately.

It's a bit ugly but from a functional point of view it would work just fine.

Good idea, the idea of keeping the current file locations the same for now is appealing, lets try it.

Thanks

OK - I've created a new push request with backwards-compatible debian packaging added in.

merged, thanks for all the work on the readme too, awesome!

No worries. I'm guessing there might be quite a bit of feedback with teething issues, but we'll soon be there... if I can get those PECL modules as debian dependencies it will really crush the install down to a couple of commands :)

@Hertog - thanks for the pointer on DIO... I've now submitted a pull request to get that as a proper dependency instead of via PEAR, and got raspberry pi module relying on it.

Nice :)

Hi

Tried an update.

Unpacking replacement emoncms ...
Setting up emoncms (8.0.3-1) ...
Replacing existing emoncms config as follows:
- MySQL configuration   : root connecting to emoncms db on localhost
- SMTP configuration    :  connecting to  to send as emoncms (noreply) (emoncms-noreply@localhost)
- Allow password resets :
Verifying feed directories exist and have the correct ownership...
- /var/lib/phpfiwa
- /var/lib/phpfina
- /var/lib/phptimeseries
- /var/lib/timestore
Attempting to create 'emoncms' mysql database (with the 'root' user)...
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
- Failed (check username and password are correct, using dpkg-reconfigure. Check grants allow table creation, etc...)
dpkg: error processing emoncms (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
emoncms
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I ran sudo dpkg-reconfigure emoncms --force

Replacing existing emoncms config as follows:
- MySQL configuration   : root connecting to emoncms db on localhost
- SMTP configuration    :  connecting to  to send as emoncms (noreply) (emoncms-noreply@localhost)
- Allow password resets :
mv: cannot stat `/usr/share/emoncms/www/default.settings.php': No such file or directory
Verifying feed directories exist and have the correct ownership...
- /var/lib/phpfiwa
- /var/lib/phpfina
- /var/lib/phptimeseries
- /var/lib/timestore
Attempting to create 'emoncms' mysql database (with the 'root' user)...
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
- Failed (check username and password are correct, using dpkg-reconfigure. Check grants allow table creation, etc...)

I guess the mv command is part of the problem. I was using raspberry as mysql password, should it be something else?

This is a test PI so I could start from scratch if needed but thought you might like to see the problem reported.

Regards

Ian

Thanks for flagging that up Ian. That's come about because I updated the debian packaging of emoncms to be tracked by a standalone git repo last night - I'll need to pop a quick update to apt to cater for it.

Please feel free to post any issues you encounter, it's a great help. You can do that here or you can create an issue on the pkg-emoncms repo :)

I ordered a second Pi so I could run a dev environment in front of my live Pi, but I haven't received it yet, so it's a little tricky to debug myself...

Ian - I've created an issue to track your bug: https://github.com/Dave-McCraw/pkg-emoncms/issues/1

Ok

I have updated with the latest  version:-

After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up emoncms (8.0.3-3) ...
Replacing existing emoncms config as follows:
- MySQL configuration   : root connecting to emoncms db on localhost
- SMTP configuration    :  connecting to  to send as emoncms (noreply) (emoncms-noreply@localhost)
- Allow password resets :
Verifying feed directories exist and have the correct ownership...
- /var/lib/phpfiwa
- /var/lib/phpfina
- /var/lib/phptimeseries
- /var/lib/timestore
Attempting to create 'emoncms' mysql database (with the 'root' user)...
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
- Failed (check username and password are correct, using dpkg-reconfigure. Check grants allow table creation, etc...)
dpkg: error processing emoncms (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
emoncms
E: Sub-process /usr/bin/dpkg returned an error code (1)

I have looked at settings.php in  /usr/share/emoncms/www

Password is raspberry as I expected from previously using:- 

sudo dpkg-reconfigure emoncms --force

I am now sure I must have at some time messed up the password in the mysql database. Is there a way to check what the password is? All access is via SSH as I am running the PI headless.

Regards

Ian

You should be able to log into mysql by running the following command:

mysql -uroot -p

(then type password when prompted).

If it doesn't work, I have no idea how you'd recover the mysql root password, sorry!

In an ideal world there would be an emoncms user set up with appropriate grants, and you'd be able to fall back on the root user from that. Future stuff....

PS. you can see the actual command being run here. You could just try copying that line into your terminal directly, changing the two variables for root and raspberry . If you can make that work, you'll know what the settings need to be.

Partially solved!!

I tried mysql -uroot -p with every password I thought it might be. In the end I gave up and hit return. I was straight in to mysql!

Somehow I had no root password set. I set it to raspberry, ran sudo dpkg-reconfigure emoncms --force, used raspberry as a password and update completed.

sudo dpkg-reconfigure emoncms --force
Replacing existing emoncms config as follows:
- MySQL configuration   : root connecting to emoncms db on localhost
- SMTP configuration    :  connecting to  to send as emoncms (noreply) (emoncms-noreply@localhost)
- Allow password resets :
Verifying feed directories exist and have the correct ownership...
- /var/lib/phpfiwa
- /var/lib/phpfina
- /var/lib/phptimeseries
- /var/lib/timestore
Attempting to create 'emoncms' mysql database (with the 'root' user)...
- 'emoncms' either exists already or was created successfully.
To update emoncms configuration, run 'dpkg-reconfigure emoncms'

Is there a case for a default root password in the debian packaged install to avoid this possibility?

I can now reach the PI with a browser from a different computer but I get:-

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

Regards

Ian

Hi Ian,

Setting a root password on mysql is something you'd attend to as part of setting up mysql. There's lots of stuff like that I guess - not something that falls within the remit of the emoncms package installer.

(However, that *is* taken care of in the Pi images we distribute, so perhaps it needs to be explicit in the debian install guide? Something like "common issues you might encounter while setting up emoncms"?)

FYI, I've updated the readme on https://github.com/Dave-McCraw/pkg-emoncms with the intention that it will be the single point of contact on the debian install procedure (I'll soon submit a pull request to remove this from the main readme). Any suggestions for this are very welcome.

Hi

During install you fill in the database information and the email information. The email and database defaults can just be skipped over but clearly a mysql password MUST be filled in. A note at the appropriate point in the readme that this must be filled in may save someone some grief (and that the password must be noted/remembered for future database maintenance).

This part of the readme could possibly mislead,

Note that your previous answers are pre-populated for convenience; this includes the passwords (just hit enter on the empty field to use the previous password).

I am guessing some PI users maybe unfamiliar with Linux and mysql if they come from a Windows background.

I certainly was a few months ago!

Ian

Ah yes. Because we call mysql with the -p flag it won't work if someone sets up passwordless DB access, but more fundamentally, you can't enter nothing to the debconf prompts, as that means "use previous".

I'll certainly update the readme to include a note on this, along with help on debugging.

[Edit: I've updated the readme and added https://github.com/Dave-McCraw/pkg-emoncms/issues/4 to refer people to in future.]