Archived Forum

You should have no issues running a full emoncms install on a Pi with a hdd, in fact although I can't think of any platform specific feature pro's or con's I do think there are more "linux" users than windows users so support and dev can be a little easier to find.

You should also be able to transport your existing database if you want to, I know this is possible linux to linux and I don't think the sql is any different if you use the export/import in phpmyadmin. You may need to "tinker" with the database if there are any windows specific addresses within the data maybe.

Paul

Hi,

just got my new RasPi-2 :-)

Now going to setup the stuff.
I am discussing with my self, what is the best way to go.

- Taking a standard wheezy and setup the sql-stuff, like i did on my Windows-7-PC
- Using an image from emoncms

I think, it would be better for me, to take the first choice. Because I do not need emonhub & co. I only want to use the RasPi for collecting my datas and show them. (like I described in the first post).

How are you powering the hdd? the Pi B+ and Pi 2 have new power management that can provide 1.2A but is limited in settings to 600mA.

If you intend running a small HDD direct from a Pi 2 or B+ run this command (one line)

sudo sh -c 'echo  "\n# uncomment to raise USB current limiting to 1.2A (from 600mA).\nmax_usb_current=1" >> /boot/config.txt'

to add a "max_usb_current=1" option to "/boot/config.txt" that will get picked up next restart.

The easiest way to set up a Pi HDD is to put the Raspbian OS image on BOTH an sd and the hdd, boot up on the sd and then edit the hdd to prepare it to boot to, so first you need to give the hdd a unique uuid and then mount the hdd

​sudo tune2fs -U random -L hdd  /dev/sda2sudo mkdir /media/hdd

sudo mount /dev/sda2 /media/hdd

then edit the hdd's fstab

sudo nano /media/hdd/etc/fstab

change " /dev/mmcblk0p2 " to " /dev/sda2 " and Ctrl+x, y, enter,

while we are here just edit the inittab (this step is only required if you're intending to use rfm2pi or serial port)

sudo nano /media/hdd/etc/inittab

edit the very last line by adding a hash so it reads " # T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100 "

edit the cmdline.txt and reboot (the next 2 lines are 1 line wrapped, be sure it is entered as 1 line "sudo" to "txt'")

sudo sh -c 'echo "dwc_otg.lpm_enable=0 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt'

sudo shutdown -r now

Then if you want you can use the script I use to install full emoncms (I haven't tried it wit Pi2 yet but see no reason it shouldn't work) with this one liner

git clone https://github.com/pb66/dev-emoncms.git ~/dev-emoncms && ~/dev-emoncms/install

​It takes about 20mins on a B+ but it needs no interaction, you can get on with something else and leave it to to it. So that it can create the emoncms SQL tables without interaction mySQL is installed without a root password so you must create a root password for mySQL and change user Pi password asap when it's done.

It's work in progress and would love to hear how it goes if you do.

Paul 

Hi,

thx for the detailed explanation.
Its hard stuff for a linux noob. But I will try my best. (and have some questions...)

power management:
I applied a 1,8" 20GB HDD. My USB-doctor is measuring: current < 0,45A  on writing data  :-)

The easiest way to set up a Pi HDD is to put the Raspbian OS image on BOTH an sd and the hdd
Hmmm.... Just asking me....why this?
I know, that the PI only can boot from the SD. What is the reason for putting the OS on the HDD?
I thought that the normal way is to put the OS on the SD, mount the HDD and put the whole emoncms-stuff on the HDD.

Put the OS on HDD:
How? The same way as on the SD-Card? Copy the same image (using win32diskimager in my case) onto the HDD?
Isn't this a problem, if I once make an update of the OS with "sudo apt-get update" or similiar?
Then I have different OS on SD<->HDD. And which one do I update?

But I think my main-problem is, that I do not completely understand how the Pi is working with SD and HDD together, when using your way.
In my own words: (pls correct me if wrong)
On the SD there are two partitions:

/dev/mmcblk0p1            8192      122879       57344    c  W95 FAT32 (LBA)
/dev/mmcblk0p2          122880     7744511     3810816   83  Linux

The first one, I can even read out with Windows.
Is this the partition where the Pi starts booting?
On the second one there is the whole linux-stuff.
With your change of the cmdline.txt, you want to force the system to switch from SD to HDD? And this is only possible, because on the second partition of the HDD "/dev/sda2" there is the same as on "/dev/mmcblk0p2"
And when I put the raspian image on the HDD, there will also be a first partition (/dev/sda1) like as the SD-card, but this partition is of no use

edit: aahhh, i have seen, that the files, that are on the first partition are visible under linux in the folder /boot. Good to know. Means I can make changes here from my Windows-PC

Your command for editing the cmdline.txt

sudo sh -c 'echo "dwc_otg.lpm_enable=0 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt'
 

default cmdline.txt: what about the "console=ttyAMA0,115200?
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

sudo tune2fs -U random -L hdd  /dev/sda2sudo mkdir /media/hdd
This are two commands? (sorry for this silly question)

• sudo tune2fs -U random -L hdd  /dev/sda2
• sudo mkdir /media/hdd

regards/brutzler

EDIT:
tried the stuff with "OS on HDD"
- How can I expand /dev/sda2 ? Because of writing the image to the HDD, there is only 4GB used.
- sudo raspi-config -> expand filesystem failed, because of being no SD-card.
Can I resize an active partition? Or do I have to turn back, boot completely from sd-card, unmount sda2, resize sda2, mount it, and make changes in cmdline.txt again?

The HDD draws the most during startup firstly spinning up to speed and then whilst thrashing about through the booting up, which is also when the Pi needs more power. a settled HDD writing a stream of data to one block after another will be drawing minimal current. The Pi B had less to offer and wasn't as regulated but the B+ has a better supply but it is restricted to offer better protection.

If your HDD spins up and works okay, great but if you find it keeps starting and stopping at boot it will be lack of power. the same HDD I used with an original B without problem, wouldn't run on a B+ but since getting the Pi 2 and researching it due to experiencing the same power cylcling i have set both the B+ and the Pi 2 to 1.2A and the HDD will run on any off them. 

I don't know what mine draws but it's spec'd at 1A during spin up. And I can run a Pi 2, hdd, wifi dongle and rfm2pi without a usb hub without any problems off one supply to the Pi .

With regard to how to use the HDD, you could just have the OS on the sd card and just use the hdd for data I suppose but I don't see any benifit. I wouldn't concider spliting the filesystem over both either use the SD card and hdd for data or just use the hdd for all. The hdd is running anyway and is more reliable than the sd card. By keeping a 2nd redundant copy of the OS on the sd card if you do something that locks you out of the hdd you can change the cmdline.txt in a pc and boot to the sd card and access/investigate the HDD from the card. You can also make back up images of your hdd image from the sd card.

You do not have to keep the 8gb sd card in the Pi though I have a pi that boots to hdd using an old 64MB nokia sd card .

Once you are running on the hdd it can be updated identically to the sd card via apt-get, the only difference is the media the OS is installed to the OS is still exactly the same except for the location refs ie /dev/sda2 rather than /dev/mmcblk0p2. Yes the 2 OS's will get "out of sync" because you will only install SW to the hdd and the sd won't get updated, you can of course run apt-get update/upgrade on the sd card anytime then the OS's will be identical minus the additional software you have installed eg emoncms.

The 1st partition on the hdd will be totally redundant and never used or updated but that doesn't matter it's only 58MB you can delete it if you want but then your file system will be on sda1 not sda2 so you will need to change the refs in fstab and cmdline.txt, it really isn't necessary but if you want I can walk you through that later

Yes you can use win32diskimager to put the image on the hdd exactly as you would the sd card.

"console=ttyAMA0,115200" is removed from the comline,txt to free up the serial port, essential if you were wanting to use the serial port eg for a rfm2pi but as you're not you can leave it in if you prefer, if you have no preference I would stick with the guide as future support will probably assume the serial port is available as that is the OEM norm. The important part is to change /dev/mmcblk0p2 to /dev/sda2.

Yes those are 2 seperate commands that have some how ended up on the same line during a cut and paste, sorry about that

sudo tune2fs -U random -L hdd  /dev/sda2

will name the partition "hdd" and give it its own UUID (Universally Unique ID) quite amusing name really since almost everyone uses the Raspbian OS image and then other images get made from that passing the same UUID from image to image so half the planet are using the same unique id. 

sudo mkdir /media/hdd

sudo mount /dev/sda2 /media/hdd

just creates a directory and mounts the hdd as part of the sd cards filesytem so you can edit the fstab etc.

Paul

OK,

understood all !

Now only the "problem" with the resize of the sda2-partition.

I tried out of Pi with "MiniTool Partition Wizard" on my W7-PC.
I see there is ~16GB space after the 4GB hdd(EXT4)-partition. But I can not resize it. Looks like this is blocked.

I do not dare to install emoncms before i resize the hdd!

Ahh you've edited since I started writing.

No you can do everything from the hdd once you have booted to it. you can use fdisk to increase the size of the partition and use "sudo resize2fs" to expand the file system into the larger partition. 

bigger isn't always better so don't rush into expanding the partition as big as you can. if you keep it small its easier to create a restore image and if you decide to add other partitions for other things you'll have room to do so, eg a partition to store just data or a partition to store backup images etc. you can always make the partition larger if you need more room but if the data and files get spread out across the partition it may not be easy to reduce to fit in something else.

I would recommend getting rid of "wolfram-engine" and "sonic-pi" IF you are not going to use them, this will free up 520MB which means you can add emoncms and all it's associated software and end up with a smaller image than you started.

If you create a data partition on the hdd and mount it as ~/emonData and use my script to instal emoncms all the data files including mySQL will be on that partition, which means you can easily back-up data and image separately and more importantly if you decide to reinstall the OS and emoncms etc down the line all your dashboards and data etc will be ther for the new emoncms to find.

If I were you I would add a data partition half way along the hdd, use fdisk and start the new partition 1/2 way but only make it a couple of gigs. this way you have left room to increase sda2(the filesystem) if needed later and you also have room to increase the new sda3  (data) and/or add another partition later.

EDIT - These are just my preferences. there isn't really a single right way, so don't feel you must do it this way, if you prefer one big partition that will work equally well it's just about options really and once you allocate all the space it's gone (well, it's not so easy to get back at least)

OK

You mean like this (I have a 20GB HDD):

0 - 56MB: boot (sda1)
56-4GB: hdd (sda2)
4GB...10GB: space to extend sda2
10GB - 12GB: data (sda3)
12GB...20GB; space for extend data, or make something different.

I would recommend getting rid of "wolfram-engine" and "sonic-pi"

Found this:

- sudo apt-get purge minecraft-pi wolfram-engine sonic-pi scratch -y
- sudo apt-get autoremove -y

Spot on!

However fdisk isn't the friendliest of utilities, its easy to use in that its logical and clear BUt it doesn't talk MB's and GB's to be prepared to do some maths, That's why I used the terms I did the start point is half way between the start and the end regardless of unit of measure and about 2 gig is about half of about 4gig when you print the table and can see the size of sda2 and the total size of sda you can enter the "clusters". there is a proper calc I have it here some where but it's not that important.

and yes apt-get purge should clear all refs so not sure about the second line.

Hi,

have ~590 MB more free space.
Now your script is running.

Shit:
If you create a data partition on the hdd and mount it as ~/emonData and use my script to instal emoncms all the data files including mySQL will be on that partition

Didn't read your post properly. Forgot to make the data partition before starting the script.

--> Start from beginning (Raspian-Image on HDD) or only make the data-partition and restart your script?

EDIT:

script done:
--> One Error:
ERROR: problem running ufw-init

But it doesn't look bad.
With http://"Raspi-ip"/emoncms/ i get the login screen

....going to bed now. 2:30 AM

You don't need to start again, once it has installed and it's working you can create the data partition and temporarily mount it as some thing else eg "tempData" copy everything in emonData to tempData. unmount tempData and remount it as emonData.

​Paul

Ok,

only one question to the place of the folder:

After installing emoncms with your script, I got the folder:  /home/pi/emonData. In this there is
- mysql
- phpfina
- phpfiwa
- phptimeseries

Then I made a new partition, formatted with ext4, and mounted in a parallel-folder

pi@raspberrypi ~ $ ls
dev-emoncms  emonData  python_games
pi@raspberrypi ~ $ sudo mkdir /home/pi/tempData
pi@raspberrypi ~ $ ls
dev-emoncms  emonData  python_games  tempData
pi@raspberrypi ~ $ sudo mount /dev/sda3 /home/pi/tempData

Is this OK so far?

And another question?
Where is the emoncms-software?
On windows I am used to find it in the wamp-folder: C:\wamp\www\emoncms
the mysql-folder is not accessible
pi@raspberrypi ~/emonData $ cd mysql
-bash: cd: mysql: Permission denied

And on copying with WinSCP I can not copy the mysql-folder too. (I wanted to copy the emonData to my lokal disk, and copy back to tempData, as you described)

pi doesn't have access to mySQL you would need to use sudo.

make a not of the permissions for emonData (incase something get altered it will be easy to compare)

    ls -la ~/emonData

make a copy of the data files (just in case)

    sudo cp ~emonData ~/emonBAK

​move the data files to the partition

    sudo mv ~/emonData ~/emonTemp

​unmount emonTemp

    sudo umount ~/emonTemp

​remount as emonData and check permissions again

​    sudo mkdir ~/emonData

    sudo mount /dev/sda3 ~/emonData

​    ls -la ~/emonData

you should have the same output as before, if you do try emoncms and ​if all is well delete both ~/emonTemp and ~/emonBAK

​Paul

​

Hi,

thx. tried and..... have some notes:

sudo cp ~emonData ~/emonBAK  -> sudo cp -r ~/emonData ~/emonBAK  (Only this worked for me)

I mounted /dev/sda3 to /home/pi/tempData. Guess its the same as: ~/tempData
With this: sudo mv ~/emonData ~/tempData I only get a subfolder: ~/tempData/emonData

To get to my aim,I copied the subfolders from emonData seperately:
sudo mv ~/emonData/mysql ~/tempData/mysql
...
...

The rest looks like it worked.
Checking permission rights:

pi@raspberrypi ~ $ ls -la ~/emonData
total 40
drwxr-xr-x 7 root     root   4096 Feb 16 17:20 .
drwxr-xr-x 7 pi       pi     4096 Feb 16 17:22 ..
drwx------ 2 root     root  16384 Feb 16 14:27 lost+found
drwx------ 5 mysql    mysql  4096 Feb 16 14:22 mysql
drwxr-xr-x 2 www-data root   4096 Feb 16 01:19 phpfina
drwxr-xr-x 2 www-data root   4096 Feb 16 01:19 phpfiwa
drwxr-xr-x 2 www-data root   4096 Feb 16 01:19 phptimeseries

the same (when we talk about the 'drwxr-xr-x') What does the number behind this coloumn tell us?

"sudo cp ~emonData ~/emonBAK  -> sudo cp -r ~/emonData ~/emonBAK  (Only this worked for me)"

okay, thats because the emonBAK folder didn't exist 

"With this: sudo mv ~/emonData ~/tempData I only get a subfolder: ~/tempData/emonData"

try sudo mv ~/emonData/* ~/tempData adding the " /* " should just move the contents not the folder

Paul

sudo mount /dev/sda3 ~/emonData

This is only temporarely. After reboot the /emonData-directory is empty. I have to mount again, to see mysql and the php*

Actually now, I get a database-connection error after reboot. logical

EDIT:

Added this in /etc/fstab:
#Datenpartition emoncms
/dev/sda3       /home/pi/emonData       ext4    defaults        0       0

And I found the emoncms-stuff :-)

Its in /var/www/emoncms....
Is it the same range, as i am used from my emoncms 8.4.0 on Windows?

Yes if its all working we need to add a line to the fstab to automatically mount emonData

sudo nano /etc/fstab

and insert a line after the "/dev/sda2" line 

/dev/sda3     /home/pi/emonData    ext4     defaults,    0    2

if the numbers you are asking about is the 7 7 2 5 2 2 2 I can't say I know off the top off my head

Hi,

sorry for the last question. If I would sometimes use my brain earlier.....

All in all, I now have a emoncms-working Pi.
Will test it the next days, how it is working.

If I could force the apikeys, it would be an easy change 1:1 for me. But because of different keys, i have to change the program of my data-sending arduinos.

Thx for your great support.

If you install phpmyadmin you can edit the sql tables the apikeys are found in the user table.

Or if you already have data tables you could create your own "emonData/olddata" folder note the structure and permissions and carry over your previous date.

The only thing that ties emoncms to the data is the emoncms password and user in settings.php.

eg if you were to comment out the username and password in settings.php and add you old ones, plus add the new "/olddata" folder level (inside emonData) to all data engine paths in settings.php you could have all your old data too.

always comment and add new lines so you can undo if things don't work out and/or put all the existing stuff from emonData in emonData/newfiles

Paul

>> If you install phpmyadmin...

This tool I was missing ;-)

On my windows installation with wamp, I sometimes used this to have a backdooor to my sql-table.

Found this:

- sudo apt-get purge minecraft-pi wolfram-engine sonic-pi scratch -y
- sudo apt-get autoremove -y

apt-get purge should clear all refs so not sure about the second line.

Not that it's going to make a big difference in disk space, but after zapping wolfram-engine and sonic-pi, autoremove gets rid of 10 more files. (a total of 19.5MB)

I looked into this after autoremove was mentioned and did some tests too.It does indeed make a difference, as does apt-get clean as temporary files used during installation and updates etc don't get automatically deleted either.

The latest Raspbian OS image (2015-01-31) is 2.4G on a 2.9G partition and has 393M free

(not entirely sure where the other 100+ MB is though, I assume the GiB v GB and MiB v MB etc)

Purging wolfram-engine sonic-pi minecraft-pi and scratch deletes 558M making 951M free

using auto-remove to delete packages now no longer needed gave 23M making 974M free

then using apt-get clean freed up a further 35M giving us a total of 1009M free

That is 616M reclaimed. 

However fully updating and upgrading the OS etc added 120MB making  889M free

But using apt-get clean again removed a bulky 116MB returning the total to 1005M free

Still 612MB smaller than we started with.

Adding a full emoncms with all dependencies added 238MB making 767MB free

And adding emonHub added a measly 2MB so we still have 765MB free

The final figures were 2G of a 2.9G partition used with 753MB free (~150MB now missing)

The main thing is the image is now 360MB smaller after installing emonhub and emoncms. The other great thing is if you are running emonCMS low-write on a 4G card the full OS and emon stuff  only uses half and gives you over 600MB additional storage (which is probably about a 50% boost)

I also believe if the resulting image was subjected to resize2fs -M (has to be unmounted) that image could be squashed and the partition made smaller to facilitate a smaller ZIP for download, maybe 800MB or so.

Paul

PS - I recorded these values for my own curiosity whilst doing a fresh install that had to be done anyway,

Thanks for the additional info about running apt-get clean, and zapping scratch as well.