How to build a repurposed PC super computer cluster at home for embarrassingly parallel problems

Here’s the thing super computer clusters generally use MPI to handle the splitting of tasks amongst compute nodes. The problem is that MPI needs to be incorporated down at the programming level so YOU have to learn it if you want to use it… How poor is that?

Well gnu-parallel offers an alternative solution. By utlising the power of gnu-parallel and the masive software repositories of ubuntu we can now solve many embarrasingly parallel problems in parallel across multiple machines.

In the following article I will go further and explain how you can build a live CD whcih you can drop into any old repurposed PC thereby adding its power to the collective with NO CONFIGURATION ON THE ADDED MACHINE WHATSOEVER.

Let me restate that because I think you’re going to like it!

Once you have the master PC ready and the LIVECD.iso ready ALL YO NEED TO DO TO ADD MORE POWER TO YOUR CLUSTER IS TO BURN A CD, PLUG IN THE NEW PC, PUT IN THE CD, AND TURN IT ON. That’s it. finished. No mucking about configuring individual pcs etc etc. More like put your feet up and have a beer!

OK for my example embarrassingly parallel problem I’m going to look at running a custom GIS algorithm that I build in python using the QGIS libraries. The proces for building the luster is generic but if you have a specific embarassingly parallel GIS problem that could be scripted for QGIS and sped up over a cluster then follow along all the way to the end with me.

By leveraging the power of gnu-parallel, python, and qgis you can write gis processing scripts and concurrently utilise any and all processing power you care to throw at the problem.

OK the main crux of the solution is that you need to be able to create a loop in BASH to solve the problem. Once that’s done we can rewrite the BASH command for gnu-parallel and give it to the cluster. Sure you could wrap your problem in a loop then wait for a week to process your files or you could build a parallel solution. So using my example of a problem that could be solved using python scripts in QGIS thats exactly what I’m going to do. Plus I’m going to do it in such a way that, not only could you potentially parallelise any embarrassingly parallel problem, provided it could be solved in a loop in linux but you could add or subtract any number of old second hand repurposed PCs to the super computer cluster as a means to add low cost computing power to the problem.

In fact the beauty of this solution is that its not restricted to QGIS.  The instructions I will lay out will enable you to build a low cost repurposed PC based super computer cluster for application to any embarrassingly parallel problem that can be solved using linux based programs. Actually a lot of the insrteuctions here (up to the point of the GPC cluster control script are identical to the requirements of a “normal” mpi based beowulf cluster. That includes some of the difficulties of nfs sharing over such a beast.

The basis of data sharing in this system is going to be via RAM folder on the master node shared via nfs to all the slaves over a gigabit network.  I’m hoping that by putting the sahred data in ram and then sharing over gigabit we should see some good network throughput for the shared data. There may be the possibility to add “netwrok teaming” to the master as well to further improve throughput but thats beyond the scope of this discussion.  In anycase since you’re holding ALL the data inthe ram of the master node you’re going to want to max out its RAM!

Now, an overview of the hardware and some jargon:

The cluster I will build needs:

  1. One linux based PC to be the host or “master” node (Fill it with RAM! and make sure it has gigabit ethernet)
  2. Access to an internet webserver to host public ssh keys online
  3. A gigabit capable switch with plenty of free ports
  4. A DHCP server (optionally with pxe capability for CD ROMless booting) Your current modem router will probably suffice. If your network has more than one PC on it connected to the internet then you’vemost likely got a DHCP server already.
  5. A bunch of CD (or pxe) bootable PCs to be my slave nodes.
  • The slave nodes dont need harddrives (they wont be using them any way)
  • The slave nodes do need to boot from CDROM or else you’ll need a pxe boot server but I’m not going to cover that here
  • The more RAM the better (1GB is a bare minimum)
  • The more powerful CPU the better
  • They will definitely need Ethernet ports (preferably gigabit) and enough cabling to connect all of them to the ethernet switch.
  • Save the most powerful machine you have (with a gigabit ethernet port) for the master node

Now I dont want to get in a bun fight over linux distributions – yes they all have their own proponents and detractors. Suffice to say that there is SUBSTANTIAL support for ubuntu in terms of trouble shooting and most importantly software repositories. The point being that by leveraging off the massive ubuntu software repositories and ppa system we can solve almost any problem with comparitively less programming ourselves.

A major theme in this project therefore is the fact that each compute node will have an install of ubuntu on it albeit with no gui and only the software we need installed. The real beaty of the software system used by ubuntu (and yes debian etc) is the fact that if you know you need a certain program but you dont know the software dependancies of that program “apt” (the software management proces in ubuntu) will deal with all of that for you.

OK time for an overview of the software we will use:

One the master node we will install Ubuntu 14.04 LTS 64 bit desktop. It is within this machine that we will prepare the operating systems for the slave nodes and from where we will launch the parallel operations.

On the master PC we need to add all the software required for building an Ubuntu live CD from scratch as well as the gnu-parallel program and a small contingent of networking programs for monitoring the cluster. I’m going to add the core QGIS software to this machine as well and inlcude the master as acluster node during processing, while this isn’t absolutley nessasary I’d be a fool not to utilise the cpus of the most powerful machine in my cluster.

In addition to the wholly dumb slaves in my cluster I have a handful of othe linux based machines I’m going to pull into the flock when running a parallel process such as my media server, an XBMC box and another ubuntu box I build to run my arcade machine. Most oft he time these machines are idle so there’s no reason not to include their procesing power too!

OK lets get started I’ll assume you have ubuntnu desktop installed on the master system and it’s plugged into the internet and has a local ip address. I’ll assume you know how to open a terminal and are somewhat conversant with bash commands. If you have never come across the command prefix “sudo” before you’ll probably want to stop now because it’s going to get pretty tricky from here on in. All code snippets with <you value> need to be edited by you before you copy and paste them into the terminal.

Install the required networking software on the master

sudo apt-get install ssh
sudo apt-get install nfs-common
sudo apt-get install nmap

Create an `ubuntu` user to match the user which will run on the slave nodes

sudo adduser ubuntu
sudo usermod -u 999 ubuntu
sudo groupmod -g 999 ubuntu
sudo usermod -g 999 ubuntu

Add yourself to the ubuntu user group. This way you can access the shared folder that you will create for the ubuntu user later.

usermod -a -G ubuntu $USER

Create public keys for your master user and your master-ubuntu user. Make sure you create the public keys with NO PASSWORD else you wont have passwordless ssh into your slaves.

su ubuntu
mkdir -p ~/.ssh && cd ~/.ssh
ssh-keygen
exit
mkdir -p ~/.ssh && cd ~/.ssh
ssh-keygen
sudo cat /home/ubuntu/.ssh/id_rsa.pub > ~/.ssh/pub.keys
sudo cat /home/$USER/.ssh/id_rsa.pub >> ~/.ssh/pub.keys

This has created a file in  your home directory called pub.keys. The next step is to take the pub.keys file and make it available online so that a slave can access them during boot and open the drawbridge for the master so-to-speak. I’m not going to cover that here. Sorry.  There are so many ways to make a file available online, each one requiring a long how-to you will just have to do it your self. The alternative is to add the file to the CDROM but I dont recommend this.

Very briefly: to have a web server you need

  • a static ip address
  • An A-record for a domain name pointing at that ip address
  • open port 80 and maybe 443 between the internet and your webserver
  • A webserver configured to allow file download from a web address

In my case I have an apache2 web server providing https on port 443 using http auth to allow password authenticated downloads of files using the htaccess file:

Options +FollowSymLinks +ExecCGI
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^(.*)$ authenticated_download.php [QSA]
</IfModule>

And the following authenticated_download.php file

<?php
if (!isset($_SERVER['PHP_AUTH_USER'])) {
header('WWW-Authenticate: Basic realm=""');
header('HTTP/1.0 401 Unauthorized');
echo 'File request cancelled';
exit;
} else {
$parts = explode('/', rtrim($_SERVER['REQUEST_URI'], '/'));
$filename = array_pop($parts);
if($_SERVER['PHP_AUTH_USER']=='ubuntu'&& $_SERVER['PHP_AUTH_PW']=='livecdubuntupassword') {
if(file_exists($filename)){

//Set it as Content Type
$finfo = finfo_open(FILEINFO_MIME_TYPE);
header('Content-Type: ' . finfo_file($finfo, $filename));
finfo_close($finfo);

//Specify the filename
header('Content-Disposition: attachment; filename='.basename($filename));

//No cache
header('Expires: 0');
header('Cache-Control: must-revalidate');
header('Pragma: public');

//Define file size
header('Content-Length: ' . filesize($filename));

ob_clean();
flush();
readfile($filename);
$file = 'downloadlog.txt';
date_default_timezone_set('New_Zealand/Auckland');
$date = date('m/d/Y h:i:s a', time());
// Open the file to get existing content
$current = file_get_contents($file);
// Append a new person to the file
$current .= $date." ".$_SERVER['PHP_AUTH_USER']." from IP ".$_SERVER['REMOTE_ADDR']." or ip ".$_SERVER['HTTP_X_FORWARDED_FOR']." requested ".$filename."\n";
// Write the contents back to the file
//$current .= var_dump($_SERVER$);
file_put_contents($file, $current);
exit;
} else {
var_dump($_SERVER);
}
} else {
sleep(2);
header('WWW-Authenticate: Basic realm=""');
header('HTTP/1.0 401 Unauthorized');
exit;
}
}
?>

If you wanted to store your keys on the CDROM I wouldnt blame you. Just watch the code later which creates the script to download to the slave, skip that part and pop the files directly there during the livecd build instead.

OK back to the master node terminal

Lets create the ram folder we will used to share all the data out to the cluster. We’re going to create the file, mount it as a tmpfs folder, then export it to nfs so the slaves can mount it.

Create the folder.

sudo mkdir /NETRAM
sudo chown $USER:$USER /NETRAM

Add the fstab entry to put it in RAM via a tmpfs mount. When you have finished addin gthe entrys to the bottom of your fstab folde r you can leave “nano” by typing ctrl-x

sudo nano /etc/fstab
tmpfs   /NETRAM    tmpfs  rw,user,group,exec,nodev,nosuid,uid=999,gid=999,mode=1770   0  0

Did you see the uid=999 and gid=9 notes? They are the user id and group id of your liveCD  user. Remember you added yourself to the livecd ubuntu user earlier.

Now lets mount the tmpfs folder and share the result into the home directory

sudo mount /NETRAM
ln -s /NETRAM /home/$USER/NETRAM

OK it’s time to create the nfs export to share the ram drive

sudo nano /etc/exports
/NETRAM *(rw,sync,fsid=0,no_subtree_check,async,all_squash,anonuid=999,anongid=999)

Note again the uid and gid are those of your livecd user as before. Note also that I’ve used all squash here which is less secure but enables easy permisions for the slave nodes. If you want it more secure then you need to teach yourself NFS permissions – and they arn’t easy! Particuarly when it comes to trying to mount an nfs share as a user via a remote command through ssh, which is what we’ll be doing! The point is that the access will be don elive the livecd user and will pass through the all squash and remain as the live cd user bu tonly because you set the uid and gid here.

Export your new ram folder to nfs

sudo exportfs -a

If you have terminal access on any machine in your network you can test for the nfs share with:

showmount -e MASTERNODE | grep  NETRAM

What “no address for ‘MASTERNODE'” or some such? OK so, rather than use the ip addres of my master node throughout the discussion I’m going to say “MASTERNODE” if you want you can add MASTERNODE to the hosts file of any linux PC you want by editing the hosts file as follows fo rthe example that the master node is at 192.168.0.10

sudo nano /etc/hosts
MASTERNODE 192.168.0.10

OK lets say you have some other machines on your network running Linux. I do and they’re idle a lot of the time so I’m going to add them to the fold.

As I make the required changes to my normal spare netwroked machines you’ll be able to see the changes that we’ll make in the live cd via scripts along the way.

So jump into a terminal on the idling linux machine and lets turn it into a part time slave!

First create the livecd ubuntu user

sudo adduser ubuntu
sudo usermod -u 999 ubuntu
sudo groupmod -g 999 ubuntu
sudo usermod -g 999 ubuntu

Get the master id keys using curl

NOTE YOU NEED TO KNOW THE URL,USER NAME AND PASSWORD FOR THE WEB SITE AS DESCRIBED IN THE PHP ABOVE OR ELSE MANUALLY COPY THE REQUIRED FILES INTO PLACE

sudo apt-get -y install curl
su ubuntu
cd ~
mkdir -p ~/.ssh && cd .ssh
#NB usernames passwords
curl -u ubuntu:qgisubuntupassword https://mywebsite.com/id_rsa.pub > id_rsa.pub
cat /home/ubuntu/.ssh/id_rsa.pub >> /home/ubuntu/.ssh/authorized_keys
chmod 777 /home/ubuntu/.ssh/authorized_keys
curl -u ubuntu:qgisubuntupassword https://mywebsite.com/masterip.txt > masterip.txt
MASTERIP=$(cat masterip.txt)
echo $MASTERIP

Note the master ip as supplied by the echo command.

You can add it to your hosts file to make things easier later

su <normalusername>
sudo nano /etc/hosts
ip.from.echo.command MASTERNODE

Add the master keys to the ubuntu users key repo

sudo chown ubuntu:ubuntu /home/ubuntu/.ssh
touch /home/ubuntu/.ssh/authorized_keys
sudo chown ubuntu:ubuntu /home/ubuntu/.ssh/authorized_keys
chmod 755 /home/ubuntu/.ssh
chmod 755 /home/ubuntu/.ssh/authorized_keys

Test that the masternode has share available to the part time slave with:

showmount -e MASTERNODE | grep  NETRAM

Edit the fstab to add the masternode nfs shared folder

sudo nano /etc/fstab
MASTERNODE:/NETRAM /home/ubuntu/NETRAM nfs rw,sync,noauto,users,exec,dev,_netdev 0 0
mkdir -p /home/ubuntu/NETRAM
mount /home/ubuntu/NETRAM

Go back to the master node and test if passwordless ssh to the part time slave works with

ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o BatchMode=yes ubuntu@ip.of.part.time.slave true

If you want you can run the GPC script now to have the slaves mount the shared NFS drives. Dont forget to close down the cluster after you’ve used it or else the nfs shares might become stale and you’ll need sudo access at every node to clear the stale folder handles.

Here is the  *current GPC script: write it into the path on the master node such as /usr/bin or usr/local/bin then run it with `GPC up` or  `GPC down`

sudo touch /usr/bin/GPC
sudo chmod +x /usr/bin/GPC
sudo nano /usr/bin/GPC

 

#!/bin/bash
set -ue
usage_message="Usage: GPC <up|down>"
MYIP=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1  -d'/')
#MASTER=${2:-$MYIP}
MASTER=192.168.88.239
IPRANGE=$MASTER/24
SSHLOGINTEST=loginfile.txt
PARALLELUSER=ubuntu
#MASTERUSER=richard
#sharedir="${HOME}/sshfs/${MASTER}"
SHAREDDIR="/home/$USER/NETRAM"
RAMDIR="/NETRAM"
col=80
up ()
{
if [[ "$MYIP" == "$MASTER" ]]; then
#SCRIPT FOR MASTER
cd ~/NETRAM
touch sshlogin
rm sshlogin
#touch sshlogin
(umask 000; touch sshlogin)
CPUCOUNT=$(grep "cpu\ cores" /proc/cpuinfo 2>/dev/null |sort -u |cut -d":" -f2|awk '{s+=$1} END {print s}')
CPUCOUNT=$(grep -c "processor" /proc/cpuinfo)
echo "$CPUCOUNT/:">>sshlogin
#GET LIVE IPS ON LOCAL NETWORK
echo -n "LOCATING REMOTE NODES"
LIVEIPS=$(nmap -sP -PS22 $IPRANGE | grep report| awk '{print $NF}')
#echo $LIVEIPS
#GET IPS WITH PASSWORDLESS LOGIN
PASSWORDLESSLIVEIPS=$(for IP in $LIVEIPS;do if timeout 1s ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o BatchMode=yes $PARALLELUSER@$IP true 2>/dev/null  ; then echo $IP; fi; done)
#echo $PASSWORDLESSLIVEIPS
#BUILD SSHLOGIN WITH NUMBER OF CPU CORES AT EACH HOST
if true; then
echo "[OK]"
else
echo "[FAIL]"
fi
echo "COUNTING CPUS"
echo "REMOTE IP           CPUs"
(umask 000; touch $SSHLOGINTEST) && rm $SSHLOGINTEST && for IP in $PASSWORDLESSLIVEIPS;
do
CPUCOUNT=$(ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no $PARALLELUSER@$IP grep "cpu\ cores" /proc/cpuinfo 2>/dev/null |sort -u |cut -d":" -f2|awk '{s+=$1} END {print s}')
CPUCOUNT=$(ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no $PARALLELUSER@$IP grep -c "processor" /proc/cpuinfo)
echo "$IP         $CPUCOUNT"
printf $(ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o StrictHostKeyChecking=no $PARALLELUSER@$IP grep -c "processor" /proc/cpuinfo)/$PARALLELUSER@$IP\\n>>$SSHLOGINTEST
done
#MOUNT SHARED TMPFS DIR AT RAMNET AND SHARE AS NFS SHARE
[ -d "$SHAREDDIR" ] || ln -s /NETRAM /home/$USER/NETRAM
#ORDER SLAVES TO RUN SCRIPT FOR SLAVES
for line in $(grep -v $MASTER "$SSHLOGINTEST" | grep -v ":"); do
userathost=$(echo $line | cut -d"/" -f2)
ssh -o StrictHostKeyChecking=no $userathost touch /home/ubuntu/GPC 2>/dev/null
ssh -o StrictHostKeyChecking=no $userathost rm /home/ubuntu/GPC 2>/dev/null
echo -n "Transferring Gnu-parallel Commander to $userathost"
if scp -o StrictHostKeyChecking=no /usr/bin/GPC $userathost:/home/ubuntu &>/dev/null; then
echo "[OK]"
else
echo "[FAIL]"
fi
ssh -o StrictHostKeyChecking=no $userathost './GPC up "$MASTER"' 2>/dev/null
done
else
#SCRIPT FOR SLAVES
if mount NETRAM; then
echo "$MYIP mounting success"
CPUCOUNT=$(grep "cpu\ cores" /proc/cpuinfo 2>/dev/null |sort -u |cut -d":" -f2|awk '{s+=$1} END {print s}')
echo "$CPUCOUNT/ssh -o StrictHostKeyChecking=no ubuntu@$MYIP">>NETRAM/sshlogin
else
echo "$MYIP mounting FAIL"
fi
fi
}
down ()
{
if [[ "$MYIP" == "$MASTER" ]]; then
#rm -R ~/NETRAM/*
for line in $(grep -v $MASTER "$SSHLOGINTEST" | grep -v ":"); do
userathost=$(echo $line | cut -d"/" -f2)
ssh -o StrictHostKeyChecking=no $userathost './GPC down ${MASTER}' 2>/dev/null
done
else
if umount NETRAM; then
echo "$MYIP unmounting success"
else
echo "$MYIP unmounting FAIL"
fi
fi
}
if [[ $# -eq 0 ]];then
echo "$usage_message"
exit 1
fi
if [[ "$1" == "up" ]]; then
up
elif [[ "$1" == "down" ]]; then
down
else
echo "$usage_message"
fi

Now I’m going to walk you through the LIVE-CD generation

At the end of this you will have a live CD which you can pop in any machine  and its biological and technological distinctiveness will be added to your collective.

most of the following post was pulled from https://help.ubuntu.com/community/LiveCDCustomizationFromScratch

What we’re going to do here is create a chroot jail and within that jail we’re going to install ubuntu. We’re going to add all the software our slvaes might need then we’re going to compress that system along with the bootstrap files required into another folder which we will convert to an iso which can be burned into a live cd and used to boot up slave nodes.

There are some differences/additions which make my process a little special.

In order for the LIVE CD to know where the master is we will add a boot script so that we can pull host info off the internet as descibed above and then dynamicallt that info (such as the master ip address) into the live cd OS during boot.

First lets get the software we need to build the chroot jailed system

sudo apt-get install debootstrap syslinux squashfs-tools genisoimage

Then install programs required on the master if not already

sudo apt-get install parallel nmap

Add the chroot folder

cd ~/
mkdir -p work/chroot
cd work

#Set the architecture to i386 to make it compatible with all x86 PCs

ARCH="i386"

Match the live cd to your current ubuntu release to reduce issues


RELEASE=`echo $(lsb_release -a 2>/dev/null | grep Codename)| sed -r 's/^.{10}//'`
echo "Your RELEASE is: "$RELEASE

sudo debootstrap --include=nfs-common --arch=$ARCH $RELEASE chroot
sudo cp -rf chroot chrootminimal

You have now setup the chroot jail. You can restart from this point to make any changes to the live cd installation that you will prepare from now.

Now we will copy internet access and repository acces to the chroot installation.


cd ~/work
sudo mount --bind /dev chroot/dev

sudo cp /etc/hosts chroot/etc/hosts
sudo cp /etc/resolv.conf chroot/etc/resolv.conf
sudo cp /etc/apt/sources.list chroot/etc/apt/sources.list
sudo cp /sbin/initctl initctl.bak
sudo chroot chroot

mount none -t proc /proc
mount none -t sysfs /sys
mount none -t devpts /dev/pts
export HOME=/root
export LC_ALL=C
#THE NEXT LINE IS ONLY REQUIRED IF YOU WANT TO ADD A PPA - YOU WILL NEED THE PPAs ID FROM ITS PPA PAGE. I'LL ASSUME IF YOU WANT A PPA YOU WILL KNOW HOW TO GET THE ID
#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 12345678  #Substitute "12345678" with the PPA's OpenPGP ID.
apt-get update

Now lets install a dbus into the chroot installation to get it up and running

apt-get install --yes dbus

dbus-uuidgen > /var/lib/dbus/machine-id
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

Now we will install the “live-cd” components into our chroot installation for later incoporation into the livecd iso

apt-get --yes upgrade

#Install the packes for building an iso
apt-get install --yes ubuntu-standard casper lupin-casper discover laptop-detect os-prober linux-generic

#USE TAB TO HIGHLIGH OK WITH NO GRUB INSTALLLATION
#YOU DONT WANT TO INSTALL GRUB HERE AS YOU WILL BE USING ISO/PXELINUX TO START NOT GRUB

Now we will install the packages that are specific to easy use of a cluster node

apt-get install --yes parallel ssh curl wget lm-sensors

Finally we will install those packes related to the actuall parallel programming issue at hand. In my case I wanted QGIS to be available to each and every parallel processing node so I installed it to the chroot installation now.


apt-get install --yes qgis

Now we will install/build a script which will access the internet for info about how to fins and connect with the master node in the cluster

#Build the boot script to set the master ip and import the master host key for ssh
#This will gather info about the master node at boot
#Note carefully the web address for the info
#YOU WILL NEED TO SET THAT UP
#YOU MAY WANT PASSWORDED ACCESS TO THOSE ONLINE FILES
#IN WHICH CASE USING CURL WITH AUTHENTICATION OTHER THAN WGET IS EASIER
#An alternative although not recommended option if you dont have access to an online webserver, is to hard code the master node ip and master node public key  files directly into the live cd. If you want to do this then create and locate respectively the masterip.txt file and the host_id_rsa.pub file the place them in the chroot installation for later retreval at:
/masterip.txt
and
/home/ubuntu/.ssh/host_id_rsa.pub
#masterip.txt is of the form required for the /etc/hosts file redirecting qgismaster to the master ip address eg the file simply contains 192.168.0.1 or whatever your masternode ip address is.

MASTERINFOADDRESS=https://myonlineauthdigestfilerepository/filesfolder/
MASTERIPFILE=masterip.txt
MASTERKEYFILE=id_rsa.pub
MASTERHOSTNAME=MASTERNODE
#NOTE THE USER NAME AND PASSWORD ARE FOR THE  http AUTH THAT YOU MADE EARLIER USING PHP
HTTPSUSER=ubuntu
HTTPSPASSWORD=livecdubuntupassword

echo \#\!/bin/sh>masterid.sh
echo "curl -u $HTTPSUSER:$HTTPSPASSWORD $MASTERINFOADDRESS$MASTERIPFILE > $MASTERIPFILE">>masterid.sh
#echo "wget $MASTERINFOADDRESS$MASTERIPFILE">>masterid.sh
echo "MASTERIP=\$(cat $MASTERIPFILE)">>masterid.sh
echo "MASTERHOSTNAME=$MASTERHOSTNAME">>masterid.sh
echo HOSTEDIT=\"\$MASTERHOSTNAME \$MASTERIP\">>masterid.sh
echo echo "\$HOSTEDIT>>/etc/hosts">>masterid.sh
echo "rm $MASTERIPFILE">>masterid.sh
#echo "wget $MASTERINFOADDRESS$MASTERKEYFILE">>masterid.sh
echo "curl -u $HTTPSUSER:$HTTPSPASSWORD $MASTERINFOADDRESS$MASTERKEYFILE > $MASTERKEYFILE">>masterid.sh
echo "mv $MASTERKEYFILE /home/ubuntu/.ssh/host_id_rsa.pub">>masterid.sh
echo cat\ /home/ubuntu/.ssh/host_id_rsa.pub\ \>\>\ /home/ubuntu/.ssh/authorized_keys>>masterid.sh

echo cat\ /etc/fstab\ \>\ /home/ubuntu/tmpfstab>>masterid.sh
echo echo\ \$MASTERIP\\:\\/NETRAM\\ \\/home\\/ubuntu\\/NETRAM\\ nfs\\ rw\\,noauto\\,users\\,exec\\,dev\\,suid\\,_netdev\\ 0\\ 0\>\>\/home/ubuntu/tmpfstab>>masterid.sh
echo mv\ /home/ubuntu/tmpfstab\ /etc/fstab>>masterid.sh

mkdir -p /home/ubuntu/NETRAM

#cat /etc/fstab > ~/tmpfstab
#echo MASTERNODE\:\/NETRAM\ \~\/NETRAM\ nfs\ rw\,noauto\,users\,exec\,dev\,suid\,_netdev\ 0\ 0>>~/tmpfstab
#mv ~/tmpfstab /etc/fstab

mv masterid.sh /usr/bin/masterid.sh
chown root:root /usr/bin/masterid.sh
chmod +x /usr/bin/masterid.sh
chmod 700 /usr/bin/masterid.sh
#ln -s /etc/init.d/masterid.sh /etc/rc5.d/S99masterid

echo description \"A job file for running scripts during live cd or pxe boot\">masterid.conf
echo author \"MrPurple\">>masterid.conf
echo start on runlevel [2]>>masterid.conf
echo exec masterid.sh>>masterid.conf

mv masterid.conf /etc/init/masterid.conf
init-checkconf /etc/init/masterid.conf

#A initctl too old error here means you need to update the host system from which you are running your debbootstrap within

chown root:root /etc/init/masterid.conf
#chmod +x /etc/init/masterid.conf
chmod 644 /etc/init/masterid.conf

#Copy host key to master for use by cluster script

#Clean up and export iso
rm /var/lib/dbus/machine-id
rm /sbin/initctl
dpkg-divert --rename --remove /sbin/initctl

ls boot
#Check the kernal numbers that exist choose the highest value and adjust the next line to match

ls /boot/vmlinuz-3.13.**-**-generic > list.txt

sum=$(cat list.txt | grep '[^ ]' | wc -l)
if [ $sum -gt 1 ]; then
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
fi

rm list.txt

apt-get clean

rm -rf /tmp/*

rm /etc/resolv.conf

umount -lf /proc
umount -lf /sys
umount -lf /dev/pts
exit
sudo umount -l chroot/dev

################################################################
# Build and add the ssh-keys for passwordless ssh on the master#
################################################################
#First make sure ssh is installed on the master/host
#then

sudo mkdir -p chroot/home/ubuntu/.ssh

#Build the key pair

#upload the public key to the public server defined above
#sudo cp ~/.ssh/id_rsa.pub chroot/home/ubuntu/.ssh/host_id_rsa.pub
#OR copy to website and get from script

#sudo chroot chroot
#cd /home/ubuntu/.ssh
#cat host_id_rsa.pub >> authorized_keys
#exit

#set allowed hosts in the master

sudo rm -R image/{casper,isolinux,install}
mkdir -p image/{casper,isolinux,install}

#Remember the kernal number from before?
sudo cp chroot/boot/vmlinuz-3.13.**-**-generic image/casper/vmlinuz
sudo cp chroot/boot/initrd.img-3.13.**-**-generic image/casper/initrd.lz
sudo cp /usr/lib/syslinux/isolinux.bin image/isolinux/
sudo cp /boot/memtest86+.bin image/install/memtest

#Now we create the splash screens

echo "splash.rle" > image/isolinux/isolinux.txt
echo "" >> image/isolinux/isolinux.txt
echo "************************************************************************" >> image/isolinux/isolinux.txt
echo "" >> image/isolinux/isolinux.txt
echo "This is an Ubuntu parallel/QGIS Live CD." >> image/isolinux/isolinux.txt
echo "" >> image/isolinux/isolinux.txt
echo "For the default live system, enter \"live\".  To run memtest86+, enter \"memtest\"" >> image/isolinux/isolinux.txt
echo "" >> image/isolinux/isolinux.txt
echo "************************************************************************" >> image/isolinux/isolinux.txt

#NOTE YOU CAN EDIT THE TIMEOUT TO GREATER THAN ZERO TO ALLOW FOR SELECTION OF ALTERNATE BOOT OPTIONS BUT I WANT HANDS FREE FAST BOOT TO DEFAULT

echo "DEFAULT live" > image/isolinux/isolinux.cfg
echo "LABEL live" >> image/isolinux/isolinux.cfg
echo "  menu label ^Start or install Ubuntu Remix" >> image/isolinux/isolinux.cfg
echo "  kernel /casper/vmlinuz" >> image/isolinux/isolinux.cfg
echo "  append  file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash --" >> image/isolinux/isolinux.cfg
echo "LABEL check" >> image/isolinux/isolinux.cfg
echo "  menu label ^Check CD for defects" >> image/isolinux/isolinux.cfg
echo "  kernel /casper/vmlinuz" >> image/isolinux/isolinux.cfg
echo "  append  boot=casper integrity-check initrd=/casper/initrd.lz quiet splash --" >> image/isolinux/isolinux.cfg
echo "LABEL memtest" >> image/isolinux/isolinux.cfg
echo "  menu label ^Memory test" >> image/isolinux/isolinux.cfg
echo "  kernel /install/memtest" >> image/isolinux/isolinux.cfg
echo "  append -" >> image/isolinux/isolinux.cfg
echo "LABEL hd" >> image/isolinux/isolinux.cfg
echo "  menu label ^Boot from first hard disk" >> image/isolinux/isolinux.cfg
echo "  localboot 0x80" >> image/isolinux/isolinux.cfg
echo "  append -" >> image/isolinux/isolinux.cfg
echo "DISPLAY isolinux.txt" >> image/isolinux/isolinux.cfg
echo "TIMEOUT 0" >> image/isolinux/isolinux.cfg
echo "PROMPT 0 " >> image/isolinux/isolinux.cfg
echo "" >> image/isolinux/isolinux.cfg
echo "#prompt flag_val" >> image/isolinux/isolinux.cfg
echo "# " >> image/isolinux/isolinux.cfg
echo "# If flag_val is 0, display the \"boot:\" prompt " >> image/isolinux/isolinux.cfg
echo "# only if the Shift or Alt key is pressed," >> image/isolinux/isolinux.cfg
echo "# or Caps Lock or Scroll lock is set (this is the default)." >> image/isolinux/isolinux.cfg
echo "# If  flag_val is 1, always display the "boot:" prompt." >> image/isolinux/isolinux.cfg
echo "#  http://linux.die.net/man/1/syslinux   syslinux manpage " >> image/isolinux/isolinux.cfg

sudo chroot chroot dpkg-query -W --showformat='${Package} ${Version}\n' | sudo tee image/casper/filesystem.manifest
sudo cp -v image/casper/filesystem.manifest image/casper/filesystem.manifest-desktop
REMOVE='ubiquity ubiquity-frontend-gtk ubiquity-frontend-kde casper lupin-casper live-initramfs user-setup discover xresprobe os-prober libdebian-installer4'
for i in $REMOVE
do
sudo sed -i "/${i}/d" image/casper/filesystem.manifest-desktop
done

sudo mksquashfs chroot image/casper/filesystem.squashfs -e boot
printf $(sudo du -sx --block-size=1 chroot | cut -f1) > image/casper/filesystem.size

echo "#define DISKNAME  Ubuntu Remix" > image/README.diskdefines
echo "#define TYPE  binary" >> image/README.diskdefines
echo "#define TYPEbinary  1" >> image/README.diskdefines
echo "#define ARCH  i386" >> image/README.diskdefines
echo "#define ARCHi386  1" >> image/README.diskdefines
echo "#define DISKNUM  1" >> image/README.diskdefines
echo "#define DISKNUM1  1" >> image/README.diskdefines
echo "#define TOTALNUM  0" >> image/README.diskdefines
echo "#define TOTALNUM0  1" >> image/README.diskdefines

touch image/ubuntu
mkdir -p image/.disk
cd image/.disk
touch base_installable
echo "full_cd/single" > cd_type
echo "Ubuntu Remix 14.04" > info  # Update version number to match your OS version
echo "http//your-release-notes-url.com" > release_notes_url
cd ../..

sudo -s
(cd image && find . -type f -print0 | xargs -0 md5sum | grep -v "\./md5sum.txt" > md5sum.txt)
exit

cd image
sudo mkisofs -r -V "$IMAGE_NAME" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o ../qgis-slave-ubuntu-remix.iso .
cd ..

#########
# TO DO #
#########

#for livecd/pxe set to run sudo mount commands
#for non live cd set to mount as user from edited fstab file
#need to find a way to test for sudo

ssh-rsa AAAAB3NzpC1yc2EAAAADAQABAAABAQDAw4MqNGvAfCLW0WJi3MzrzkLiXSoVwwe4zk/3yqzfAexcMUiHQZyLhLf4ZfJD5FnBuGGB73DsLa0Ax4zDWRGbS+pkdVxBPDY2xljEI1eTyg6ZhxdGFYXKaYLts8s5VVLS1i3Yj4tWWuLnAmouqsrNWDd0bKSls+bo4VqAFjo91Dh0Ymx+hbtLG/YKcbj1Dd17Xikpngh0VZS5ETsG6zcauuM9ajJqc40Uawzz/JTdoV5rqDbct6bqSP6W7d8NIUrc5w4eZu+SSxTjpKpRD1Ozij4frBnKEjbKuySHeXa1d2ptOUct2HeXa1d2ptOUct2YDmPQMhXrvucqe4XlXEHOvKhGl92GJDgneR richard@richard-B85M-D3HYDmPQMhXrvucqe4XlXEHOvKhGl92GJDgneR me@mypc

#########################################################################
# SCRIPT TO READ IN AND EDIT CONF FILES TO ALLOW CONNECTION FROM MASTER #
#########################################################################
#NEED TO GET IP ADDRESS OF MASTER TO APPEND TO HOST FILE
#NEED TO GET MASTER FILE SSH KEY AND INSTALL
#NB MASTER HOSTNAME GOES INTO GPC script

#SPECIFIC SETTING CHANGER
#!/bin/sh
if [ `grep -c '^setting=' example.conf` == 0 ]
then
echo "setting=value" >> example.conf
else
sed -i 's/^setting=.*/setting=value/g' example.conf
fi

#GENERIC SETTING CHANGER
#!/usr/bin/env awk
BEGIN { FS = OFS = "=" }
$1 == "setting" { $2 = "value"; found=1 }
{print}
END { if (!found) { print "setting=value" }

#ADD MASTER TO HOSTS FILE

wget https://secure.purplelinux.co.nz/GPC/masterip.txt
MASTERIP=$(cat masterip.txt)
MASTERHOSTNAME=qgismaster
HOSTEDIT=$MASTERHOSTNAME $MASTERIP
echo $HOSTEDIT>>/etc/hosts
rm masterip.txt

#ADD MASTER KEY TO .ssh folder
wget https://secure.purplelinux.co.nz/GPC/id_rsa.pub
mv id_rsa.pub /home/ubuntu/.ssh/host_id_rsa.pub

MASTERINFOADDRESS=https://secure.purplelinux.co.nz/GPC/
MASTERIPFILE=masterip.txt
MASTERKEYFILE=id_rsa.pub

echo \#\!/bin/sh>masterid
echo "wget $MASTERINFOADDRESS$MASTERIPFILE">>masterid
echo "MASTERIP=\$(cat $MASTERIPFILE)">>masterid
echo "MASTERHOSTNAME=qgismaster">>masterid
echo HOSTEDIT="\$MASTERHOSTNAME \$MASTERIP">>masterid
echo "\$HOSTEDIT>>/etc/hosts">>masterid
echo "rm $MASTERIPFILE">>masterid
echo "wget $MASTERINFOADDRESS$MASTERKEYFILE">>masterid
echo "mv $MASTERKEYFILE /home/ubuntu/.ssh/host_id_rsa.pub">>masterid

mv masterid /etc/init.d/masterid
chown root:root /etc/init.d/masterid
chmod +x /etc/init.d/masterid
chmod 700 /etc/init.d/masterid
ln -s /etc/init.d/masterid /etc/rc3.d/S99masterid

Leave a Reply

Your email address will not be published. Required fields are marked *


9 − = zero