This is a how-to for installing distcc, a distributed compiler, to run on a Tycoon cluster. This guide includes an introduction to using vxargs to execute commands on multiple remote machines at once (section 2.2); you may find vxargs useful for many tasks on your Tycoon cluster.
distcc is available at http://distcc.samba.org.
In this guide the client is the machine directing the compilation, probably your local machine, and the servers are the machines to which the jobs are distributed. Of course distcc can also use the client to do some of the work.
It is assumed that the user has established accounts on several machines and has downloaded the distcc tarball.
This guide first shows how to install distcc on the client, then on one server at a time, and finally on multiple servers at once. After the installation steps is a brief guide to getting distcc up and running.
1. Installation on client
1. Install gcc on client machine if you don't have it:
#yum -y install gcc.
2. Install distcc.
- Unzip and untar distcc tarball.
- Go into the distcc directory and install it with
$./configure
$make
#make install
2. Installation on multiple servers at once using vxargs
vxargs is a python script which allows you to execute a command on multiple remote hosts at once. It is available at http://dharma.cis.upenn.edu/planetlab/vxargs/. You may find many uses for vxargs on your Tycoon cluster; this particular application to installing distcc will give you an idea how to use the vxargs command syntax.
2.1 Using vxargs
There are many options available in the vxargs syntax but the most useful form for executing commands is as follows:
$vxargs -a [host list] -o [directory for log files] [your command]
Here [host list] is a text file with the IP address of the remote machines on which you want to execute your commands. The text file must have the machine addresses by themselves on separate lines, for example:
15.0.114.15
15.0.114.16
In what follows I will use "iplist.txt" to mean the file containing my host list.
[directory for log files] is a directory where vxargs will put files containing the stdout and stderr outputs of your commands ,as well as a status output (1 or 0), for each remote host. This is the only place you will be able to see the outputs of your commands and very important part of using vxargs.
Finally, [your command] is the command you wish to execute on the remote machine. The only difference in syntax from what you would usually type is that you put "{}", literally, where you would normally put the IP address of the remote machine.
The following instructions for installing distcc remotely illustrate the use of vxargs and its syntax.
2.2 Distcc installation process
1. Install gcc on the servers:
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh root@{} yum -y install gcc
Remember to literally include the "{}" in your command.
2. Open port 3632 on the servers:
$vxargs -a iplist.txt -o vxargs_log tycoon host bid {} 1000000000 "{ ('TCPv4Port', '3632'): 1}"
3. Install distcc on the servers:
- Copy distcc tarball to server: $vxargs -a iplist.txt -o vxargs_log tycoon_scp distcc-2.18.3.tar.bz2 {}:
Don't forget the colon at the end of the copy command.
- Unpack and install distcc.
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} bunzip2 distcc-2.18.3.tar.bz2
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} tar -xvf distcc-2.18.3.tar
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} 'cd distcc-2.18.3; ./configure'
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} 'cd distcc-2.18.3; make'
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh root@{} 'cd [your home directory]/distcc-2.18.3; make install'
Notice the syntax for executing multiple commands using tycoon_ssh. It is desirable to run the distcc installation commands (./configure, make, make install) from within the distcc directory so that the output files will be placed there.
4. Start the distcc daemon on the servers:
$vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} ./distcc-2.18.8/distccd --daemon -a {client IP address} --log-file log --log-level info
Remember that the client is the machine from which you will be directing the compile (executing distcc).
3. Installation on one server at a time
If you are having trouble using vxargs, or trouble running commands on the servers, you can install distcc on your servers one at a time.
To do this merely repeat the commands from the above section on installation to multiple machines, omitting the
"vxargs -a iplist.txt -o vxargs_log " from the front of each command. Remember to type the remote host's IP address where you
typed "{}" in the vxargs command. Alternatively, you can tycoon_ssh into the remote host and operate from there.
4. Running distcc
This is a brief guide to how to issue a distcc command. For more information see the distcc man page at http://distcc.samba.org/man/distcc_1.html.
Once distcc is installed on the client and servers, you have opened port 3632 on the servers and started the daemons there, there is one thing left to do. On the client, do an
$export DISTCC_HOSTS='{server 1 IP address} {server 2 IP address} ...'
This variable tells distcc where to send the jobs. You can include "localhost" in this list and jobs will be compiled on the client too. The order of the servers in the DISTCC_HOSTS variable determines the priority, so put the fastest machines first for best results.
The man pages include a list of other options for this command, as well as other shell variables distcc can use.
Finally, to compile using distcc, just type
$make -j {#} CC=distcc {your job}.
Here {#} indicates the number of jobs to run at once (an integer). This must be at least as large as the number of servers if you want to use them all. My personal experience has been that using a number between 2 and 3 times the number of servers gives the best results. You may find otherwise for your jobs.
There are other ways to send the make command to distcc; for example, you can set distcc to be the default compiler. See man pages.
This guide is also available in text format (attached below).
Attachments
- distcc_tycoon.txt (5.3 kB) - added by DennisWilkinson 3 years ago.
