About 9 or 10 years ago I was an active participant in “distributed-computing”. Projects with an enormous workload that call upon the world to help them compute their data and/or research. This is done by people running clients on their (home) computers, which download “work units”, calculate a result and send back the data to the project’s servers. Often this is combined with a competitive element: you receive points for a completed work unit. Statistics are generated on a regular basis and you can look up how well you did compared to others around the globe. People can also form a team together and compete against other teams.
The Dutch Power Cows is such a team, and a large one at that. Intended for Dutch speaking people, of course others are welcome as well, to gather under a single banner and compute for all kinds of projects.
Every now and then the Dutch Power Cows organize an event they call a “stampede”. They nominate a month in which they try to attract as many people as possible to work on a chosen project. Boosting their output significantly for a month for the project, surprising other teams, and often taking first place in the daily statistics during that month. And by doing so they increase attention for a project, make the statistics interesting again and possibly end up with some new members.
Recently they announced the start of a new stampede, starting April 1st. The project of interest this time is the Cosmology@Home project. As an ex-DPC member the news interested me and I decided to help out during this stampede. So I had to install a client on my machines.
Installing clients on Windows machines is easily done, and many instructions can be found. Most projects also offer support for Linux, and some even do for FreeBSD and Mac OS X. In the case of Cosmology@Home, only Windows and Linux support is offered. And instructions for FreeBSD are pretty much non-existent. So I was disappointed at first, because I want to run this on my FreeBSD box as well, preferably in a jail too! But than I thought about this: FreeBSD has the “Linuxulator” for running Linux binaries. Perhaps I can use that to run Cosmology@Home on my FreeBSD box that’s idling away most of the time.
What do I need to run this:
- optional: fresh jail to install everything in.
- Linuxulator (in the jail): Available as a port
- BOINC client: a FreeBSD port is available.
- Enough RAM, at least 512MB per CPU core and have enough left to run the system.
Set up a new jail
Using a jail for this is optional. If you want to run the client on the host OS, that’s fine too, you can skip this step.
qjail to manage my jails. If you use something else, like
ezjail, or do it by hand that will be fine too.
I create a filesystem for my jail. You will need at least 2GB of space to install a Linux environment and the client in. I like to use quotas to limit disk usage, but I have plenty of space, so I set my limit at 8GB to have some headroom. I disable compression for this jail because I don’t expect to gain much by it and I can use the CPU cycles I save on the project.
# zfs create -o quota=8G compression=off zroot/freebsd9/usr/jails/cow.sjoosten.nl
Let’s create a fresh jail, refresh my ports tree for them and start the jail.
# qjail create -n re0 cow.sjoosten.nl 192.168.0.103 # qjail update -p # qjail start cow.sjoosten.nl
Prepare the host
Before we dive into setting up the BOINC client on the jail, we need to set some option on the host first.
First, let’s load the
linux kernel module. It’s available by default with the FreeBSD 9.1-RELEASE GENERIC kernel.
# kldload linux
BOINC runs fine on FreeBSD, but it’s not the actual application that will compute the work units. It governs the programs doing the actual work and provides control over them. Because the actual program doing the work in this case is written for Linux, we need to change a sysctl variable. I found this at http://people.freebsd.org/~pav/boinc.html and the BOINC Wiki - Installing BOINC - FreeBSD:
# sysctl kern.elf32.fallback_brand=3
This allows the ELF32 application to run as it should. Otherwise the kernel wouldn’t know that to do with it.
The next item solves a problem I ran into. When trying to start the BOINC daemon it complained it couldn’t set it’s
idprio. The error message I got was:
idprio: RTP_SET: Operation not permitted
idprio of a process defines it’s priority for the CPU scheduler. BOINC is configured to run with the lowest priority possible, so you can still use your system. And does so by calling
idprio directly instead of using
nice. I believe because using
idprio is more effective, but I do not know any specifics about it. I did discover
idprio is a protected setting by default. You can allow a more liberal use of
idprio by setting the following sysctl:
# sysctl security.bsd.unprivileged_idprio=1
FreeBSD Linux Binary Compatibility
A good read is the FreeBSD Handbook - Chapter 11 Linux Binary Compatibility, in case you want to know more about it. This part has been made easy by use of a FreeBSD Port:
linux_base-f10. All you need to do is install it.
First I enter my fresh jail:
# qjail console cow.sjoosten.nl
Now I install the
linux_base-f10 port. Do note this takes some time to compile and install, so grab some coffee to drink while you are waiting.
cow > cd /usr/ports/emulators/linux_base-f10/ cow > make install distclean
Note I use
distclean, this removes all the downloaded files used to install the Linux userland. You don’t need them anymore after this, so that’s why I clean them up. If you don’t want to do that, just use
Once done it’s time to install the BOINC client.
cow > cd /usr/ports/net/boinc-client/ cow > make install clean
Now you are presented the configuration for the BOINC client. I adapted the defaults a bit because I do not need a graphical client and I certainly do not need X11 on this headless machine. If you run a desktop, feel free to keep them enabled. Additionally I do want to accept Linux science applications, so do enable that one. Using the
OPTIMIZE is optional. I first compiled it without, ran a quick benchmark, recompiled it with the option and ran the benchmark again. There was a big difference, more details later.
[x] CLIENT Build BOINC client [ ] MANAGER Build BOINC manager GUI [ ] X11 Build graphics API [x] LINUX Accept Linux science applications [x] USER Create/check BOINC client user [ ] SKINS Install more skins for BOINC manager [x] OPTIMIZE Enable compiler optimization flags
I left all follow up questions by the dependencies at their defaults.
Enable BOINC daemon
/etc/rc.conf to enable the BOINC daemon.
cow > vi /etc/rc.conf
# Enable BOINC client boinc_client_enable="YES" boinc_client_flags="--allow_remote_gui_rpc"
When you run BOINC in a jail using a non-local address, you need to add the
--allow_remote_gui_rpc flag. Otherwise the
boinccmd program cannot access the client in order to control it. In case you use a cloned local interface for your jail, you might not require this, same goes for running BOINC on the FreeBSD host directly.
Now you are ready to enable the BOINC daemon. Do so by typing:
cow > /usr/local/etc/rc.d/boinc-client start
This fired up the BOINC client under the user
boinc just fine over here.
Controlling the BOINC client
Included with the client is an application called
boinccmd, which can be used to control it using the command line. You can also use it to create a new account or look up an existing one, add projects to work on and ask for the current status of the client.
To use it, switch over to the
cow > su - boinc
Confirm you can talk to the client using
boinccmd --get_host_info. My result:
$ boinccmd --get_host_info timezone: 3600 domain name: cow.sjoosten.nl IP addr: #CPUS: 2 CPU vendor: amd64 CPU model: AMD Athlon(tm) II X2 245 ProcessorAMD Athlon(tm) II X2 245 Processor  [sse sse2 sse3 3dnow 3dnowext mmx] CPU FP OPS: 1000000000.000000 CPU int OPS: 1000000000.000000 CPU mem BW: 1000000000.000000 OS name: FreeBSD OS version: 9.1-RELEASE mem size: 8531730432.000000 cache size: 1000000.000000 swap size: 0.000000 disk size: 8589934592.000000 disk free: 8285062144.000000
See something similar for your machine? Good, that means
boinccmd can talk to the client.
Add account and project
First, you need to inform the client about your account or create a new one. In my case, I needed to create a new account. Syntax for this is
boinccmd --create_account <project URL> <email> <password> <nickname>
$ boinccmd --create_account http://www.cosmologyathome.org firstname.lastname@example.org mysecret mynickname status: Success poll status: operation in progress account key: <your_account_key_is_displayed_here>
In case you already have an account, use the following syntax:
$ boinccmd --lookup_account <project URL> <email> <password>
Next, you need to attach a project to your client, using your account key:
$ boinccmd --project_attach http://www.cosmologyathome.org <your_account_key>
Now the client will contact the project’s server and request work units to start computing. Give it a moment to download the data, and it will start to process the units. Have a look at your processor usage with
top or something similar.
There are multiple ways to check how the client is doing. I tend to use:
$ boinccmd --get_simple_gui_info
This displays some information about you and your account, as well as information about the work units currently being processed. Type
boinccmd without any parameters to have it display other command line switches. Or you can also have a look at BOINC Wiki - boinccmd.
Optional: Test performance of OPTIMIZE option
If you want to figure out if the OPTIMIZE options helps you in any way, you need to measure it’s effect. This can be done by running a built-in benchmark by executing
boinccmd --run_benchmarks. This exists quietly and seems to perform it’s run in the background. Wait a minute and try the
--get_host_info command, note the
CPU FP OPS and
CPU int OPS.
This was my result without OPTIMIZE:
CPU FP OPS: 2444736714.238674 CPU int OPS: 8291353458.765148
Afterwards I deinstalled BOINC, enabled OPTIMIZE, reinstalled and ran the benchmark again. Results:
CPU FP OPS: 3853493691.434822 CPU int OPS: 12159082025.870707
Quite the difference!
If these scores actually matter for the project you will be working on I do not know. These are synthetic benchmarks, specifically Whetstone (floating-point) and Dhrystone (integer), and do not represent real world tasks. So don’t trust these results to represent actual performance or performance difference of the OPTIMIZE switch. It might only prove that your compiler is good at optimizing the code of these benchmarks and mean nothing for an actual project work unit. Perhaps it helps to speed up the BOINC client a bit in telling the actual compute application what to do, or not. It doesn’t seem to do any harm, so I enable it.
Perhaps folks at BOINC can tell you more about it if you really want to find out.