BOINC (Cosmology@Home) on a FreeBSD jail

About 9 or 10 years ago I was an active participant in “distributed-computing”. Projects with an enormous workload that call upon the world to help them compute their data and/or research. This is done by people running clients on their (home) computers, which download “work units”, calculate a result and send back the data to the project’s servers. Often this is combined with a competitive element: you receive points for a completed work unit. Statistics are generated on a regular basis and you can look up how well you did compared to others around the globe. People can also form a team together and compete against other teams.

The Dutch Power Cows is such a team, and a large one at that. Intended for Dutch speaking people, of course others are welcome as well, to gather under a single banner and compute for all kinds of projects.

Every now and then the Dutch Power Cows organize an event they call a “stampede”. They nominate a month in which they try to attract as many people as possible to work on a chosen project. Boosting their output significantly for a month for the project, surprising other teams, and often taking first place in the daily statistics during that month. And by doing so they increase attention for a project, make the statistics interesting again and possibly end up with some new members.

Recently they announced the start of a new stampede, starting April 1st. The project of interest this time is the Cosmology@Home project. As an ex-DPC member the news interested me and I decided to help out during this stampede. So I had to install a client on my machines.

Installing clients on Windows machines is easily done, and many instructions can be found. Most projects also offer support for Linux, and some even do for FreeBSD and Mac OS X. In the case of Cosmology@Home, only Windows and Linux support is offered. And instructions for FreeBSD are pretty much non-existent. So I was disappointed at first, because I want to run this on my FreeBSD box as well, preferably in a jail too! But than I thought about this: FreeBSD has the “Linuxulator” for running Linux binaries. Perhaps I can use that to run Cosmology@Home on my FreeBSD box that’s idling away most of the time.

Requirements

What do I need to run this:

  • optional: fresh jail to install everything in.
  • Linuxulator (in the jail): Available as a port linux_base-f10
  • BOINC client: a FreeBSD port is available.
  • Enough RAM, at least 512MB per CPU core and have enough left to run the system.

Set up a new jail

Using a jail for this is optional. If you want to run the client on the host OS, that’s fine too, you can skip this step.

I use qjail to manage my jails. If you use something else, like ezjail, or do it by hand that will be fine too.

I create a filesystem for my jail. You will need at least 2GB of space to install a Linux environment and the client in. I like to use quotas to limit disk usage, but I have plenty of space, so I set my limit at 8GB to have some headroom. I disable compression for this jail because I don’t expect to gain much by it and I can use the CPU cycles I save on the project.

# zfs create -o quota=8G compression=off zroot/freebsd9/usr/jails/cow.sjoosten.nl

Let’s create a fresh jail, refresh my ports tree for them and start the jail.

# qjail create -n re0 cow.sjoosten.nl 192.168.0.103
# qjail update -p
# qjail start cow.sjoosten.nl

Prepare the host

Before we dive into setting up the BOINC client on the jail, we need to set some option on the host first.

First, let’s load the linux kernel module. It’s available by default with the FreeBSD 9.1-RELEASE GENERIC kernel.

# kldload linux

BOINC runs fine on FreeBSD, but it’s not the actual application that will compute the work units. It governs the programs doing the actual work and provides control over them. Because the actual program doing the work in this case is written for Linux, we need to change a sysctl variable. I found this at http://people.freebsd.org/~pav/boinc.html and the BOINC Wiki - Installing BOINC - FreeBSD:

# sysctl kern.elf32.fallback_brand=3

This allows the ELF32 application to run as it should. Otherwise the kernel wouldn’t know that to do with it.

The next item solves a problem I ran into. When trying to start the BOINC daemon it complained it couldn’t set it’s idprio. The error message I got was: idprio: RTP_SET: Operation not permitted

The idprio of a process defines it’s priority for the CPU scheduler. BOINC is configured to run with the lowest priority possible, so you can still use your system. And does so by calling idprio directly instead of using nice. I believe because using idprio is more effective, but I do not know any specifics about it. I did discover idprio is a protected setting by default. You can allow a more liberal use of idprio by setting the following sysctl:

# sysctl security.bsd.unprivileged_idprio=1

FreeBSD Linux Binary Compatibility

A good read is the FreeBSD Handbook - Chapter 11 Linux Binary Compatibility, in case you want to know more about it. This part has been made easy by use of a FreeBSD Port: linux_base-f10. All you need to do is install it.

First I enter my fresh jail:

# qjail console cow.sjoosten.nl

Now I install the linux_base-f10 port. Do note this takes some time to compile and install, so grab some coffee to drink while you are waiting.

cow > cd /usr/ports/emulators/linux_base-f10/
cow > make install distclean

Note I use distclean, this removes all the downloaded files used to install the Linux userland. You don’t need them anymore after this, so that’s why I clean them up. If you don’t want to do that, just use clean instead.

BOINC client

Installation

Once done it’s time to install the BOINC client.

cow > cd /usr/ports/net/boinc-client/
cow > make install clean

Now you are presented the configuration for the BOINC client. I adapted the defaults a bit because I do not need a graphical client and I certainly do not need X11 on this headless machine. If you run a desktop, feel free to keep them enabled. Additionally I do want to accept Linux science applications, so do enable that one. Using the OPTIMIZE is optional. I first compiled it without, ran a quick benchmark, recompiled it with the option and ran the benchmark again. There was a big difference, more details later.

My configuration:

[x] CLIENT    Build BOINC client                    
[ ] MANAGER   Build BOINC manager GUI               
[ ] X11       Build graphics API                    
[x] LINUX     Accept Linux science applications     
[x] USER      Create/check BOINC client user        
[ ] SKINS     Install more skins for BOINC manager  
[x] OPTIMIZE  Enable compiler optimization flags

I left all follow up questions by the dependencies at their defaults.

Enable BOINC daemon

Edit /etc/rc.conf to enable the BOINC daemon.

cow > vi /etc/rc.conf

And add:

# Enable BOINC client
boinc_client_enable="YES"
boinc_client_flags="--allow_remote_gui_rpc"

When you run BOINC in a jail using a non-local address, you need to add the --allow_remote_gui_rpc flag. Otherwise the boinccmd program cannot access the client in order to control it. In case you use a cloned local interface for your jail, you might not require this, same goes for running BOINC on the FreeBSD host directly.

Starting BOINC

Now you are ready to enable the BOINC daemon. Do so by typing:

cow > /usr/local/etc/rc.d/boinc-client start

This fired up the BOINC client under the user boinc just fine over here.

Controlling the BOINC client

Included with the client is an application called boinccmd, which can be used to control it using the command line. You can also use it to create a new account or look up an existing one, add projects to work on and ask for the current status of the client.

To use it, switch over to the boinc user:

cow > su - boinc

Confirm you can talk to the client using boinccmd --get_host_info. My result:

$ boinccmd --get_host_info 
timezone: 3600
domain name: cow.sjoosten.nl
IP addr: 
#CPUS: 2
CPU vendor: amd64
CPU model: AMD Athlon(tm) II X2 245 ProcessorAMD Athlon(tm) II X2 245 Processor [] [sse sse2 sse3 3dnow 3dnowext mmx]
CPU FP OPS: 1000000000.000000
CPU int OPS: 1000000000.000000
CPU mem BW: 1000000000.000000
OS name: FreeBSD
OS version: 9.1-RELEASE
mem size: 8531730432.000000
cache size: 1000000.000000
swap size: 0.000000
disk size: 8589934592.000000
disk free: 8285062144.000000

See something similar for your machine? Good, that means boinccmd can talk to the client.

Add account and project

First, you need to inform the client about your account or create a new one. In my case, I needed to create a new account. Syntax for this is boinccmd --create_account <project URL> <email> <password> <nickname>

$ boinccmd --create_account http://www.cosmologyathome.org my.email@provider.net mysecret mynickname
status: Success
poll status: operation in progress
account key: <your_account_key_is_displayed_here>

In case you already have an account, use the following syntax:

$ boinccmd --lookup_account <project URL> <email> <password>

Next, you need to attach a project to your client, using your account key:

$ boinccmd --project_attach http://www.cosmologyathome.org <your_account_key>

Now the client will contact the project’s server and request work units to start computing. Give it a moment to download the data, and it will start to process the units. Have a look at your processor usage with top or something similar.

Monitor progress

There are multiple ways to check how the client is doing. I tend to use:

$ boinccmd --get_simple_gui_info

This displays some information about you and your account, as well as information about the work units currently being processed. Type boinccmd without any parameters to have it display other command line switches. Or you can also have a look at BOINC Wiki - boinccmd.

Happy computing!

Optional: Test performance of OPTIMIZE option

If you want to figure out if the OPTIMIZE options helps you in any way, you need to measure it’s effect. This can be done by running a built-in benchmark by executing boinccmd --run_benchmarks. This exists quietly and seems to perform it’s run in the background. Wait a minute and try the --get_host_info command, note the CPU FP OPS and CPU int OPS.

This was my result without OPTIMIZE:

CPU FP OPS: 2444736714.238674
CPU int OPS: 8291353458.765148

Afterwards I deinstalled BOINC, enabled OPTIMIZE, reinstalled and ran the benchmark again. Results:

CPU FP OPS: 3853493691.434822
CPU int OPS: 12159082025.870707

Quite the difference!

If these scores actually matter for the project you will be working on I do not know. These are synthetic benchmarks, specifically Whetstone (floating-point) and Dhrystone (integer), and do not represent real world tasks. So don’t trust these results to represent actual performance or performance difference of the OPTIMIZE switch. It might only prove that your compiler is good at optimizing the code of these benchmarks and mean nothing for an actual project work unit. Perhaps it helps to speed up the BOINC client a bit in telling the actual compute application what to do, or not. It doesn’t seem to do any harm, so I enable it.

Perhaps folks at BOINC can tell you more about it if you really want to find out.

blogroll

social