Distcc With MSVC

Project Name

distcc, distributed compilations for C/C++ compilers.

Project Description

The goal of this project is to speed up Windows Mozilla Builds by spreading object compilations over several machines. This can be done by using an open source product called distcc created by Martin Pool. Unfortunately this is only available on *nix machines using gcc. Our project involves porting this software to Windows and use Microsoft's Visual C++ compiler using cygwin.

Project Leader(s)

• Tom Aratyn
• Cesar Oliveira

Project Contributor(s)

We have stuff for you to do!

• Dave Liebreich - Thanks for checkin whether cygwin's ln will create working links to cl.exe (e.g. a link name clfoo.exe to cl.exe).
• Ben Hearsum - for the windows box and support! Thank you!
• Jason Spiro - For helping figure out why cl wouldn't execute in the unit testing framework and assisting in finding the right path to solve this issue.
• Phil - helped with changing the page on helping people do research into cl options for us.
• Moe
• Mark P. (aka RealMarkP)
• David Humphrey
• Jeff
• Melz

Product Documentation

Current Status

• distcc seems to be distributing, but something is not working on the server side which is causing distcc to fallback to localhost. This problem has not been solved as of Dec 7, 2006.
• Some of the Test Cases may be incorrect. Neither Tom nor Cesar have extensive experience in CL options. So it may not distribute when it should, or it may distribute when it shouldn't.

How to get distcc to compile use cl

Tom and Cesar created this patch keeping backwards compatibility in mind. You should be able to still use distcc to compile your c programs if you wish. But if you applied the patch, you probably want to do more than that.

Firstly, the environment variables for distributing with cl is different. DISTCC_HOSTS is still valid for gcc, but cl will be looking for DISCC_CL_HOSTS instead. In fact, that is the only thing it looks for. Unlike gcc, cl will not look for ~/.distcc directory. This might be implemented later on.

Secondly, masquerade mode works only for gcc. If you want want to use cl, you must explicitly say so (distcc cl hello.c). If you are using gnu make, you must also explictly tell it to use cl as well (eg using CC, CXX, etc).

Maintainer-check

NOTE :: If you run the checks, please check your processes (ps aux, or the task manager) for leaked distcc/distccd programs. We will fix this ASAP

You may ignore this section if you installed Microsoft's CL in the default C:\ directory and your tests work fine.

In order to properly run the tests, you may need to make a few changes. The PATH variable is reinitalized when the test cases are run. Normally, the tests will fail when they try the cl tests. This is because PATH does not look for the cl executable and the libraries. Tom and Cesar have made certain assemptions when extending the include path, and included these following paths :

/cygdrive/c/Program Files/Microsoft Visual Studio 8/VC/bin
/cygdrive/c/Program Files/Microsoft Visual Studio 8/Common7/IDE

If it is different for your machine, you need to find these in the Makefile.in or Makefile and change them to your settings.

Extending the program

This will be postponed until we get distcc working properly (not fully) with cl.

Project Details

Mozilla (and others :) ) is interested in speeding up their builds by using a distributed compiler (distcc). The problem is that distcc only works with gcc but not with the msvc compiler (cl).

The plan has two stages:

1. change distcc to support multiple compilers
2. add cl support

Initially, we planned to use two vtables (one for distcc and another for distccd) to keep track of which compiler specific operation to use when. However that bombed out so we're currently using a single vtable.

We have kept the two vtable branch because we actually got a lot of the cl specific stuff done (before we realized we missed some functions).

Help Us Out

Here's what you can do to help us out...:

• Distcc with MSVC:CL Options
• help us test!
• Windows Apps for Testing Distcc With MSVC

The First Patch (Oops)

This was released to the distcc mailing list. This has a few functions missing. It was meant to provide the two vtable support without cl. It should be ignored and we'll be re-releasing a working version of this soon.

Project News

Jan 5, 2007

As of r89 the issue of cl getting confused by unix style paths is gone. Windows builds now have a function compiled in which converts unix style paths (/path/from/root/to/file.c) to windows style paths (c:\path/to/file).

We have chosen to not use the cygpath code since we're unlikely to need to check the mount table and do all that fancy stuff. This function is "smart" enough to realize that "/home" is under the cygwin install dir. However the build system is not clever enough to check where cygwin is installed. This should definitly be more dynamic.

Jan 4, 2007

The post made to the build newgroup seems to have gathered dust. If mozilla is limited to only 4 jobs, than distcc is somewhat limited. But that's not our problem.

Tom has found one problem with using cygwin. Using absolute pathnames. For example :

$ cl hello.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.42 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

hello.c
Microsoft (R) Incremental Linker Version 8.00.50727.42
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:hello.exe
hello.obj

Works. But this doesn't :

$ cl /cygdrive/c/home/temp/hello.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.42 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

cl : Command line warning D9002 : ignoring unknown option '/cygdrive/c/home/temp
/hello.c'
cl : Command line error D8003 : missing source filename

There are a few things we can do about this :

• replace /cygdrive/c/ with C:\
• user must replace the enviroment variable TMPDIR which defaults to /tmp

Cesar found another problem. Both C/C++ preprocessed file ends in .i. cl has no way of knowning if it's a C/C++ preprocessed file. So it refuses to compile it (it treats it as an object file). So when we preprocess, we have to redirect to the same extension as the source file.

$ cl hello.i
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.42 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

cl : Command line warning D9024 : unrecognized source file type 'hello.i', objec
t file assumed
Microsoft (R) Incremental Linker Version 8.00.50727.42
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:hello.exe
hello.i
hello.i : fatal error LNK1107: invalid or corrupt file: cannot read at 0xEE7F

Dec 17, 2006

Tom has made a post to the build newsgroup. The bug (Dec 6) still hasn't been fixed.

Also, I (Cesar) needs to do trial of building mozilla using the original distcc (gcc only). We need to present an exact log of where the problem lies. IIRC, it has something to do with gtk. But that isn't usefull enough. This information is on a ubuntu machine in the linux club at seneca college, but I think it's down because I cannot connect to it. I am not going to school on my break to turn on a friggin machine.

Dec 6, 2006

The major bug right now is that compiled object files don't come back :(. This will be fixed soon. Really. We promise.

Nov 29, 2006

We have finally finished a big part of our project - the functionality! Now we are in the process of testing. Inital tests were somewhat positive. We still need to debug certain problems in the patch. It will probably be finished after the semester is over. If anyone is still interested in the project and wants to help out, we still are interested in getting the results reproduced by average users.

Nov 22. 2006

We are down to our last 3 functions. We hope to have test cases done so we can really start testing out the whole application and try it with the two machines we do have.

Nov 6. 2006

There is a scheduled 'hack day' being set up. Were putting it at every tuesday at around 5:20 in the TEL open lab.

Nov 2. 2006

After a haitus from touching distcc, we battled the serpent of the C (haha) once again and retreated with our lives, barely, but lost our ego and motivation (at least Cesar did). We cleared up a path setting bug that caused CL to die and return a bad return value. Then we (eventually) passed ParseHost case, which parses an environment variable for ip addresses and domain names. A inproperly done svn merge wasted a good few hours on that.

We also created safeguards, so that any unimplemented vtable function that is called will throw an error message instead of having a segmentation fault.

Oct 20. 2006

We have a usable windows box for doing our msvc testing (finally!). Thanks to User:Bhearsum for the boxes.

Oct 12. 2006

Argh. I have been trying to build firefox using distcc for about 2 weeks, with many small details getting in the way. At one point things were working, but than it stopped and I don't know why. I finally, at one point, got things to build on andrew's machine. But the build failed complaining about (this is one of the many, many things) undefined reference operator delete(void*). I think it might be because distcc is calling gcc rather than c++. I changed the MOZCONFIG to use CC='distcc gcc' and CXX='distcc g++'. This threw me this ugliness :

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

Which I told reed about, but has never came across this problem. One nasty solution is to use -j without specifing a number to allow unlimited jobs. At least it's working.

In other news, Tom sent a patch.

Oct. 2 2006

A preliminary test of running distcc in cygwin revealed we were failing 4 tests; one or two more than our first patch. We tried to fix it, but never had success and didn't issue a commit. A while later we try running distcc again on another machine. This time we were only failing one test, which is less than our first patch. Here is the known tests to fail

1. EmptySource_Case --- This is expected to fail since this is a bug in the earlier gcc compiler.

The explination for the extra three preliminary tests failing is because we interupted one of the tests, which we believed caused the daemon to be orphaned and exacting it's revenge on the murder's of it's parent processes.

Sept. 26 2006

Tried to do parallel builds with mozilla in linux (maybe we should have done this first). Here is something usefull to know : To parallel build in mozilla, you cannot pass the -jN option to make and expect it to work. It didn't for me, and make was killing itself without an explination. I later learned that you have to add the following line to mozconfig

mk_add_options MOZ_MAKE_FLAGS=-j8

Sept. 24 2006

We think we've got the architecture right now. Ofcourse the only way to know is to start porting cl :). We're now going to try to port the tests to work with cl.exe.