Difference between revisions of "Add support for more compilers to distcc"

From CDOT Wiki
Jump to: navigation, search
m
m (fixed instructions)
 
(7 intermediate revisions by the same user not shown)
Line 19: Line 19:
 
<li>[[User:Vlam6|Vincent Lam]]</li>
 
<li>[[User:Vlam6|Vincent Lam]]</li>
 
<li>[[User:Vbala|Vijey Balasundaram]]</li>
 
<li>[[User:Vbala|Vijey Balasundaram]]</li>
 +
<li>[[User:Simon|Simon Jung]]</li>
 
</ul>
 
</ul>
  
Line 25: Line 26:
 
  <li>publish/find some more c# files to test with</li>
 
  <li>publish/find some more c# files to test with</li>
 
  <li>NOTE: Please ensure the files compile with Mono c# before sending them.</li>
 
  <li>NOTE: Please ensure the files compile with Mono c# before sending them.</li>
−
  <li>test this on different OSs</li>
+
  <li>test this on different OSs & networks</li>
−
  <li>Make a comprehensive list of the set-up steps on one page, unlike the scattered links there are now</li>
+
  <li>Make a comprehensive list of the set-up steps on one page, unlike the scattered links there are now. No coding required, but time consuming.</li>
−
  <li>more to be added later</li>
 
 
</ul>
 
</ul>
  
 
== Project Details/News ==
 
== Project Details/News ==
−
=== 0.2 Release ===
+
=== 0.3 release (Under construction)===
 +
==== Code ====
 +
All the code for the 0.3 release can be found [http://zenit.senecac.on.ca/wiki/imgs/Pcallaghanzeropointthree.zip here].  This contains three files:
 +
<ul>
 +
<li>pcallaghanzeropointthreeWHOLE.txt - the entire patch</li>
 +
<li>pcallaghanzeropointthreeCODE.txt - the changes to the src directory</li>
 +
<li>pcallaghanzeropointthreeTEST.txt - the changes to test/distcctest.py</li>
 +
</ul>
 +
These should be applied from the trunk directory.  NOTE:  use either pcallaghanzeropointthreeWHOLE.txt OR both the others, depending on your preference.  The end result is the same.  For more details see the instructions section.
 +
 
 +
==== Instructions ====
 +
In order to use this program, several steps must be completed.
 +
 
 +
<ol>
 +
<li>download the Mono C# [http://www.mono-project.com/Main_Page compiler]</li>
 +
  <ul>
 +
  <li>put mono in your PATH</li>
 +
  </ul>
 +
<li>set up an environment variable DISTCC_MCS_HOSTS=localhost</li>
 +
<li>Checkout the distcc source</li>
 +
  <ul>
 +
  <li>svn co svn://cdot.senecac.on.ca/distcc/trunk</li>
 +
  <li>You should get revision 117</li>
 +
  </ul>
 +
<li>If you are using windows [http://www.cygwin.com/ get cygwin]</li>
 +
  <ul>
 +
  <li>configure cygwin by running the setup file that gets downloaded.  I used the mirror: http://mirror.rhsmith.umd.edu</li>
 +
  <li>NOTE: This is the step most errors will stem from, so pay careful attention to detail.</li>
 +
  <ul>
 +
    <li>In addition to the packages downloaded by default, get the following packages:</li>
 +
    <ul>
 +
<li>binutils 20060817-1 (Devel)</li>
 +
<li>diffutils 2.8.7-1 (Utils)</li>
 +
<li>gcc 3.4.4-3 (Devel)  NOTE:  MUST be version 3.4.4-3 </li>
 +
<li>gcc-core 3.4.4-3 (Devel)</li>
 +
<li>gcc-g++ 3.4.4-3 (Devel)</li>
 +
<li>gcc-mingw 20040810-1 (Devel)</li>
 +
<li>gcc-mingw-core 20050522-1 (Devel)</li>
 +
<li>gcc-mingw-g++ 20050522-1 (Devel)</li>
 +
<li>mingw_runtime 3.13-1 (Devel, Libs)</li>
 +
<li>python 2.5.1-2 (Interpreters, Python)</li>
 +
<li>subversion 1.4.5-2 (Devel)</li>
 +
<li>openssl 0.9.8g-1 (Libs, Net)</li>
 +
<li>patch 2.5.8-9 (Utils)</li>
 +
<li>patchutils 0.2.31-1 (Devel, Libs, Text)</li>
 +
<li>tcltk 20060202-1 (Libs)</li>
 +
<li>zlib 1.2.3-2 (Libs)</li>
 +
<li>w32api 3.10-1 (Libs)</li>
 +
<li>libapr1 1.2.11-1 (Libs)</li>
 +
<li>libaprutil1 1.2.10-1 (Libs)</li>
 +
<li>libneon26 0.26.4-1 (Libs)</li>
 +
<li>libcharset1 1.11-1 (Libs)</li>
 +
    </ul>
 +
    <li>modify the cygwin.bat file</li>
 +
    <li>Copy and paste the following file, then change the paths so they accurately represent your system.  It's VERY important</li>
 +
    <li>The particular lines to change (if yours are different) are: 4,9,12,16,19,22</li>
 +
    <li>If you copy and paste you should not have to make any changes past line 22.</li>
 +
    <pre>@echo off
 +
 
 +
C:
 +
chdir C:\cygwin\bin
 +
 
 +
rem --- Set HOME so that cvs and ssh work correctly
 +
rem --- cvs uses HOME to locate your .cvspass file, and ssh to locate your .ssh file
 +
rem --- if you are using ssh, your HOME should match the home directory specified in /etc/passwd. See http://www.cygwin.com/faq/faq0.html.
 +
set HOME=C:\home
 +
 
 +
rem --- Set VCVARS to wherever the MSVC vcvars.bat file is found
 +
set VCVARS=C:\ff\vcvars32.bat
 +
 
 +
rem --- Set MSSDK to wherever the MS SDK is installed
 +
rem --- Only required for MSVC7 or the Free MSVC editions that don't come with an SDK
 +
set MSSDK=C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2
 +
 
 +
rem --- Set MOZ_TOOLS to wherever you have the moztools packaged installed
 +
set MOZ_TOOLS=C:\moztools
 +
rem --- Set MONO to wherever you have mono installed.
 +
rem --- Make sure there are no spaces in the path, otherwise the unit tests won't work
 +
set MONO=C:\Mono-1.2.5.1
 +
 
 +
rem --- Set CYGWINBASE to wherever cygwin is installed
 +
rem --- Do not use CYGWIN or else cygserver, cygrunsrv, and Cygwin services will not function properly
 +
rem --- Variable CYGWIN is also used to modify Cygwin's behaviour a little bit.
 +
set CYGWINBASE=C:\cygwin
 +
rem --- Make sure Cygwin does not print out a DOS style path warning
 +
set CYGWIN=nodosfilewarning
 +
 
 +
rem --- Prepend Cygwin path
 +
rem --- This is necessary so that cygwin find is ahead of windows find.exe in the PATH, but cgywin link is after MSVC link.exe.
 +
set PATH=%CYGWINBASE%\bin;%PATH%
 +
set PATH=%MONO%\bin;%PATH%
 +
rem --- Set MSVC environment vars
 +
call "%VCVARS%"
 +
 +
rem --- Prepend SDK paths
 +
rem --- Only required for MSVC7 or the Free MSVC editions that don't come with an SDK
 +
set PATH=%MSSDK%\bin;%PATH%;
 +
set INCLUDE=%MSSDK%\include;%MSSDK%\include\atl;%INCLUDE%
 +
set LIB=%MSSDK%\lib;%LIB%
 +
 
 +
rem --- moztools comes last
 +
set PATH=%PATH%;%MOZ_TOOLS%\bin;C:\ff\distcc\src;
 +
 
 +
 
 +
rem --- Now the PATH variable contains:
 +
rem MS-SDK; MSVC; Cygwin; Windows; glib/libIDL; Moztools
 +
 
 +
rem --- Typically the last thing the script does is launch a cygwin shell
 +
rem --- watch for your ~/.profile and /etc/profile which may overwrite your carefully setup PATH!
 +
bash --login -i
 +
</pre>
 +
  </ul>
 +
  </ul>
 +
<li>get the 0.3 [http://zenit.senecac.on.ca/wiki/imgs/Pcallaghanzeropointthree.zip patch]</li>
 +
<li>apply the patch</li>
 +
  <ul>
 +
  <li>from the trunk directory</li>
 +
  <li>patch -p0 < pcallaghanzeropointthreeWHOLE.txt</li>
 +
  <li>OR</li>
 +
  <ul>
 +
    <li>patch -p0 < pcallaghanzeropointthreeCODE.txt</li>
 +
    <li>patch -p0 < pcallaghanzeropointthreeTEST.txt</li>
 +
  </ul>
 +
  </ul>
 +
<li>configure and build distcc</li>
 +
  <ul>
 +
  <li>from the trunk directory</li>
 +
  <li>./configure && make && make install</li>
 +
  </ul>
 +
<li>get a simple C# [http://docs.google.com/Doc?id=dfkp8fkz_1hbrfkk file]</li>
 +
<li>compile it</li>
 +
  <ul>
 +
  <li>distcc mcs hello.cs</li>
 +
  </ul>
 +
<li>run it</li>
 +
  <ul>
 +
  <li>./hello</li>
 +
  </ul>
 +
  <li>As an alternative to steps 9 through 11, find or write some other c# files and test them.</li>
 +
  <li>Tell me how it went.</li>
 +
</ol>
 +
 
 +
=== Plans for 0.3 release ===
 +
While all the required functions exist, unit tests would streamline the testing process and ensure each function does exaclty what it's supposed to.  Therefore, the major component of the 0.3 release is going to be a series of tests for the functions that can be run after any changes.  In addition, (Time permitting) I'll clean up the instructions and make them clearer.
 +
 
 +
Having found that these instructions are inadequate, I'm setting out to fix them.  First up is configuring cygwin.  The Instructions given on the linked page aren't enough to compile distcc with.  There's a list of some 165 packages I had.  Most of them were obviously extraneous and quite a few were downloaded by default.  So I'm down to about 40 to test.  The required ones will be listed with the 0.3 release instructions.
 +
 
 +
=== 0.25 Release (Obsolete) ===
 +
A new [http://docs.google.com/Doc?id=dfkp8fkz_3hqpw4f patch] has been created.  All required functions are now complete for the Mono C# compiler.  Now is a great time for more contributors to test this project and try to find (or fix) any bugs that pop up.  I'm still having trouble with SVN, but I expect to get that sorted out soon.
 +
 
 +
to use this patch:
 +
<ol>
 +
<li>[http://www.cygwin.com/ get cygwin]</li>
 +
<li>[http://developer.mozilla.org/en/docs/Windows_build_prerequisites_using_cygwin configure cygwin]</li>
 +
<li>download the Mono C# [http://www.mono-project.com/Main_Page compiler]</li>
 +
  <ul>
 +
  <li>put mono in your PATH</li>
 +
  </ul>
 +
<li>set up an environment variable DISTCC_MCS_HOSTS=localhost</li>
 +
<li>Checkout the distcc source</li>
 +
  <ul>
 +
  <li>svn co svn://cdot.senecac.on.ca/distcc/trunk</li>
 +
  <li>You should get revision 117</li>
 +
  <li>cd trunk && ./configure && make && make install</li>
 +
  <li>put distcc in your PATH, before cygwin</lI>
 +
  </ul>
 +
<li>get the 0.25 [http://docs.google.com/Doc?id=dfkp8fkz_3hqpw4f patch]</li>
 +
<li>apply the patch</li>
 +
  <ul>
 +
  <li>from the trunk directory</li>
 +
  <li>patch -p0 < pcallaghanzeropointtwofive.txt</li>
 +
  </ul>
 +
<li>configure and build distcc</li>
 +
  <ul>
 +
  <li>from the trunk directory</li>
 +
  <li>./configure && make && make install</li>
 +
  </ul>
 +
<li>get a simple C# [http://docs.google.com/Doc?id=dfkp8fkz_1hbrfkk file]</li>
 +
<li>compile it</li>
 +
  <ul>
 +
  <li>distcc mcs hello.cs</li>
 +
  </ul>
 +
<li>run it</li>
 +
  <ul>
 +
  <li>./hello</li>
 +
  </ul>
 +
  <li>As an alternative to steps 9 through 11, find or write some other c# files and test them.</li>
 +
  <li>Tell me how it went.</li>
 +
</ol>
 +
 
 +
 
 +
 
 +
=== 0.2 Release (Obsolete)===
 
A new [http://docs.google.com/Doc?id=dfkp8fkz_2gxdzhr patch] has been created.  All except one of the required functions (dcc_mcs_scan_args) are now complete.  As the work required for this update turned out to be a lot less than I anticipated, a 0.25 patch should be released soon that will implement that last function.  An SVN branch is not available yet, but should also be ready in time for 0.25.
 
A new [http://docs.google.com/Doc?id=dfkp8fkz_2gxdzhr patch] has been created.  All except one of the required functions (dcc_mcs_scan_args) are now complete.  As the work required for this update turned out to be a lot less than I anticipated, a 0.25 patch should be released soon that will implement that last function.  An SVN branch is not available yet, but should also be ready in time for 0.25.
  
Line 44: Line 236:
 
Continue work on the previously identified functions.  As of 0.1 several were still little more than shells, by the completion of 0.2 all except dcc_mcs_scan_args should be fully completed.  A supplementary goal is to set up the source as a branch of the existing trunk on cdot, so testers may just check out the new code rather than having to check out the trunk and patch it.
 
Continue work on the previously identified functions.  As of 0.1 several were still little more than shells, by the completion of 0.2 all except dcc_mcs_scan_args should be fully completed.  A supplementary goal is to set up the source as a branch of the existing trunk on cdot, so testers may just check out the new code rather than having to check out the trunk and patch it.
  
−
=== 0.1 Release ===
+
=== 0.1 Release (Obsolete)===
 
A [http://docs.google.com/Doc?id=dfkp8fkz_0hdwb6s compiler] has been created.
 
A [http://docs.google.com/Doc?id=dfkp8fkz_0hdwb6s compiler] has been created.
 
Note that it is by no means anywhere near completion.
 
Note that it is by no means anywhere near completion.

Latest revision as of 04:26, 14 December 2007

Project Name

Add support for more compilers to distcc

Project Description

An earlier project added multi-compiler support to distcc, and then MSVC support. The framework is now in place to support even more compilers. Add support for another compiler; either Java or C#.

Necessary Skills: C, Python, Makefile

Project Leader(s)

Peter Callaghan (pcal)

Project Contributor(s)

Some options for contributions:

  • publish/find some more c# files to test with
  • NOTE: Please ensure the files compile with Mono c# before sending them.
  • test this on different OSs & networks
  • Make a comprehensive list of the set-up steps on one page, unlike the scattered links there are now. No coding required, but time consuming.

Project Details/News

0.3 release (Under construction)

Code

All the code for the 0.3 release can be found here. This contains three files:

  • pcallaghanzeropointthreeWHOLE.txt - the entire patch
  • pcallaghanzeropointthreeCODE.txt - the changes to the src directory
  • pcallaghanzeropointthreeTEST.txt - the changes to test/distcctest.py

These should be applied from the trunk directory. NOTE: use either pcallaghanzeropointthreeWHOLE.txt OR both the others, depending on your preference. The end result is the same. For more details see the instructions section.

Instructions

In order to use this program, several steps must be completed.

  1. download the Mono C# compiler
    • put mono in your PATH
  2. set up an environment variable DISTCC_MCS_HOSTS=localhost
  3. Checkout the distcc source
  4. If you are using windows get cygwin
    • configure cygwin by running the setup file that gets downloaded. I used the mirror: http://mirror.rhsmith.umd.edu
    • NOTE: This is the step most errors will stem from, so pay careful attention to detail.
      • In addition to the packages downloaded by default, get the following packages:
        • binutils 20060817-1 (Devel)
        • diffutils 2.8.7-1 (Utils)
        • gcc 3.4.4-3 (Devel) NOTE: MUST be version 3.4.4-3
        • gcc-core 3.4.4-3 (Devel)
        • gcc-g++ 3.4.4-3 (Devel)
        • gcc-mingw 20040810-1 (Devel)
        • gcc-mingw-core 20050522-1 (Devel)
        • gcc-mingw-g++ 20050522-1 (Devel)
        • mingw_runtime 3.13-1 (Devel, Libs)
        • python 2.5.1-2 (Interpreters, Python)
        • subversion 1.4.5-2 (Devel)
        • openssl 0.9.8g-1 (Libs, Net)
        • patch 2.5.8-9 (Utils)
        • patchutils 0.2.31-1 (Devel, Libs, Text)
        • tcltk 20060202-1 (Libs)
        • zlib 1.2.3-2 (Libs)
        • w32api 3.10-1 (Libs)
        • libapr1 1.2.11-1 (Libs)
        • libaprutil1 1.2.10-1 (Libs)
        • libneon26 0.26.4-1 (Libs)
        • libcharset1 1.11-1 (Libs)
      • modify the cygwin.bat file
      • Copy and paste the following file, then change the paths so they accurately represent your system. It's VERY important
      • The particular lines to change (if yours are different) are: 4,9,12,16,19,22
      • If you copy and paste you should not have to make any changes past line 22.
        • @echo off

C:
chdir C:\cygwin\bin

rem --- Set HOME so that cvs and ssh work correctly
rem --- cvs uses HOME to locate your .cvspass file, and ssh to locate your .ssh file
rem --- if you are using ssh, your HOME should match the home directory specified in /etc/passwd. See http://www.cygwin.com/faq/faq0.html.
set HOME=C:\home

rem --- Set VCVARS to wherever the MSVC vcvars.bat file is found
set VCVARS=C:\ff\vcvars32.bat

rem --- Set MSSDK to wherever the MS SDK is installed
rem --- Only required for MSVC7 or the Free MSVC editions that don't come with an SDK
set MSSDK=C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2

rem --- Set MOZ_TOOLS to wherever you have the moztools packaged installed
set MOZ_TOOLS=C:\moztools
rem --- Set MONO to wherever you have mono installed.
rem --- Make sure there are no spaces in the path, otherwise the unit tests won't work
set MONO=C:\Mono-1.2.5.1

rem --- Set CYGWINBASE to wherever cygwin is installed
rem --- Do not use CYGWIN or else cygserver, cygrunsrv, and Cygwin services will not function properly
rem --- Variable CYGWIN is also used to modify Cygwin's behaviour a little bit.
set CYGWINBASE=C:\cygwin
rem --- Make sure Cygwin does not print out a DOS style path warning
set CYGWIN=nodosfilewarning

rem --- Prepend Cygwin path
rem --- This is necessary so that cygwin find is ahead of windows find.exe in the PATH, but cgywin link is after MSVC link.exe.
set PATH=%CYGWINBASE%\bin;%PATH%
set PATH=%MONO%\bin;%PATH%
rem --- Set MSVC environment vars
call "%VCVARS%"
 
rem --- Prepend SDK paths
rem --- Only required for MSVC7 or the Free MSVC editions that don't come with an SDK
set PATH=%MSSDK%\bin;%PATH%;
set INCLUDE=%MSSDK%\include;%MSSDK%\include\atl;%INCLUDE%
set LIB=%MSSDK%\lib;%LIB%

rem --- moztools comes last
set PATH=%PATH%;%MOZ_TOOLS%\bin;C:\ff\distcc\src;


rem --- Now the PATH variable contains:
rem MS-SDK; MSVC; Cygwin; Windows; glib/libIDL; Moztools

rem --- Typically the last thing the script does is launch a cygwin shell
rem --- watch for your ~/.profile and /etc/profile which may overwrite your carefully setup PATH!
bash --login -i
  5. get the 0.3 patch
  6. apply the patch
    • from the trunk directory
    • patch -p0 < pcallaghanzeropointthreeWHOLE.txt
    • OR
      • patch -p0 < pcallaghanzeropointthreeCODE.txt
      • patch -p0 < pcallaghanzeropointthreeTEST.txt
  7. configure and build distcc
    • from the trunk directory
    • ./configure && make && make install
  8. get a simple C# file
  9. compile it
    • distcc mcs hello.cs
  10. run it
    • ./hello
  11. As an alternative to steps 9 through 11, find or write some other c# files and test them.
  12. Tell me how it went.

Plans for 0.3 release

While all the required functions exist, unit tests would streamline the testing process and ensure each function does exaclty what it's supposed to. Therefore, the major component of the 0.3 release is going to be a series of tests for the functions that can be run after any changes. In addition, (Time permitting) I'll clean up the instructions and make them clearer.

Having found that these instructions are inadequate, I'm setting out to fix them. First up is configuring cygwin. The Instructions given on the linked page aren't enough to compile distcc with. There's a list of some 165 packages I had. Most of them were obviously extraneous and quite a few were downloaded by default. So I'm down to about 40 to test. The required ones will be listed with the 0.3 release instructions.

0.25 Release (Obsolete)

A new patch has been created. All required functions are now complete for the Mono C# compiler. Now is a great time for more contributors to test this project and try to find (or fix) any bugs that pop up. I'm still having trouble with SVN, but I expect to get that sorted out soon.

to use this patch:

  1. get cygwin
  2. configure cygwin
  3. download the Mono C# compiler
    • put mono in your PATH
  4. set up an environment variable DISTCC_MCS_HOSTS=localhost
  5. Checkout the distcc source
  6. get the 0.25 patch
  7. apply the patch
    • from the trunk directory
    • patch -p0 < pcallaghanzeropointtwofive.txt
  8. configure and build distcc
    • from the trunk directory
    • ./configure && make && make install
  9. get a simple C# file
  10. compile it
    • distcc mcs hello.cs
  11. run it
    • ./hello
  12. As an alternative to steps 9 through 11, find or write some other c# files and test them.
  13. Tell me how it went.


0.2 Release (Obsolete)

A new patch has been created. All except one of the required functions (dcc_mcs_scan_args) are now complete. As the work required for this update turned out to be a lot less than I anticipated, a 0.25 patch should be released soon that will implement that last function. An SVN branch is not available yet, but should also be ready in time for 0.25.

to use this patch:

  • Either reverse the first patch or get a non-patched version of the trunk
  • Apply the new patch


Goal for 0.2 release

Continue work on the previously identified functions. As of 0.1 several were still little more than shells, by the completion of 0.2 all except dcc_mcs_scan_args should be fully completed. A supplementary goal is to set up the source as a branch of the existing trunk on cdot, so testers may just check out the new code rather than having to check out the trunk and patch it.

0.1 Release (Obsolete)

A compiler has been created. Note that it is by no means anywhere near completion. That's why this is the 0.1 release.

This patch has been tested on Windows XP, using cygwin to tell distcc to compile a single, simple c# file. To use it:

  1. get cygwin
  2. configure cygwin
  3. download the Mono C# compiler
    • put mono in your PATH
  4. set up an environment variable DISTCC_MCS_HOSTS=localhost
  5. Checkout the distcc source
  6. get the 0.1 patch
  7. apply the patch
    • from the trunk directory
    • patch -p0 < pcallaghanzeropointone.txt
  8. configure and build distcc
    • from the trunk directory
    • ./configure && make && make install
  9. get a simple C# file
  10. compile it
    • distcc mcs hello.cs
  11. run it
    • ./hello

Goal for 0.1 Release

- Add support for Mono's cSharp compiler[1]. By the 0.1 release, Distcc should be able to at least compile C# locally. While the earlier work done on Distcc With MSVC suggests that only a hand-full of functions need to be coded in order for it to add a new compiler, both Java and C# compile in a manner significantly different than C. This will likely cause some trouble shortly...regardless, the following functions are the minimum which need to be created:

  • arg.c
    • dcc_<compilerName>_set_action_opt
    • dcc_<compilerName>_set_output
    • dcc_<compilerName>_scan_args
  • strip.c
    • dcc_<compilerName>_strip_local_args
    • dcc_<compilerName>_strip_dasho
  • filename.c
    • dcc_<compilerName>_is_source
    • dcc_<compilerName>_is_preprocessed
    • dcc_<compilerName>_is_object
    • dcc_<compilerName>_preproc_exten
  • hosts.c
    • dcc_<compilerName>_get_hostlist

Known Bugs

See the list within Distcc With MSVC

Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Add_support_for_more_compilers_to_distcc&oldid=14026"