Mozilla Source and Symbol Server

Overview

Mozilla has a symbol server for Windows builds that allows developers or users to debug nightly and release builds without having to ship debug symbols along with the builds. This can be very useful when a user has a problem that developers can't reproduce, or when you hit a crash or hang during your normal browsing, not using a debug build.

While this is very useful, it doesn't provide source-level debugging. You can see function names and call stacks, but not the source at the location, which makes stepping through code very difficult. Luckily, Microsoft does provide a means to add source info to debug symbols, called a source server. The mission of this project is to sort through the Microsoft documentation and figure out how to add this to the existing Mozilla symbol server.

Project Name

Mozilla Source and Symbol Server

Project Description

Create a source code server to work with Mozilla's current symbol server. This will allow users to do full debugs of nightly release builds with access to remote symbols and full source code.

Project Leader(s)

Lukas Blakk (lsblakk)

Project Contributor(s)

For right now I am still working on source indexing. If anyone knows how to do this - or is interested in the continued research I am doing, please get in touch with me and I'll get you up to speed on what I know so far. Most of this info can be found on my blog

• Timothy Joseph Duavis (here to help out in testing, researching or anything else that you need assistance)

Project Details

Related Bugs

• Bug 385792: Compress PDB files
• Bug 395310: Mozilla needs a source and symbol server
• Uploading PDB files (fixed)

0.1 Release

   * set up a localhost server
   * make buildsymbols from local build
   * load those symbols onto the local server
   * connect them up to my debugger to make sure it all works
   * get the microsoft scripts to work indexing source code to my local pdb files

Updates

October 18, 2007 - Success on 0.1 goals. I ran indexing tools on the pdb files and successfully indexed them. You can read all about it on my blog and here are the pdb files that have been indexed as well as a sample of what running srctool.exe on a pbd to show what files are listed within.

* The PDBs with source code indexing (warning: 67.2MB file)

October 12, 2007 - So far I have set up a server and configured it to serve up PDB files. This involved running a little script to allow the MIME type for pdb files which is application/octet-stream.

I ran make buildsymbols on my local build of minefield and successfully generated a whole ton of symbol files. These were copied to my server and I have successfully loaded them into Visual Studio when debugging.

Right now the biggest barrier is a lack of information about the Microsoft Source Server - which comes with support for several version control systems but whose CVS version is apparently known to be potentially unstable. I have made one attempt at indexing the symbol files, it was unsuccessful.

0.2 Release

Here are the goals for my 0.2 release as of October 25th, 2007

   * release build of ff with debugger-info enabled to create pdbs without being a full debug build
   * using lib xul - make an installer for people to download
   * take the symbols and index them and distribute the source
   * set up the symbol server but indexed so that someone can pull the source code
   * which involves running the cv2http.cmd on the indexed code

How to set up the Source and Symbol Server (rough draft)

• add a path to my symbolServer http://avnerd.tv/symbols/2007110200 as per MDC docs in Tools->Debugging->Symbols

• check the option for turning on the source server in Tools->Debugging->General

• Download the installer from here Firefox Installer

• install the firefox you just downloaded (it will install as Minefield in Program Files\Minefield)

• make sure you run it with --no-remote and -profilemanager so you don't mess up your normal Firefox experience

• once it's running, go to Tools->attach to process... and attach VStudio to the firefox instance

• after it's attached, you can go to Debug->break all and then poke around at the firefox functions code (You will get a warning about downloading code from an unknown source, click on "run" because you trust me)

This is what it might look like:

Contribution Opportunities for 0.2

• Please follow the instructions above and email me to let me know a) if it works for you and b) if the directions are clear

0.3 Release

The goal for the 0.3 release of the Source Server was to hack symbolstore.py so that it would gather up a list of the source files in each pdb that had cvs information and then feed them to pdbstr.exe which is a binary that comes in the Debugging Tools for Windows Package. That binary will now live in mozilla/toolkit/crashreporter/tools/win32/ along with dump_syms.

Here is my patch for symbolstore.py and in order to get a working version of this 0.3 Source Server happening on your computer please follow these steps:

• As per 0.2 make sure your debugger is set up to enable source server and that you have directed it to save symbols in a directory where you will now copy my symbols to: Symbols - sorry, I can't serve them up right now so it's best if you just download the .zip and place them in the folder manually

• This is the installer for build id 2007120422 which goes along with the above symbols - it will install in the Program Files/Minefield directory with no options given for choosing another install location

• You will need a pointer to cvs.exe in your path - either point to mozilla-build/mysys/bin/cvs.exe or download it here and put it where you like

Once you have these items set up - you may fire up your debugger and Minefield - attach to process and then when you break the debugger will ask you for permission to run cvs commands.

The source files will be checked out into a mozilla/... path - mine are in Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla... and once you point your debugger to the first file, it finds the others for you automagically.

Thanks to all the folks on the #seneca channel and especially thanks to Dave and Ted for constantly checking out the progress especially at the end when Ted found the right magic to activate pdbstr.exe properly from inside of symbolstore.py.

You can leave feedback and comments on my blog

Additional Resources

• MSDN articles on source server:
  • http://msdn2.microsoft.com/en-us/library/ms680641.aspx
  • http://msdn.microsoft.com/msdnmag/issues/06/08/UseTheSource/default.aspx
• Mozilla's symbol server: http://benjamin.smedbergs.us/blog/2007-06-11/debugging-official-builds-or-how-cool-is-the-mozilla-symbol-server/
• bonsai-test and bonsai-dev on landfill.mozilla.org support a "raw=1" parameter to get back a plain text version of a file at a specific revision.
  • cvsblame.cgi info
  • Example Usage
• WinDbg comes with a perl script for doing some of this work, probably at C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\cv2http.pl
• The symbolstore.py script in the Mozilla tree that builds the Breakpad symbol store already has support for grabbing CVS revision numbers