Mozilla Source and Symbol Server

Overview

Mozilla has a symbol server for Windows builds that allows developers or users to debug nightly and release builds without having to ship debug symbols along with the builds. This can be very useful when a user has a problem that developers can't reproduce, or when you hit a crash or hang during your normal browsing, not using a debug build.

While this is very useful, it doesn't provide source-level debugging. You can see function names and call stacks, but not the source at the location, which makes stepping through code very difficult. Luckily, Microsoft does provide a means to add source info to debug symbols, called a source server. The mission of this project is to sort through the Microsoft documentation and figure out how to add this to the existing Mozilla symbol server.

Project Name

Mozilla Source and Symbol Server

Project Description

This project is coming together in two parts:

Part One (DPS909) Create a source code server to work with Mozilla's current symbol server. This will allow users to do full debugs of nightly release builds with access to remote symbols and full source code.

Part Two (DPS911) Create a pdb writing executable that can source index pdb files as per the pdbstr.exe which is currently used by the source code server so that this tool can be distributed in the Mozilla trunk.

Project Leader(s)

Lukas Blakk (lsblakk)

Project Details

Related Bugs

• Bug 385792: Compress PDB files
• Bug 395310: Mozilla needs a source and symbol server
• Uploading PDB files (fixed)

0.1 Release

   * set up a localhost server
   * make buildsymbols from local build
   * load those symbols onto the local server
   * connect them up to my debugger to make sure it all works
   * get the microsoft scripts to work indexing source code to my local pdb files

Updates

October 18, 2007 - Success on 0.1 goals. I ran indexing tools on the pdb files and successfully indexed them. You can read all about it on my blog and here are the pdb files that have been indexed as well as a sample of what running srctool.exe on a pbd to show what files are listed within.

* The PDBs with source code indexing (warning: 67.2MB file)

October 12, 2007 - So far I have set up a server and configured it to serve up PDB files. This involved running a little script to allow the MIME type for pdb files which is application/octet-stream.

I ran make buildsymbols on my local build of minefield and successfully generated a whole ton of symbol files. These were copied to my server and I have successfully loaded them into Visual Studio when debugging.

Right now the biggest barrier is a lack of information about the Microsoft Source Server - which comes with support for several version control systems but whose CVS version is apparently known to be potentially unstable. I have made one attempt at indexing the symbol files, it was unsuccessful.

0.2 Release

Here are the goals for my 0.2 release as of October 25th, 2007

   * release build of ff with debugger-info enabled to create pdbs without being a full debug build
   * using lib xul - make an installer for people to download
   * take the symbols and index them and distribute the source
   * set up the symbol server but indexed so that someone can pull the source code
   * which involves running the cv2http.cmd on the indexed code

How to set up the Source and Symbol Server (rough draft)

• add a path to my symbolServer http://avnerd.tv/symbols/2007110200 as per MDC docs in Tools->Debugging->Symbols

• check the option for turning on the source server in Tools->Debugging->General

• Download the installer from here Firefox Installer

• install the firefox you just downloaded (it will install as Minefield in Program Files\Minefield)

• make sure you run it with --no-remote and -profilemanager so you don't mess up your normal Firefox experience

• once it's running, go to Tools->attach to process... and attach VStudio to the firefox instance

• after it's attached, you can go to Debug->break all and then poke around at the firefox functions code (You will get a warning about downloading code from an unknown source, click on "run" because you trust me)

This is what it might look like:

Contribution Opportunities for 0.2

• Please follow the instructions above and email me to let me know a) if it works for you and b) if the directions are clear

0.3 Release

The goal for the 0.3 release of the Source Server was to hack symbolstore.py so that it would gather up a list of the source files in each pdb that had cvs information and then feed them to pdbstr.exe which is a binary that comes in the Debugging Tools for Windows Package. That binary will now live in mozilla/toolkit/crashreporter/tools/win32/ along with dump_syms.

Here is my patch for symbolstore.py and in order to get a working version of this 0.3 Source Server happening on your computer please follow these steps:

• As per 0.2 make sure your debugger is set up to enable source server and that you have directed it to save symbols in a directory where you will now copy my symbols to: Symbols - sorry, I can't serve them up right now so it's best if you just download the .zip and place them in the folder manually

• This is the installer for build id 2007120422 which goes along with the above symbols - it will install in the Program Files/Minefield directory with no options given for choosing another install location

• You will need a pointer to cvs.exe in your path - either point to mozilla-build/mysys/bin/cvs.exe or download it here and put it where you like

Once you have these items set up - you may fire up your debugger and Minefield - attach to process and then when you break the debugger will ask you for permission to run cvs commands.

The source files will be checked out into a mozilla/... path - mine are in Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla... and once you point your debugger to the first file, it finds the others for you automagically.

Thanks to all the folks on the #seneca channel and especially thanks to Dave and Ted for constantly checking out the progress especially at the end when Ted found the right magic to activate pdbstr.exe properly from inside of symbolstore.py.

You can leave feedback and comments on my blog

Updates

Filed bug 408134 and attached patch for this release. Please follow work in the bug.

DPS911 - Towards a 1.0 Release - Winter 2008

0.4 Release, January 18th, 2008

This release is primarily research. After talking with Ted (ted), I have some documentation about pdb file formats and pdb "exploding" tools to play with. Also I am currently trying to make my laptop more friendly to the kind of development this project requires. In the last term I started using a MacBook mid-semester and was able to work with a Windows XP VM for the testing of my patch. This was a cumbersome task because I did not have (nor could I afford) the RAM upgrade which would make VM usage much less laggy. Yesterday I finally picked up the 2 X 2GB chips that have made my VM usage much speedier.

The bug I filed (bug 408134) will be my guideline and the goal of this project is to pass super review and have my patch (and whatever tools are needed to make it work) checked into mozilla trunk. In accordance with the specs of this class, here is an outline for how I will accomplish a 1.0 release of my project:

• All the syntax and minor logic issues will be cleared up first (by the next release), this means that I will save a copy of the filename variable right before the call to GetVCSInfo, the large block of text for the pdbstream will be rewritten as a triple-quoted string, and I will also figure out what portions of the pdbstream are unnecessary and remove them to streamline the source indexing
• The source indexing will be pulled out into a method called SourceIndex and will be located in Dumper, called from symbolstore.py
• Changes need to be made so that the source indexing is called from the command line , since it cannot (and should not) be called upon in anything other than Win32 platforms
• All hard coding will be cleared out
• A replacement program for pdbstr.exe (which is provided by Microsoft) will be written so that it can be distributable

The resources I am researching on this release were provided to me by Ted and are:

• http://www.jorgon.freeserve.co.uk/Other/pdb.htm
• http://www.rawol.com/?topic=41
• http://www.rawol.com/download/zip/win_pdbx.zip

0.5 Release, January 31, 2008

Bugfixing bug 408134 - made changes to symbolstore.py as per comments in the bug review.

Done:

• Re-wrote the pdbStreamFile.write as a triple-quoted string

 pdbStreamFile.write('''SRCSRV: ini ------------------------------------------------\r\nVERSION=1\r\n
SRCSRV: variables ------------------------------------------\r\n
CVS_EXTRACT_CMD=%fnchdir%(%CVS_WORKINGDIR%)cvs.exe -d %fnvar%(%var2%) checkout -r %var4% %var3%\r\n
CVS_EXTRACT_TARGET=%targ%\%var2%\%fnbksl%(%var3%)\%fnfile%(%var1%)\r\nCVS_WORKING_DIR=%targ%\%var2%\%fnbksl%(%var3%)\r\n
MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot\r\n
SRCSRVTRG=%CVS_WORKING_DIR%\r\nSRCSRVCMD=%CVS_EXTRACT_CMD%\r\n
SRCSRV: source files ---------------------------------------\r\n''')

pdbStreamFile.write(fileStream) 
# can't do string interpolation because the source server also uses this and so there are % in the above
pdbStreamFile.write("SRCSRV: end ------------------------------------------------\r\n\n")

• Removed unecessary information, and tested it to be sure it still worked

(which it does from the indexing end of things)

• Pulled out the creating of the stream file to a separate method called

SourceIndex

Upcoming and awaiting feedback :

• I did not move the gathering of filenames because as I see it, if I do this

before the calling of GetVCSFilename then I have a list of all files and not just the ones that have cvs info.

• I need clarification about the calling from the command line - is this

something that would involve changes in the make buildsymbols command? Any additional info about where to look for this, perhaps an example of something that does use calling different variables at the command line level?

• Obviously I cannot change the hardcoding of pdbstr.exe yet, once this patch

is cleaned up I can start working on the pdbstr.exe replacement (I have however discussed this with Ted as to how this replacement would be implemented and that would be in C++, 0.6 should see the beginnings of this code)

• The feedback on the bug points out that hardcoding the cvs checkout username and password is not the best way, but I have asked why this is an issue when everyone uses pserver: anonynmous

0.6 Release - Feb 15, 2008

GOAL: To get the symbolstore.py patch committed so that I can then start to debug/improve from the user end with WinDBg and VStudio

• Feb 6, 2008: conversation with Ted on IRC about questions raised in bug. Discussed the usage of flag in the Makefile to denote source indexing and then changes to symbolstory.py to ensure that source indexing only happens when the flag is on (ie: Winnt platform) time: 1 hour
• Feb 7, 2008: Made changes to the files as per discussion and submitted new patch for feedback time: 2 hours
• Feb 10, 2008: Spent time testing Firefox Beta 3, worked on 2 bugs (bug 416425, bug 416225) time: 3 hours
• Feb 10, 2008: Spoke with Ted and asked for feedback that will get this to the point of being committable, he said by Wednesday he would have feedback time: 15 mins
• Feb 11, 2008: Wrote blog post about Places, bookmarks and the gap in user experiences of Firefox. time: 45 minutes
• Feb 12, 2008: Exploring hex code of PDB files, trying an strace (with Wine) on pdbstr, generally poking at them. time: 1.5 hours
• Feb 14, 2008: Downloaded and installed StraceNT, prompted Ted for feedback, attempted to reach Timeless, managed to get a StraceNT output from running pdbstr -r, got feedback on the bug from Ted and have the road map to a r+. time: 1.5 hrs
• Feb 17, 2008: Better late than never, I have incorporated the comments from Ted and created a new patch which is awaiting review and (fingers crossed) approval for being committed. time: 5 hours

0.7 Release - Feb 29, 2008

GOAL: Continuing on the goal of 0.6 which is to get patch checked in

• After a couple more back and forth conversations about the finer points of the patch, more changes were made and it was r+ and then approved/checked in on February 26, 2008 time: 8 hours
• Following this, testing is needed to make sure that it now works on the user end
• To be able to test, the tinderboxen and/or the ref platform needs to have the Windbg installed so that the binary can be accessed through an environment variable which will be set using tinder:config
• Filed bug 419904 asking for this installation to be done
• Other Mozilla involvement for this period included being trained to be an AMO editor, as well I have started working on an extension that will display the current profile you are using in the statusbar - something that I would really like to have now that I am running an incredible amount of Fx instances on both Mac and WinXP at the same time doing a million things at once.

Now that this goal is met, the goal for my 0.8 release is twofold: 1. To have successful testing information on the nightly debug builds and (fingers crossed) to have the patch working completely 2. To learn anything significant about the pdbstr.exe replacement and how to begin writing it. This could be anything from an actual conversation with Timeless to more reading up in the Windows 2000 secrets book that Ted sent me a pdf link to.

0.8 Release - March 21, 2008

While I wait for the ref platform install...

Turns out that the "Debugging Tools for Windows" was already on the ref platform. That's great news! So I've filed a bug #424240 to add a line to tinder-config.pl adding

$ENV{PDBSTR_PATH} = 'C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\pdbstr.exe';

If the patch is committable, I hope that there will be a testing version in the near future.

I've created a page on MDC where I will begin to document the steps for using the Source Server builds. Using the Mozilla Source Server

This release is a focus on testing and debugging the source server from the user end.

What's needed:

• Windbg needs to be installed on the ref platform - bhearsum is helping me look into getting this done. I will monitor the progress of the bug 419904 which makes the request for install. Bug was marked "RESOLVED" since the tools were already installed.
• Nightly builds need to be flagged for indexing so I can download them and test that the source server works as it should Patch submitted, now waiting for commit.

I anticipate that there will be tweaks needed for the cvs and the file repository on the user's computer to work as desired.

Once the testing is complete and the source indexing/source server works properly I can do a user documentation for it on the MDC wiki with screenshots and a full demonstration of how to use these builds.

0.9 Release - April 4, 2008

As expected, there are some tweaks needed in order to make the source server work properly. Ted filed a bug (bug #424817) outlining what he did to make it work by tweaking the srcsrv.ini file on his local version. Last night (March 24, 2008) I spent several hours trying to fix these issues in symbolstore.py and in the end I was able to submit this patch.

Goals for this release:

• That source server works out of the box and that nightlies can be downloaded and tested
• To have a "mother of all demos" demo on Thursday April 10th where people will test out the source server - I will look into finding bugs where source server functionality would be really useful

Update April 5th, 2008 - the tweaks were committed and then they had to be backed out yesterday because the buildsymbols was broken on Mac and Linux platforms. Small setback. I'll be looking into what caused this over the weekend and hopefully get a fix in for early next week. Still intent on doing the Demo on the 10th - it might double as 'testing' :)

1.0 Release - April 18, 2008

Goals for this release:

• MDC wiki will have comprehensive documentation (with screenshots!) of how to use the source server MDC Documentation

April 9, 2008 - The patch is committed and today's nightly debug build was tested and found to work with minor adjustments on the local computer.

The adjustments are in the documentation as well.

The stumbling block right now is that the build machine that creates the nightly debug builds uses a cvs path that needs a key:

:ext:ffxbld@cvs.mozilla.org:/cvsroot

What the average user needs in order to make source server work is an open cvs access path, we commonly use :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot

In order to work around this (for now) the user must have a srcsrv.ini file and simply putting:

[variables]
MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot

in that file allows for the overriding of the "bad" cvs path.

April 11, 2008 -- It now works on both WinDBG and Visual Studio 2005 (am currently installing VS2008 Express to test) with the tweaks as per bug 428518 and bug 428615

See my blog post for more information, and watch the Documentation too.

April 15, 2008 - Source Server completely functional and documented.

• Testing on VS Express (both 2005 and 2008) showed that srcsrv is not enabled in those versions.
• Minor tweak needed for Visual Studio 2005 where the srcsrv.dll from Debugging Tools for Windows needs to be copied to replace the srcsrv.dll (which is older) that ships with VS. The newer srcsrv.dll is able to handle %fnchdir% properly and appears to work just fine with VS.
• Important note - the box "Require that source code match original source code exactly" in the Debugging options of Visual Studio must be unchecked otherwise VS says it can't find any code.
• Documentation for WinDBG was written by Ted, Timelyx contacted me two days ago and said he would test the srcsrv sometime later this week. Hopefully feedback will indicate that this functionality is of use to some developers.
• Ted blogged about it
• I will post my presentation material here by Friday

April 18, 2008

• Demo video
• Tested on Visual Studio 9.0 2008 - works as expected

Minor loose ends: Cross indexing on MDC so the doc can be found easier, SVN bug 426648 is still outstanding

Additional Resources

• MSDN articles on source server:
  • http://msdn2.microsoft.com/en-us/library/ms680641.aspx
  • http://msdn.microsoft.com/msdnmag/issues/06/08/UseTheSource/default.aspx
• Mozilla's symbol server: http://benjamin.smedbergs.us/blog/2007-06-11/debugging-official-builds-or-how-cool-is-the-mozilla-symbol-server/
• bonsai-test and bonsai-dev on landfill.mozilla.org support a "raw=1" parameter to get back a plain text version of a file at a specific revision.
  • cvsblame.cgi info
  • Example Usage
• WinDbg comes with a perl script for doing some of this work, probably at C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\cv2http.pl
• The symbolstore.py script in the Mozilla tree that builds the Breakpad symbol store already has support for grabbing CVS revision numbers