Open main menu

CDOT Wiki β

Mozilla Source and Symbol Server

Revision as of 13:25, 12 February 2008 by Backinblakk (talk | contribs) (→‎0.6 Release - Feb 15, 2008)

Overview

Mozilla has a symbol server for Windows builds that allows developers or users to debug nightly and release builds without having to ship debug symbols along with the builds. This can be very useful when a user has a problem that developers can't reproduce, or when you hit a crash or hang during your normal browsing, not using a debug build.

While this is very useful, it doesn't provide source-level debugging. You can see function names and call stacks, but not the source at the location, which makes stepping through code very difficult. Luckily, Microsoft does provide a means to add source info to debug symbols, called a source server. The mission of this project is to sort through the Microsoft documentation and figure out how to add this to the existing Mozilla symbol server.

Project Name

Mozilla Source and Symbol Server

Project Description

This project is coming together in two parts:

Part One (DPS909) Create a source code server to work with Mozilla's current symbol server. This will allow users to do full debugs of nightly release builds with access to remote symbols and full source code.

Part Two (DPS911) Create a pdb writing executable that can source index pdb files as per the pdbstr.exe which is currently used by the source code server so that this tool can be distributed in the Mozilla trunk.

Project Leader(s)

Lukas Blakk (lsblakk)

Project Contributor(s)

Project Details

Related Bugs

0.1 Release

   * set up a localhost server
   * make buildsymbols from local build
   * load those symbols onto the local server
   * connect them up to my debugger to make sure it all works
   * get the microsoft scripts to work indexing source code to my local pdb files

Updates

October 18, 2007 - Success on 0.1 goals. I ran indexing tools on the pdb files and successfully indexed them. You can read all about it on my blog and here are the pdb files that have been indexed as well as a sample of what running srctool.exe on a pbd to show what files are listed within. 

* The PDBs with source code indexing (warning: 67.2MB file)


October 12, 2007 - So far I have set up a server and configured it to serve up PDB files. This involved running a little script to allow the MIME type for pdb files which is application/octet-stream.

I ran make buildsymbols on my local build of minefield and successfully generated a whole ton of symbol files. These were copied to my server and I have successfully loaded them into Visual Studio when debugging.

Right now the biggest barrier is a lack of information about the Microsoft Source Server - which comes with support for several version control systems but whose CVS version is apparently known to be potentially unstable. I have made one attempt at indexing the symbol files, it was unsuccessful.

0.2 Release

Here are the goals for my 0.2 release as of October 25th, 2007 

   * release build of ff with debugger-info enabled to create pdbs without being a full debug build
   * using lib xul - make an installer for people to download
   * take the symbols and index them and distribute the source
   * set up the symbol server but indexed so that someone can pull the source code
   * which involves running the cv2http.cmd on the indexed code


How to set up the Source and Symbol Server (rough draft)

 

  • check the option for turning on the source server in Tools->Debugging->General

 

  • install the firefox you just downloaded (it will install as Minefield in Program Files\Minefield)
  • make sure you run it with --no-remote and -profilemanager so you don't mess up your normal Firefox experience
  • once it's running, go to Tools->attach to process... and attach VStudio to the firefox instance
  • after it's attached, you can go to Debug->break all and then poke around at the firefox functions code (You will get a warning about downloading code from an unknown source, click on "run" because you trust me)

This is what it might look like:

 


Contribution Opportunities for 0.2

  • Please follow the instructions above and email me to let me know a) if it works for you and b) if the directions are clear

0.3 Release

The goal for the 0.3 release of the Source Server was to hack symbolstore.py so that it would gather up a list of the source files in each pdb that had cvs information and then feed them to pdbstr.exe which is a binary that comes in the Debugging Tools for Windows Package. That binary will now live in mozilla/toolkit/crashreporter/tools/win32/ along with dump_syms.

Here is my patch for symbolstore.py and in order to get a working version of this 0.3 Source Server happening on your computer please follow these steps:

  • As per 0.2 make sure your debugger is set up to enable source server and that you have directed it to save symbols in a directory where you will now copy my symbols to: Symbols - sorry, I can't serve them up right now so it's best if you just download the .zip and place them in the folder manually
  • This is the installer for build id 2007120422 which goes along with the above symbols - it will install in the Program Files/Minefield directory with no options given for choosing another install location
  • You will need a pointer to cvs.exe in your path - either point to mozilla-build/mysys/bin/cvs.exe or download it here and put it where you like

Once you have these items set up - you may fire up your debugger and Minefield - attach to process and then when you break the debugger will ask you for permission to run cvs commands.

The source files will be checked out into a mozilla/... path - mine are in Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla... and once you point your debugger to the first file, it finds the others for you automagically.

Thanks to all the folks on the #seneca channel and especially thanks to Dave and Ted for constantly checking out the progress especially at the end when Ted found the right magic to activate pdbstr.exe properly from inside of symbolstore.py.

You can leave feedback and comments on my blog

Updates

Filed bug 408134 and attached patch for this release. Please follow work in the bug.

DPS911 - Towards a 1.0 Release - Winter 2008

0.4 Release, January 18th, 2008

This release is primarily research. After talking with Ted (ted), I have some documentation about pdb file formats and pdb "exploding" tools to play with. Also I am currently trying to make my laptop more friendly to the kind of development this project requires. In the last term I started using a MacBook mid-semester and was able to work with a Windows XP VM for the testing of my patch. This was a cumbersome task because I did not have (nor could I afford) the RAM upgrade which would make VM usage much less laggy. Yesterday I finally picked up the 2 X 2GB chips that have made my VM usage much speedier.


The bug I filed (bug 408134) will be my guideline and the goal of this project is to pass super review and have my patch (and whatever tools are needed to make it work) checked into mozilla trunk. In accordance with the specs of this class, here is an outline for how I will accomplish a 1.0 release of my project:

  • All the syntax and minor logic issues will be cleared up first (by the next release), this means that I will save a copy of the filename variable right before the call to GetVCSInfo, the large block of text for the pdbstream will be rewritten as a triple-quoted string, and I will also figure out what portions of the pdbstream are unnecessary and remove them to streamline the source indexing
  • The source indexing will be pulled out into a method called SourceIndex and will be located in Dumper, called from symbolstore.py
  • Changes need to be made so that the source indexing is called from the command line , since it cannot (and should not) be called upon in anything other than Win32 platforms
  • All hard coding will be cleared out
  • A replacement program for pdbstr.exe (which is provided by Microsoft) will be written so that it can be distributable

The resources I am researching on this release were provided to me by Ted and are:

0.5 Release, January 31, 2008

Bugfixing bug 408134 - made changes to symbolstore.py as per comments in the bug review.

Done:

  • Re-wrote the pdbStreamFile.write as a triple-quoted string 
 pdbStreamFile.write('''SRCSRV: ini ------------------------------------------------\r\nVERSION=1\r\n
SRCSRV: variables ------------------------------------------\r\n
CVS_EXTRACT_CMD=%fnchdir%(%CVS_WORKINGDIR%)cvs.exe -d %fnvar%(%var2%) checkout -r %var4% %var3%\r\n
CVS_EXTRACT_TARGET=%targ%\%var2%\%fnbksl%(%var3%)\%fnfile%(%var1%)\r\nCVS_WORKING_DIR=%targ%\%var2%\%fnbksl%(%var3%)\r\n
MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot\r\n
SRCSRVTRG=%CVS_WORKING_DIR%\r\nSRCSRVCMD=%CVS_EXTRACT_CMD%\r\n
SRCSRV: source files ---------------------------------------\r\n''')

pdbStreamFile.write(fileStream) 
# can't do string interpolation because the source server also uses this and so there are % in the above
pdbStreamFile.write("SRCSRV: end ------------------------------------------------\r\n\n")
  • Removed unecessary information, and tested it to be sure it still worked

(which it does from the indexing end of things)

  • Pulled out the creating of the stream file to a separate method called

SourceIndex

Upcoming and awaiting feedback :

  • I did not move the gathering of filenames because as I see it, if I do this

before the calling of GetVCSFilename then I have a list of all files and not just the ones that have cvs info.

  • I need clarification about the calling from the command line - is this

something that would involve changes in the make buildsymbols command? Any additional info about where to look for this, perhaps an example of something that does use calling different variables at the command line level?

  • Obviously I cannot change the hardcoding of pdbstr.exe yet, once this patch

is cleaned up I can start working on the pdbstr.exe replacement (I have however discussed this with Ted as to how this replacement would be implemented and that would be in C++, 0.6 should see the beginnings of this code)

  • The feedback on the bug points out that hardcoding the cvs checkout username and password is not the best way, but I have asked why this is an issue when everyone uses pserver: anonynmous

0.6 Release - Feb 15, 2008

GOAL: To get the symbolstore.py patch committed so that I can then start to debug/improve from the user end with WinDBg and VStudio


  • Feb 6, 2008: conversation with Ted on IRC about questions raised in bug. Discussed the usage of flag in the Makefile to denote source indexing and then changes to symbolstory.py to ensure that source indexing only happens when the flag is on (ie: Winnt platform) time: 1 hour
  • Feb 7, 2008: Made changes to the files as per discussion and submitted new patch for feedback time: 2 hours
  • Feb 10, 2008: Spent time testing Firefox Beta 3, worked on 2 bugs (bug 416425, [https://bugzilla.mozilla.org/show_bug.cgi?id=416225 bug 416225) time: 3 hours
  • Feb 10, 2008: Spoke with Ted and asked for feedback that will get this to the point of being committable, he said by Wednesday he would have feedback time: 15 mins
  • Feb 11, 2008: Wrote blog post about Places, bookmarks and the gap in user experiences of Firefox. time: 45 minutes

Additional Resources

Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Mozilla_Source_and_Symbol_Server&oldid=14851"