1
edit
Changes
→1.0 Release - April 18, 2008
== Project Description ==
This project is coming together in two parts:
'''Part One (DPS909)'''
Create a source code server to work with Mozilla's current symbol server. This will allow users to do full debugs of nightly release builds with access to remote symbols and full source code.
'''Part Two (DPS911)'''
Create a pdb writing executable that can source index pdb files as per the pdbstr.exe which is currently used by the source code server so that this tool can be distributed in the Mozilla trunk.
== Project Leader(s) ==
== Project Contributor(s) ==
== Project Details ==
* using lib xul - make an installer for people to download
* take the symbols and index them and distribute the source
* set up the symbol server but indexed so that someone can pull the source code * which involves running the cv2http.cmd on the indexed code === How to set up the Source and Symbol Server (rough draft) === * add a path to my symbolServer http://avnerd.tv/symbols/2007110200 as per [http://developer.mozilla.org/en/docs/Using_the_Mozilla_symbol_serverfirefox MDC docs] in Tools->Debugging->Symbols [[Image:Lukas022.png|alt Adding the Symbol Path to VStudio]] * check the option for turning on the source server in Tools->Debugging->General [[Image:Lukas021.png|alt Turning on the Source Server]] * Download the installer from here [http://avnerd.tv/sharedFiles/firefox-3.0a9pre.en-US.win32.installer.exe.zip Firefox Installer] * install the firefox you just downloaded (it will install as Minefield in Program Files\Minefield) * make sure you run it with --no-remote and -profilemanager so you don't mess up your normal Firefox experience * once it's running, go to Tools->attach to process... and attach VStudio to the firefox instance * after it's attached, you can go to Debug->break all and then poke around at the firefox functions code (You will get a warning about downloading code from an unknown source, click on "run" because you trust me) This is what it might look like: [[Image:Lukas023.png|alt This is what success looks like]] ===Contribution Opportunities for 0.2===* Please follow the instructions above and email me to let me know a) if it works for you and b) if the directions are clear == 0.3 Release == The goal for the 0.3 release of the Source Server was to hack [http://mxr.mozilla.org/mozilla/source/toolkit/crashreporter/tools/symbolstore.py symbolstore.py] so that it would gather up a list of the source files in each pdb that had cvs information and then feed them to pdbstr.exe which is a binary that comes in the Debugging Tools for Windows Package. That binary will now live in mozilla/toolkit/crashreporter/tools/win32/ along with dump_syms. Here is [http://avnerd.tv/sharedFiles/symbolstorepatch.txt my patch] for symbolstore.py and in order to get a working version of this 0.3 Source Server happening on your computer please follow these steps: * As per 0.2 make sure your debugger is set up to enable source server and that you have directed it to save symbols in a directory where you will now copy my symbols to: [http://avnerd.tv/symbols/crashreporter-symbols-firefox-3.0b2pre-WINNT- 2007120422.zip Symbols] - sorry, I can't serve them up right now so it's best if you just download the .zip and place them in the folder manually * This is the [http://avnerd.tv/sharedFiles/ff_SourceServer.exe.zip installer] for build id 2007120422 which goes along with the above symbols - it will install in the Program Files/Minefield directory with no options given for choosing another install location * You will need a pointer to cvs.exe in your path - either point to mozilla-build/mysys/bin/cvs.exe or [http://other.jrsoftware.org/ic/cvs-1.11.2.zip download it here] and put it where you like Once you have these items set up - you may fire up your debugger and Minefield - attach to process and then when you break the debugger will ask you for [http://crashopensource.files.wordpress.com/2007/12/picture-1.png permission to run cvs commands]. The source files will be checked out into a mozilla/... path - mine are in Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla... and once you point your debugger to the first file, it finds the others for you automagically. Thanks to all the folks on the #seneca channel and especially thanks to Dave and Ted for constantly checking out the progress especially at the end when Ted found the right magic to activate pdbstr.exe properly from inside of symbolstore.py. You can leave feedback and comments on my [http://crashopensource.wordpress.com blog] ===Updates=== Filed [https://bugzilla.mozilla.org/show_bug.cgi?id=408134 bug 408134] and [https://bugzilla.mozilla.org/attachment.cgi?id=292890 attached patch] for this release. Please follow work in the bug. = DPS911 - Towards a 1.0 Release - Winter 2008 === 0.4 Release, January 18th, 2008 ==This release is primarily research. After talking with Ted (ted), I have some documentation about pdb file formats and pdb "exploding" tools to play with. Also I am currently trying to make my laptop more friendly to the kind of development this project requires. In the last term I started using a MacBook mid-semester and was able to work with a Windows XP VM for the testing of my patch. This was a cumbersome task because I did not have (nor could I afford) the RAM upgrade which would make VM usage much less laggy. Yesterday I finally picked up the 2 X 2GB chips that have made my VM usage much speedier. The bug I filed ([https://bugzilla.mozilla.org/show_bug.cgi?id=408134 bug 408134]) will be my guideline and the goal of this project is to pass super review and have my patch (and whatever tools are needed to make it work) checked into mozilla trunk. In accordance with the specs of this class, here is an outline for how I will accomplish a 1.0 release of my project: * All the syntax and minor logic issues will be cleared up first (by the next release), this means that I will save a copy of the filename variable right before the call to GetVCSInfo, the large block of text for the pdbstream will be rewritten as a triple-quoted string, and I will also figure out what portions of the pdbstream are unnecessary and remove them to streamline the source indexing* The source indexing will be pulled out into a method called SourceIndex and will be located in Dumper, called from symbolstore.py* Changes need to be made so that the source indexing is called from the command line , since it cannot (and should not) be called upon in anything other than Win32 platforms* All hard coding will be cleared out* A replacement program for pdbstr.exe (which is provided by Microsoft) will be written so that it can be distributable The resources I am researching on this release were provided to me by Ted and are:* http://www.jorgon.freeserve.co.uk/Other/pdb.htm * http://www.rawol.com/?topic=41 * http://www.rawol.com/download/zip/win_pdbx.zip == 0.5 Release, January 31, 2008 == '''Bugfixing''' [https://bugzilla.mozilla.org/show_bug.cgi?id=408134 bug 408134] - made changes to symbolstore.py as per comments in the bug review. === Done: === * Re-wrote the pdbStreamFile.write as a triple-quoted string <pre> pdbStreamFile.write('''SRCSRV: ini ------------------------------------------------\r\nVERSION=1\r\nSRCSRV: variables ------------------------------------------\r\nCVS_EXTRACT_CMD=%fnchdir%(%CVS_WORKINGDIR%)cvs.exe -d %fnvar%(%var2%) checkout -r %var4% %var3%\r\nCVS_EXTRACT_TARGET=%targ%\%var2%\%fnbksl%(%var3%)\%fnfile%(%var1%)\r\nCVS_WORKING_DIR=%targ%\%var2%\%fnbksl%(%var3%)\r\nMYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot\r\nSRCSRVTRG=%CVS_WORKING_DIR%\r\nSRCSRVCMD=%CVS_EXTRACT_CMD%\r\nSRCSRV: source files ---------------------------------------\r\n''') pdbStreamFile.write(fileStream) # can't do string interpolation because the source server also uses this and so there are % in the abovepdbStreamFile.write("SRCSRV: end ------------------------------------------------\r\n\n")</pre>* Removed unecessary information, and tested it to be sure it still worked(which it does from the indexing end of things)* Pulled out the creating of the stream file to a separate method calledSourceIndex === Upcoming and awaiting feedback : === * I did not move the gathering of filenames because as I see it, if I do thisbefore the calling of GetVCSFilename then I have a list of all files and notjust the ones that have cvs info. * I need clarification about the calling from the command line - is thissomething that would involve changes in the make buildsymbols command? Anyadditional info about where to look for this, perhaps an example of somethingthat does use calling different variables at the command line level? * Obviously I cannot change the hardcoding of pdbstr.exe yet whether , once this patchis cleaned up I can start working on the pdbstr.exe replacement (I have however discussed this with Ted as to how this replacement would be implemented and that would be in C++, 0.6 should see the beginnings of this code)* The feedback on the bug points out that hardcoding the cvs checkout username and password is not the best way, but I have asked why this is an issue when everyone uses pserver: anonynmous == 0.6 Release - Feb 15, 2008 == '''GOAL''': To get the symbolstore.py patch committed so that I can then start to debug/improve from the user end with WinDBg and VStudio * Feb 6, 2008: conversation with Ted on IRC about questions raised in bug. Discussed the usage of flag in the Makefile to denote source indexing and then changes to symbolstory.py to ensure that source indexing only happens when the flag is on (ie: Winnt platform) time: 1 hour* Feb 7, 2008: Made changes to the files as per discussion and submitted new patch for feedback time: 2 hours* Feb 10, 2008: Spent time testing Firefox Beta 3, worked on 2 bugs ([https://bugzilla.mozilla.org/show_bug.cgi?id=416425 bug 416425], [https://bugzilla.mozilla.org/show_bug.cgi?id=416225 bug 416225]) time: 3 hours* Feb 10, 2008: Spoke with Ted and asked for feedback that will get this to the point of being committable, he said by Wednesday he would have feedback time: 15 mins* Feb 11, 2008: Wrote blog post about Places, bookmarks and the gap in user experiences of Firefox. time: 45 minutes* Feb 12, 2008: Exploring hex code of PDB files, trying an strace (with Wine) on pdbstr, generally poking at them. time: 1.5 hours* Feb 14, 2008: Downloaded and installed StraceNT, prompted Ted for feedback, attempted to reach Timeless, managed to get a StraceNT output from running pdbstr -r, got feedback on the bug from Ted and have the road map to a r+. time: 1.5 hrs* Feb 17, 2008: Better late than never, I have incorporated the comments from Ted and created a new patch which is awaiting review and (fingers crossed) approval for being committed. time: 5 hours == 0.7 Release - Feb 29, 2008 == '''GOAL''': Continuing on the goal of 0.6 which is to get patch checked in * After a couple more back and forth conversations about the finer points of the patch, more changes were made and '''it was r+ and then approved/checked in on February 26, 2008''' time: 8 hours* Following this, testing is needed to make sure that it now works on the user end* To be able to test, the tinderboxen and/or the ref platform needs to have the Windbg installed so that the binary can be accessed through an environment variable which will be set using tinder:config* Filed [https://bugzilla.mozilla.org/show_bug.cgi?id=419904 bug 419904] asking for this installation to be done * Other Mozilla involvement for this period included being trained to be an AMO editor, as well I have started working on an extension that will display the current profile you are using in the statusbar - something that I would really like to have now that I am running an incredible amount of Fx instances on both Mac and WinXP at the same time doing a million things at once. Now that this goal is met, the goal for my 0.8 release is twofold:1. To have successful testing information on the nightly debug builds and (fingers crossed) to have the patch working completely2. To learn anything significant about the pdbstr.exe replacement and how to begin writing it. This could be anything from an actual conversation with Timeless to more reading up in the Windows 2000 secrets book that Ted sent me a pdf link to. == 0.8 Release - March 21, 2008 == <s>'''While I wait for the ref platform install...'''</s> Turns out that the "Debugging Tools for Windows" was already on the ref platform. That's great news! So I've filed a bug [https://bugzilla.mozilla.org/show_bug.cgi?id=424240 #424240] to add a line to [http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/tools/tinderbox-configs/firefox/win32/tinder-config.pl tinder-config.pl] adding <code>$ENV{PDBSTR_PATH} = 'C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\pdbstr.exe';</code> If the patch is committable, I hope that there will be a testing version in the near future. I've created a page on MDC where I will begin to document the steps for using the Source Server builds. [http://developer.mozilla.org/en/docs/Using_the_Mozilla_source_server Using the Mozilla Source Server] This release is a focus on testing and debugging the source server from the user end. What's needed: *<s> Windbg needs to be installed on the ref platform - bhearsum is helping me look into getting this done. I will monitor the progress of the [https://bugzilla.mozilla.org/show_bug.cgi?id=419904 bug 419904] which makes the request for install.</s> Bug was marked "RESOLVED" since the tools were already installed.* <s>Nightly builds need to be flagged for indexing so I can download them and test that the source server works as it should</s> Patch submitted, now waiting for commit. I anticipate that there will be tweaks needed for the cvs and the file repository on the user's computer to work as desired. Once the testing is complete and the source indexing/source server works properly I can do a user documentation for it on the MDC wiki with screenshots and a full demonstration of how to use these builds. == 0.9 Release - April 4, 2008 == As expected, there are some tweaks needed in order to make the source server work properly. Ted filed a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=424817 bug #424817]) outlining what he did to make it work by tweaking the srcsrv.ini file on his local version. Last night (March 24, 2008) I spent several hours trying to fix these issues in symbolstore.py and in the end I was able to submit this [https://bugzilla.mozilla.org/attachment.cgi?id=311503&action=diff patch]. Goals for this release:* That source server works out of the box and that nightlies can be downloaded and tested* To have a "mother of all demos" demo on Thursday April 10th where people will test out the source server - I will look into finding bugs where source server functionality would be really useful '''Update April 5th, 2008''' - the tweaks were committed and then they had to be backed out yesterday because the buildsymbols was broken on Mac and Linux platforms. Small setback. I'll be looking into what caused this over the weekend and hopefully get a fix in for early next week. Still intent on doing the Demo on the 10th - it might double as 'testing' :) == 1.0 Release - April 18, 2008 == Goals for this release:* <s>MDC wiki will have comprehensive documentation (with screenshots!) of how to use the source server</s> [http://developer.mozilla.org/en/docs/Using_the_Mozilla_source_server MDC Documentation] '''April 9, 2008''' - The patch is committed and today's nightly debug build was tested and found to work with minor adjustments on the local computer. The adjustments are in the documentation as well. The stumbling block right now is that the build machine that creates the nightly debug builds uses a cvs path that needs a key: <pre>:ext:ffxbld@cvs.mozilla.org:/cvsroot</pre> What the average user needs in order to make source server work is an open cvs access path, we commonly use :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot In order to work around this (for now) the user must have a srcsrv.ini file and simply putting:<pre>[variables]MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot</pre> in that file allows for the overriding of the "bad" cvs path. '''April 11, 2008''' -- It now works on both WinDBG and Visual Studio 2005 (am currently installing VS2008 Express to test) with the tweaks as per [https://bugzilla.mozilla.org/show_bug.cgi?id=428518 bug 428518] and [https://bugzilla.mozilla.org/show_bug.cgi?id=428615 bug 428615] See my [http://crashopensource.blogspot.com/2008/04/source-server-will-be-ready-for-public.html blog post] for more information, and watch the [http://developer.mozilla.org/en/docs/Using_the_Mozilla_source_server Documentation] too. '''April 15, 2008''' - Source Server completely functional and documented. * Testing on VS Express (both 2005 and 2008) showed that srcsrv is not enabled in those versions.* Minor tweak needed for Visual Studio 2005 where the srcsrv.dll from Debugging Tools for Windows needs to be copied to replace the srcsrv.dll (which is older) that ships with VS. The newer srcsrv.dll is able to handle %fnchdir% properly and appears to work just fine with VS.* Important note - the box "Require that source code match original source code exactly" in the Debugging options of Visual Studio must be unchecked otherwise VS says it can't find any code.* [http://developer.mozilla.org/en/docs/Using_the_Mozilla_source_server Documentation] for WinDBG was written by Ted, Timelyx contacted me two days ago and said he would test the srcsrv sometime later this week. Hopefully feedback will indicate that this functionality is of use to some developers.* Ted [http://blog.mozilla.com/ted/2008/04/15/debugging-nightly-builds-with-the-source-server/ blogged] about it* I will post my presentation material here by Friday '''April 18, 2008'''* [http://crashopensource.blogspot.com/2008/04/final-demo-of-source-server-for-mozilla.html Demo video]* Tested on Visual Studio 9.0 2008 - works as expected Minor loose ends: Cross indexing on MDC so the doc can be found easier, [https://bugzilla.mozilla.org/show_bug.cgi?id=426648 SVN bug 426648] is still outstanding
= Additional Resources =
