Mercurial history browsing

Project Name

Adding Functionality to the Browser Based UI for Mercurial History Browsing, hgweb

Project Description

Mozilla is just starting to use an exciting new distributed version control tool: Mercurial.

One of the exciting things about Mercurial is that the history—the list of all the changes that have ever been checked in—is not linear. It frequently has branches and merges. This is actually a good, important feature, as you know if you've read a little about distributed version control. The downside is that the history becomes a maze of twisty little passages. Benjamin Smedberg's demo shows the history of a repository with lots of merges (each box is a check-in; you can click on the boxes to move around).

There have been a couple attempts at showing history in an intuitive, graphical way. Mercurial comes with a web UI for browsing the repository, including history (here's what it looks like). It also comes with an "hg glog" extension that draws history as ASCII art, and an "hg view" extension that does roughly the same thing with a little GUI. Sadly, none of these applications qualifies as awesome.

The project's goal is to rectify this sad situation by writing an awesome browser-based UI for navigating Mercurial repository history.

Resources: jorendorff, bsmedberg

Project Leader(s)

Siddharth Kalra

Project Contributor(s)

• Tiago Moreira

Project Mentors

• Jason Ordendorff (jorendorff)
• Dirkjan Ochtman (djc)
• Ted Mielczarek (ted)
• Benjamin Smedberg (bsmedberg)

Project Ideas

September 23rd, 2008

Hgweb ideas

Chat with jorendorff We have to make sure you understand hg we have to decide on some very small thing that you can get running very soon we should talk it over and make sure it's clear how to do it we should plan to meet up later and brainstorm some, once you've had a chance to play around with hg Ideas from the blog post by jorendorff jorendorff's ideas History scroll forever, up and down. Something like a tumblelog Able to identify whether what one is looking at is in the mainline or a branch Major lines of development color coded i.e. blue for mozilla-central, orange for tracemonkey etc... This is possible using information from those repos’ pushlogs Pushlog info could be used to make those special paths relatively straight, with other smaller branches and mini-merges happening to the side or hidden by default with some sort of collapse/expand widget Show more info about each changeset, but only in an unobtrusive way Be able to know which dirs were touched and roughly the size of the diffs Be able to filter the history by file or dir A vertical timeline Location of a changeset on screen would tell something about when it was developed or pushed Be able to zoom in and out and see weeks, months, years of work Showing dates of when changes were "pushed" to mozilla-central, not the dates they were "committed" This makes the history linear and to some degree shows if a merge is big or small Jesse Ruderman Ideas Consider starting with the pushloghtml page, which the Moz community seems to use, rather than the shortlog page Gc Ideas Search history restricted to a particular data range and dirs because when a bug is found to start on a particular build from a month ago, need to find what was changed since the previous build, in relevant dirs RyanVM Ideas Be able to view more than 10 entries at a time in the changelog Jonas Sicking Ideas Uses HG web to digg thorugh logs for a given file Wondering "why was this code changed" or "what is this code trying to do" Today it is painful to dig this info up HG Blame allows you to see who last changed a given line, but many times that is just cleanup patches that don't change the functionality ability to see who made changes that actually changed the functionality Bonsai had better functionality Showed checkin comment for every given line and jump to that revision You could then manually step on revision back since the version numbers were numeric HG log should be able contain links to the previous version for every line Boris Bonsai had the feature of being able to see the list of files touched by a checkin Clint Anything to make hunting down regression ranges Able to query by the person that checked in the patch and by source code directory.

January 15th, 2009

DPS911 - Project Description and Plan

I will be continuing my work on hgweb for this new course. I’ve had a talk with jorendorff and ted regarding the bugs I can try to fix. I have various options that I can take on. There are definitely things available that I can continue to work on throughout the semester. Ideas for Releases Ideas for Releases bug 459727 - pushloghtml should show more than 10 entries at a time There are still issues with this bug that need to be fixed. The basic functionality works but there are some things that still need fixing. Things like the buglink functionality, loading more entries according to screen size, optimizing the code for efficiency etc still requires work. History that is filtered by directory Filtering the pushlog according to the directories touched. According to jorendorff this is a large feature that will require work. I told him that we were required to put out releases every 2 weeks so he recommended that I choose one large feature to work on (this one). I can implement this feature and improve its functionality for at least a few releases. Hg Annotate is very slow This is a bug that people have been working on for a while. Files using Hg Annotate are taking forever to load in any browser in comparison to loading the same file in bonsai. This is a medium difficulty bug that I could fix in maybe 2 releases or so. Expand/Collapse functionality for files touched Last semester I fixed a bug that added files touched functionality to the pushlog. My bug passed the review but now I want to add expand/collapse functionality for the files touched. Goals My goals are to give priority to the bugs that users want done. I will probably start by working on the pushloghtml should show more than 10 entries at time bug and then see what I want to work on next. Ideally I would like to get all the four ideas I have listed above done but it is hard to say whether that will be possible. Inevitably there will be issues with my patches and I will have to probably dedicate some releases to bug fixing. Possible Problems Last semester’s experience with this stuff means that I won’t have a large learning curve. It should be smooth sailing but problems will definitely creep up. I will probably get stuck and will require help. In that scenario I will need to consult somebody with more knowledge than myself. Usually ted, jorendorff and djc are available but if they are not, then there might be issues. My project doesn’t involve waiting for anything or for anybody to do anything (except when I will be waiting for reviews) so in the end everything is on me to get things working.

Project Details

v0.1 Release Details

to get pushloghtml working on my machine (some progress made on this part) to locate the source files that dictate the amount of pushlog entries that get displayed and then alter it to display more entries Turns out that there was some miscommunication and I need to add an AJAX scrolling feature that dynamically loads more changesets to add code that allows expand/collapse functionality (apparently all I need is a js lib according to humph) Turns out that there was another miscommunication as Ted has already added this feature. Look here. However the patch has some problems that will need to be fixed. I might fix this for my 0.2 release (undecided for now) To release v0.1 on time

v0.2 Release Details (More Information) (Final result)

To continue work on bug 459727 Implement a loader GIF (View here) Getting the correct Chronological order when adding more entries OnScroll (View here) (Final result) Getting merge changesets to show up correctly when adding more entries OnScroll (View here) (final result) Making sure the new entries that load OnScroll comply with bug 445560 (Expand/Collapse feature) ==> REMOVED from v0.2 - this functionality should wait until the expand/collapse functionality itself is approved and goes live. This should be a separate bug To make buglinks work To make changeset links work To get the correct date formating On first page load show 20 entries instead of 10 (View here) To get the correct author names (right now, sometimes it returns the email address while other times the name of the author) (View solution here) To fix ted's patch for bug 445560 (View here) To implement "the which files were touched by a changeset" feature for bug 448707 (View here)

v0.3 Release Details (View here)

Problem with my patch for bug 445560 (View here) Fixed problems with my patch by adding 2 new patches (View here) Fix 1: Should use ids instead of dates Fix 2: Should store unique identifiers in the class instead of the id of the tag My patch for bug 448707 approved! (View here) Implementing the Graph view for the pushlog Downloading Dirkjan's (djc) Graph view's source code (View here) Getting djc's Graph view running on my computer (localhost) with hg serve (View here) Understanding how djc's Graph view works (View here) (View here 2) Coding the Graph view for the pushlog 1st interation - Make it look similar to the coal template's graph view (View here) Final Result of the Graph view (View here) Added bug link feature Formatted correctly to look similar to other gitweb_mozilla pages Added on mouse over tooltip functionality Problem with my patch for bug 459727 (View here) (fix for my patch) To calculate the max use SELECT COUNT(*) FROM ... instead of asking for all the rows and then counting them getMaxEntries called, then 'start' used once the user scrolls. But the user might scroll before the first result comes back, while start is still zero -- a race condition. The JS code isn't consistently indented Patch seems to contain some non-ASCII characters. Bugzilla doesn't render it properly. Consider using an HTML entity reference, like – or —, instead of an actual Unicode character. In a few places the b tag is used where the server uses the strong tag; be consistent Loads more results when the user scrolls. What if the initial results don't fill up the window? Better to move CSS and formatting to stylesheets where possible Problem #1 explained in comment #9 by jorendorff still existed with my new patch (it is fixed, view here) Improved the bug link feature for bug 459727 (View here) My patch for bug 445560 approved! (View here) v0.3 complete!

v0.4 Release Details

Task Details Status Link(s) or Notes Optimize the client side code Maintain/add functionality but decrease the lines of code. Some lines are being repeated that can be put into functions. Done Fixes for bug 459727 View detailed info Fix the buglink feature Create a better regex so that correct buglinks are formed 100% of the time Make sure that buglinks are formed even when there is more than 1 buglink on one line Done Fix the onPageLoad feature Load more changeset entries according to the user's monitor size in order for the scroll bar to appear. Currently a script error is occurring. Done Loader.gif should appear properly When the user gets to the end of page loader.gif should show to signify that something is being loaded. That isn't happening properly, the user should see the GIF appear. Done Fix potential XSS attack vulnerability Get rid of var pushData = new Function("return " + pushCheckins.responseText) () and use something like JSON.parse Done Merge Changesets Background Color Problems No merge changesets have a grey background color, instead they wrongly have a white background color Done

v0.5 Release Details

Task Details Status Link(s) or Notes Show the first screenful in a second or less Use floats Get rid of the table and use either a div or a pre Format the page using CSS instead of a table Move CSS to style sheet Try different patches posted on the bug page Try the coal template and the default hgweb template Done Fixes for bug 459823 View Detailed Info Don’t lock up the browser while the rest loads Done Don’t take forever to load the rest of the file (currently the loading time is ~20sec) Done

v0.6 Release Details (View Detailed Info)

Task Details Status Link(s) or Notes Increase the loading time of hg annotate for the paper theme Use floats Get rid of the table and use either a div or a pre Format the page using CSS instead of a table Done Download the patch Refactor variable names in hgpoller and hg_templates Use clearer variable names Done Fixes for bug 445560 Refactor hgpoller/pushlog-feed.py and hg_templates/pushlog.tmpl to resolve the bitrotting issue Examine the new changes in pushlog-feed.py and pushlog.tmpl Refactor pushlog-feed.py and pushlog.tmpl to fix the bitrotting issue Retain the original functionality of the patch Done Fixes for bug 459727 Remove the page navigation links from pushloghtml Done Fixes for bug 459727

v0.7 Release Details (Detailed Info)

Task Details Status Link(s) or Notes The function which retrieves the total amount of entries in the database, getMaxEntries() is causing the browser to freeze. Find a solution for this problem Get rid of getMaxEntries() Pass in the max total by using the template system Done Fixes for bug 459727 View Detailed Info Solve the scroll bar problem Dynamically load enough entries, according to the users screen size, until the scroll bar appears Done Solve the split bug Some merge changesets are not being displayed at all Find a different way of parsing merge strings, split() is not working properly Done Solve the duplicate bug with merge changesets sometimes the last entry in a merge changeset is repeated in the next entry Is this actually a bug or not? Not a Bug Use a function to create rev links to avoid code duplication Done

Project News

DPS909 Timeline

September 18th, 2008 Picked the Mercurial History Browsing project September 23rd, 2008 Had the first conversation with jorendorff Talked what I know about mercurial and python Chatted with jorendorff, ted and djc about starting on the project. Brainstormed some ideas about what to do during the project Jorendorff made a blog post calling for ideas about the project September 25th, 2008 Chatted with jorendorff on setting my development environment Email conversation with jorendorff, asked some preliminary questions about the project and a hello world type of activity jorendorff replied I replied to jorendorff concerning setting up my development environment September 26th, 2008 Email conversation with jorendorff about setting up the development environment September 28th, 2008 Played around with hg. Details here Email conversation with jorendorff, asked for help on setting up the development environment and suggestions on a v0.1 release September 29th, 2008 Email conversation with djc about setting up hg_templates Email conversation with jorendorff about having the development environment completed and ready to use and picking what I will be working on for my v0.1 release ideas October 1st, 2008 Chatted with ted about getting a hold of the pushlog database Email conversation with jorendorff about setting up pushloghtml on my development environment and some questions/concerns regarding pushloghtml October 2nd, 2008 Development Environment almost setup and running except for pushloghtml Decided on v0.1 release (tentative, still need to clear it with humph) Email conversation with jorendorff clarifying some concerns about pushloghtml Email conversation with ted explaining how to setup pushloghtml, concerning the buglink.py file October 3rd, 2008 Decided on v0.1 release (confirmed with humph). Updated v0.1 release blog post Email conversation asking about simplejson and pysqlite2 modules Email conversation with bsmedberg explaining simplejson and pysqlite2 questions October 5th, 2008 Problems with getting pushloghtml to work here. Work in progress... October 6th, 2008 Multiple email conversations asking djc and bsmedberg about a "no module found" error relating to simplejson and pysqlite2 Conversation with me asking why the error is happening djc replies asking whether I get a traceback I reply with the traceback output bsmedberg replies with possible causes I replied with various things that I have tried to get it to run properly October 7th, 2008 Another email conversation with djc, discussing ideas about the causes of the "no module found" error October 8th, 2008 Email conversation with djc asking questions about building Mercurial from source. I was having some trouble with it October 9th, 2008 Email conversation with djc discussing further issues with building Mercurial from source Email to let jorendorff, djc, ted and bsmedberg know that the chronic problems with Mercurial have been resolved Development environment setup. Ready for coding to begin! October 13th, 2008 Filed the bug: pushloghtml should show more than 10 entries at a time ted has already filed a bug for the expand/collapse feature: html pushlog should fold merge changesets into one line Chatted with jorendorff and ted about setting up my development environment Chatted with ted about the expand/collapse feature October 14th, 2008 Applied Ted's patch to my development environment to fix the bug: html pushlog should fold merge changesets into one line, check out my blog post for details Chatted with djc about pushloghtml This is where the confusion occured, I thought they wanted to load more items staticly when they wanted an OnScroll features I asked them what AJAX library I should used Chatted with ted about a problem with applying his patch October 16th, 2008 Email conversation with ted discussing some questions about implementing an AJAX scrolling feature to load more pushlog changesets ted replied with possible solutions/steps I could take October 17th, 2008 Email conversation asking questions about the json-pushes script October 18th, 2008 Made a blog post regarding bug 459727 - Pushloghtml Should Show More Than 10 Entries, Working on v0.1 Release Email conversation asking ted about xmlHttpRequest and JSON questions October 19th, 2008 Edited the blog post regarding bug 459727 - Pushloghtml Should Show More Than 10 Entries, Working on v0.1 Release October 23rd, 2008 Email conversation with djc regarding bug 459727. In particular asking about the JSON script that returns turns the data from the server side, which is gotten via an xmlHttpRequest() October 29th, 2008 Had a conversation with djc regarding bug 459727 Discussed how my project was progressing Asked him a question related to the email conversation we had Had a conversation with ted regarding the bug 445560 expand/collapse feature Asked him what his patch doesn't do properly Wanted to find out what needed to be fixed November 3rd, 2008 Had a conversation with djc regarding the date formatting for bug 459727 Asked how the date is formatted in the way it is right now November 4th, 2008 Had a conversation relating to v0.2 release with jorendorff (decided what I will include in v0.2) [http://sidkalra.com/files/mercurial/email/conversation_email_Nov04-08.txt Email conversation with djc regarding bug 459727 (the OnScroll feature). Asked questions the data that the JSON script returns djc replied with possible solutions/methods to solving my problems November 5th, 2008 Made a blog post relating to bug 459727 November 6th, 2008 Had a discussion with jorendorff about how to implement the which files were touched by a changeset feature November 7th, 2008 Made a blog post discussing the details for v0.2 release Made a blog post about a problem with implementing merge changesets for bug 459727 November 8th, 2008 Made a blog post adding a loader GIF for bug 459727 Made a blog post about fixing bug 445560 November 10th, 2008 Had a discussion with ted about the which files were touched by a changeset feature Asked him about the back-end code related to this bug Had a discussion with jorendorff about the which files were touched by a changeset feature Showed him what I had done so far and asked him if he was OK with my implementation November 11th, 2008 Made a blog post about fixing bug 448707 Had a chat with jorendorff and ted regarding whether bug 459727 should comply with bug 445560 from the get go Made a blog post about showing the correct author for new entries OnScrollDown for bug 459727 Made a blog post regarding bug 459727: On initial PageLoad the pushlog should load 20 entries instead of 10 Made a blog post concerning a problem with the chronological order for bug 459727 November 12th, 2008 Had a discussion with ted and bsmedberg about the issue explained by the blog post here Edited the blog post about not getting the right author name. The edit provides a solution to the problem Edited the blog post detailing my problem with ctx.parents()[i].description() Chatted with ted about a problem with not getting the correct author name. Sometimes the email was being shown while other times the actual name was showing up November 13th, 2008 Had a long discussion with jorendorff about fixing the issue described by the blog post here Edited the blog post detailing the problem and fix with ctx.parents()[i].description() November 15th, 2008 Made a blog post to add merge changeset functionality to bug 459727 November 16th, 2008 Posted patch for bug 459727 Posted patch for bug 445560 Posted patch for bug 448707 November 19th, 2008 Had a discussion with jorendorff about what to do for my v0.3 release November 24th, 2008 Made a blog post about looking forward to my v0.3 release Had a discussion with ted about fixing my patch for bug 445560 ted identified a problem with the patch I said I would fix it Made a blog post regarding this discussion Posted new hgpoller patch for bug 445560 Posted new hg_templates patch for bug 445560 Made a blog post about the two new patches for bug 445560 Emailed djc asking questions about his graph view implementation November 29th, 2008 Email conversation with djc. He replied to my earlier email asking about his graph view implementation December 1st, 2008 Had a discussion with jorendorff on the Graph view Made celebratory blog post about my patch for bug 448707 being accepted Made a blog post about understanding how graph data is passed from the server side to the client side December 2nd, 2008 Made a blog post about completing the 1st iteration for the graph (View here) Made a blog post about problems identified by jorendorff concerning my patch for bug 459727 December 3rd, 2008 Added a new patch for hgpoller for bug 459727 Added a new patch for hg_templates for bug 459727 Blogged about the above patches here Made a blog post explaining a fix concerning a problem with my newly submitted patch (see above) for bug 459727. Problem #1 explained in comment #9 by jorendorff still existed even with my new patch December 4th, 2008 Made a blog post about improving the bug link feature for bug 459727 December 5th, 2008 Had a chat with jorendorff and ted Talked about who should be the reviewer for bug Talked about continuing on and working on this project next semester Final discussion for the project for this semester Made a blog post about another patch of mine getting approved for bug 445560 December 7th, 2008 Made a blog post about implementing the graph view Made a blog post about completing v0.3 release

January 2009

Date Comments January 13th, 2009 Had a discussion with jorendorff and ted about what I will be working on this semester. January 15th, 2009 Made a preliminary blog post about starting off a new course for this semester. January 16th, 2009 Made a blog post about my v0.4 Release goals January 17th, 2009 Made a blog post about optimizing code regarding bug 459727 for my v0.4 release January 17th, 2009 Made a blog post about making better regexps for identifying bugLinks regarding bug 459727 for my v0.4 release January 18th, 2009 Made a blog post about linkifying all bug strings on a line by using recursion for my v0.4 release January 19th, 2009 Made a blog post about fixing XSS vulnerabilities and loader.gif issues for my v0.4 release January 22nd, 2009 Made a blog post about fixing the unresolved script error for entries loaded onPageLoad regarding my v0.4 release January 22nd, 2009 Made a blog post about fixing the problem with the background color of merge changesets regarding my v0.4 release January 23rd, 2009 Made a blog post about completing my v0.4 release January 27th, 2009 Had a chat with jorendorff relating to hg annotate January 28th, 2009 Made a blog post describing my v0.5 release goals for fixing hg annotate February 22nd, 2009 Created a patch to speed up hg annotate for the paper theme

February 2009

Date Comments February 2nd, 2009 Made a blog post explaining the problem with hg annotate February 3rd, 2009 Made a blog post examining other hgweb templates February 4th, 2009 Made a blog post examining other patches for hg annotate February 6th, 2009 Made a blog post explaining my fix for hg annotate February 7th, 2009 Made a blog post about running an experiment with Mat's patch for hg annotate February 7th, 2009 Made a blog post about completing my v0.5 release February 15th, 2009 Made a blog post about my v0.6 release goals February 16th, 2009 Made a blog post about refactoring variable names for the patch that fixes bug 445560 February 17th, 2009 Made a blog post about fixing annotate for the Paper theme February 20th, 2009 Made a blog post about examining all the changes with hgpoller/pushlog-feed.py February 21st, 2009 Made a blog post about fixing the bitrotting issue with hgpoller/pushlog-feed.py February 21st, 2009 Made a blog post about removing the page navigation links from the pushlog February 21st, 2009 Created a patch to speed up hg annotate for the paper theme February 21st, 2009 Made a blog post about completing my v0.6 release

March 2009

Date Comments March 12th, 2009 Made a blog post outlining v0.7 release goals March 12th, 2009 Made a blog post explaining how to retrieve the total number of entries stored in the database March 13th, 2009 Made a blog post explaining the scroll bar problem March 13th, 2009 Made a blog post explaining the scroll bar solution March 14th, 2009 Made a blog post explaining the split bug March 14th, 2009 Made a blog post solving the split bug March 14th, 2009 Made a blog post about code duplication March 14th, 2009 Made a blog post explaining the repeating bug with merge changesets March 15th, 2009 Made a blog post about completing my v0.7 release

Project Bugs

• bug 459727 - pushloghtml should show more than 10 entries at a time
• bug 445560 - html pushlog should fold merge changesets into one line
• bug 448707 - display files in pushlog
• bug 468089 - pushlog should have a graph view
• bug 459823 - hg annotate output on hgweb is painfully slow
• bug 471321 - Needs for an UI to run changeset queries

External Links

• Distributed Version Control
• Mercurial
• Mercurial Basics
• mercurial-central shortlog page
• graph view
• better shortlog
• mozilla-central pushlog
• Desired features for hgweb
• jQuery tutorials
• Using XMLHttpRequest
• bonsai
• hg annotate problem
  • hg annotate output on hgweb is painfully slow
  • http://www.selenic.com/mercurial/bts/issue1310