Mercurial SVN Integration Using Erlang

I wanted to make it possible to keep a Mercurial (Hg) repository in sync with a series of Subversion (SVN) source repositories. Due to the way certain Subversion repositories are structured it is not always convenient to have a common Mercurial and Subversion root. I am actually using Mercurial to coalesce a collection of related modules into a single checkout for an agile development team. Consequently, the Python hgsvn scripts are unworkable for the scenario I am in. So I rolled my own hgsvn in Erlang and it was surprisingly easy to do.

The Erlang hgsvn module follows this process:

1. Get the current revision from svn info
2. Get all future revisions with changes from svn log -rBASE:HEAD
3. Parse out the Author, Date, and Comment from svn log
4. Update to the next revision using svn up -r #
5. Add any new files added and remove any file deleted by that revision to Mercurial using hg addremove
6. Commit the revision to Mercurial as that Author on that Date with that Comment
7. Repeat from (2) until no more revisions are available

Using Erlang for this made the code very straight-forward. I previously attempted this in Groovy and beyond failing to get it working I found the code incredibly difficult to understand without detailed comments.

To run this code you would invoke the following command in the root of a Subversion working copy in your Mercurial repository:

erl -pa <path to hgsvn.beam parent dir> -noshell -s hgsvn -s init stop

You can also specify a set of related repositories (ex: from a common server) and update them together in revision order:

erl -pa <path to hgsvn.beam parent dir> -noshell -s hgsvn -s init stop -repo_set path/to/repo1 path/to/repo2

You can even specify multiple sets of related repositories and update them set by set in revision order:

erl -pa <path to hgsvn.beam parent dir> -noshell -s hgsvn -s init stop -repo_set path/to/repo1 path/to/repo2 -repo_set path/to/repo3 -repo_set path/to/repo4

Further to that you can specify stop revisions for each repository in a set:

erl -pa <path to hgsvn.beam parent dir> -noshell -s hgsvn -s init stop -repo_set path/to/repo1 path/to/repo2 -stop_set 1023 1432

If you specify a stop_set for one repo_set you must provide one for every repo_set. It is perfectly workable to put nothing after the -repo_set as hgsvn will interpret that to mean HEAD for every repository in the repo_set.

You can have an unlimited number of repository sets and an unlimited number of Subversion paths per repository set.

This code was developed to work with the output format from Subversion 1.4.4 (r25188) on Mac OS X 10.5 and Subversion 1.4.6 (r28521) on Ubuntu Hardy Heron. It should work with any 1.4.x Subversion though. Make sure to post any issues here as comments (please include the output of svn info).

UPDATE (2008/07/15): now handles case where one or more SVN WCs have no new changes

UPDATE (2008/07/15): adds SVN watcher that automatically collects changes (ex: -watch 30min)

UPDATE (2008/08/01): this is now erlang_hgsvn on GitHub

Agile Software Development and The Mercurial SCM

Mercurial enables individual developers to rapidly make and commit incremental change-sets locally without affecting others. This poses a risk to an Agile team in that it enables isolated development. Why would Mercurial be good in this situation then? As far as I am concerned certain things have to be done on branches: examples include destabilizing projects such as experimental changes to underpinning framework, code stabilization prior to a release, and production bug fixing. Tools like SVN actually aggravate integration by making branching so complex.

Due to the difficulties of merging in SVN my company only merges fixes from the stabilization and production branches once per sprint. Mercurial makes this problem mostly disappear by designing around something that operates very much like private developer branches. In taking this approach it must absolutely make merging easy to succeed and it does. Gone is SVN's absurd revision range selection and merge to working copy. Instead you "pull" changes from another repository into your own. This does not affect your working copy and instead reconciles the acyclic revision graphs of the two repositories internally. Sounds complex, but 1) it is not, and 2) it is handled for you.

If your repository has revisions with different identifiers but common parents when compared to the other repository Mercurial knows that there is a branch. Because Mercurial uses a cryptographic hash for the revision identifiers it is able to trivially reconcile identical revisions made in separate repositories and fold them into one instead of creating unnecessary branches. In the case that a branch results from the "pull" Mercurial notifies you by saying that there are now multiple head revisions and that a merge is required. You then run "hg merge" to reconcile the branches.

Trivial differences such as files deleted/created/modified by one branch and untouched in the other are immediately applied. More complex cases are passed to a auto-merge tool (unless you are using Windows, where this is manual right now) and any non-trivial merge results are then passed on to a graphical tool like FileMerge on Mac OS X or Kdiff3 elsewhere.

Once you have an actual conflict and the graphical merge tool is up it is the same challenge that merging always is. However, with the simplicity of kicking off a merge it is easily done daily or even hourly to avoid complex conflict resolution. A day's worth of changes are much easier to merge than a month's and it makes unexpected conflicts surface much more quickly. That brings us back to Agile and Scrum by tying into developer communication.

Developing further on the topic of developer communication is direct developer to developer repository sharing. If I were working with a colleague on a project and we both used Mercurial we could push and pull changes directly from each other. Quite often this is how branches are used at my company. Why burden the central server with a peer-to-peer usage scenario? Once myself and my colleague have unit tested, documented, and reviewed the code in our sharing arrangement we can "push" those changes to the central server for backup and sharing with the larger team. This allows us to honour the scrum rules without being hamstrung by them in terms of being unable to track and preserve sub-incremental work. In effect it protects your adoption of Scrum by not forcing your developers to go against the grain of the tools and their desire to preserve their work in order to follow it.

Mercurial gives the last mile to individual developers by allowing them to manage incremental unready changes and in so doing it removes a key barrier to adopting Scrum. Subversion and Mercurial can trivially coexist even in the same working copy. Just make sure to add Mercurial's meta-files (.hg, .hgignore) svn:ignore and SVN's meta-files (.svn) to .hgignore. Now you can use Mercurial for inter-developer code sharing of unready change-sets and Subversion for authoritative production-ready code. A happy marriage at last.

Ideally, I would have axe Subversion and have Mercurial on the server for the authoritative code as well, but it is beneficial in that it creates a tool-based distinction between preserving and sharing incremental changes with peers and committing authoritative changes to the main code-base. A similar advantage is attainable with a Mercurial server setup by using separate protocols or authentication could distinguish pushing to the authoritative server from simply pushing incremental changes to your peers. As an added benefit, using Mercurial on the server also preserves the incremental changes for review if needed.

Further benefits come from the fact that Mercurial warns you if the push will create remote branches. This means that other people have pushed to the authoritative server and you should merge in their changes first. To do this you pull *from* the authoritative server so you can merge and test locally to avoid breaking the code in the repository as a result of incompatibilities between yours and the other user's changes. Once you have merged you push again and it should go through. Even if someone decides to ignore the remote branches and force the push anyway Mercurial prevents this from poisoning the working copy of the server by stopping any updates there until a merge is done. This could cause new code to be delayed in getting into a nightly build, but is not likely to occur when angry teammates come to your desk asking why you forced your changes into the authoritative repository instead of merging. It will not, however, cause the build to fail which would happen in the Subversion case.

Digg - Agile Software Development and The Mercurial SCM

Getting to The Mercurial SCM: How CVS and SVN Failed Me

Why is it that we are drawn to particular revision control systems like CVS and SVN? If you have never used version control, then this post will make no sense so you should save yourself and walk away. Cheers, and apologies.

My first encounter with Revision Control was using CVS on an engineering work term. At the time it seemed like a revolutionary concept. You could actually track source code changes with a list of notes explaining the changes made (a task I previously thought was only possible to do manually and primarily done so in a log book). So enthusiastic about CVS that during the following academic term I ran my own CVS server with SSH logins for my team to use in the term software project.

During that term I rapidly discovered some serious limitations of CVS, notably renaming and directory versioning. The idea of losing revision history after renaming a file seemed crazy to me. I went looking for a replacement to CVS and predictably stumbled upon Subversion (SVN). It gave me a CVS-style interface plus my desired rename support and versioned directories. Beyond that it allowed arbitrary metadata properties to be attached to files and directories and provided cheap branching. Branching, I thought to myself, why had I not been using that in CVS? I quickly went through the tutorials in the now-famous SVN Book and imagined a world of branch strategies to handle all the times code had been blindly overwritten during the previous term.

During the following work term I tried, unsuccessfully, to affect a migration from CVS to SVN at my then employer. I had more success in the following academic terms when I replaced CVS with Subversion on my server and convinced a good number of people in my class to use it rather than the school's provided CVS server. The project team I was on used SVN branches in a purely exploratory way and we eventually abandoned the idea as causing more trouble than it was worth due to the idiosyncrasies of Subversion's approach to merging. I later discovered that SVN had inherited its merge to working copy model from CVS.

Subversion's merge to working copy and commit model has some weaknesses which are not obvious on initial inspection. Among others, it cannot track the parents of a merge and it requires a revision range. My only successful approach to coming up with the revision range has been to go back through the change logs until I find roughly appropriate dates or a previous merge log message which a colleague entered showing the last revision range merged. I have never understood why SVN and CVS require a revision range, but my searches for better alternatives at the time turned up nothing.

It was not until over a year later while I was hunting for random learning topics in Google's TechTalks series that I stumbled upon a video about The Mercurial Project. The topic description was not even obviously about SCM and I only watched it out of pure random exploration. I am very glad that I did. Bryan O'Sullivan's presentation was concise and comprehensible and opened my eyes to a substantially more robust approach to branching: representing branches as revisions with more than one parent revision. Mercurial does much more with branching than that, but without strangling the user with complexity that discourages the use of branching. Mercurial is Consise Software to the core. Even a passing glance at the Behind the Scenes look from Distributed Version Control with Mercurial makes that clear to me. Unlike most systems I have used I cannot see anything which I would remove from Mercurial.