Tuesday, January 29, 2013

We're on Git!

I love Git and I believe in Open Source. It's been a long dream of mine to move our codebase at RIPE NCC from Subversion to Git and also open source part of my team's work there. Finally it's happening.

Subversion => Git

We had a Subversion repository with 40k+ revisions. This was an opportunity to also clean up some junk. At least I had to make sure that:
  • existing history, tags and branches in Subversion are preserved
  • Maven release plugin works
  • Bamboo (our continuous integration server) can build plans
  • Sonar (our static code analysis tool) keeps working
There's plenty of tutorials and case studies on the internet about how to do the actual migration. This was my streamlined process:
  1. Clone the entire repository for a specific project, which means:
    1. map the Subversion user names to authors in an authors-transform.txt file
    2. clone the repository using the author mapping file
  2. Convert svn:ignore properties to .gitignore
  3. Fetch branches and tags, which means:
    1. Copy remote branches/tags as local ones
    2. Turn the Subversion tags that git svn maintains as branches into proper Git tags
  4. Clean-up unused branches, etc. (optional)
  5. Push the newly created local tags and branches to the final Git repository

1. Cloning the Subversion repository

I had a Subversion work directory for each project so it was easy to extract the usernames for that repository:
svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors-transform.txt
Then I had to find a name and email address for each username. This can be challenging when no one in your department remembers whom that username belonged to.

Having the authors-mapping file I was able to clone the Subversion repository and create a local Git repository:
git svn clone -s https://localhost:10443/svn/project/ --no-metadata -A authors-transform.txt project
This command assumes that the standard Subversion layout (trunk, tags, branches) is used which was the case for most of our projects.

2. Convert svn:ignore properties to .gitignore

The svn:ignore property contains a list of file patterns which certain Subversion operations will ignore. This is perhaps the most commonly used special property and it would be great to preserve them in a form that Git understands. git-svn knows what I want:
git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

3. Fetch and convert branches and tags

git-svn does not seem to copy remote the branches and tags as local ones so let's encourage it a bit:
git fetch . refs/remotes/*:refs/heads/*
Having all branches as local ones we can erase the remote ones. The local trunk branch is redundant with our Git master branch:
for branch in `git branch -r`; do git branch -rd $branch; done
git branch -d trunk
At this point actually we can detach our local repository from the remote one, that is, remove the git-svn configuration options and folders:
git config --remove-section svn-remote.svn
rm -Rf .git/svn/
For some reason git svn maintains the Subversion tags as branches. Let's turn them into proper git tags:
git for-each-ref --format='%(refname)' refs/heads/tags | cut -d / -f 4 | while read ref; do git tag "$ref" "refs/heads/tags/$ref"; git branch -D "tags/$ref"; done

4. Clean-up

We may want to rename some tags by creating a new reference to them and then removing the old one:
git tag new_tag old_tag
git tag -d old_tag
This is also a good time for deleting tags or useless branches.

It's nice to remove the empty commits:
git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"' HEAD

If we have files that should have never been under version control this is the time to remove them (e.g. *.iml files):
git filter-branch --tree-filter 'git rm -r -f --ignore-unmatch *.iml' --prune-empty -f -- --all 

5. Push to the final Git repository

I set up our Git server using gitolite. That's the repository from which the team and the continuous integration server will fetch the code.

We can simply add a remote to our local repository and push there. "--all" pushes all refs under refs/heads but not the tags so we have to remember to push those as well.
git remote add origin git@our.git.server.net:project.git
git push origin --all
git push origin --tags

Final words

So now we have git. Offline operations are unbelievably fast: commands take more time to type than they do to execute. I can keep committing on the train on the way home and push at my earliest convenience. I can have cheap local branches. Fine-grained local commits that I can merge or reword before pushing. Endless possibilities.

I did this migration as a side project (mostly in my spare time) incrementally and it was all done (including the Continuous Integration server updates) in about a week without any major interruption to the team's work.
What's next? Let's open source! To be continued...

Sunday, July 15, 2012

Java Swing UI Quirks

In the enterprise world I don't get to doing desktop development often. A year ago I took over the maintenance of BDSup2Sub, a tool to convert and tweak bitmap based subtitle streams. This program is written in Java and uses Swing for the user interface. The original author mainly developed and tested it on Windows and did not polish the UI for other platforms. These days Ubuntu Linux is my primary desktop operating system (followed by Mac OS X) so I did some polishing.

Both the JVM GTK+ Look and Feel and the Ambiance theme in Ubuntu 12.04 could use a little love. While one could argue here that the bugs should be fixed in those projects instead of introducing workarounds in the applications the process is slow or fixes will never be backported to the currently stable releases.

Major annoyances

  • unreadable menu bar
Unreadable menu bar

  • dropdown menus and popups are missing borders and dropdown-shadow
Dropdown menu without borders and shadow
Popup menu without borders and shadow

  • missing menu separator
Missing menu separator

Determining the actual Look And Feel and theme

I am going to describe workarounds for the GTK Look and Feel, and one of them is only applicable for the Ubuntu Ambiance theme. I wrote some convenience methods to determine the current Look And Feel and the theme name:

Unreadable menu bar

This one is a JVM bug exposed by the Ambiance theme in Ubuntu 12.04. According to the bug report it is fixed in Java 8, but not (and never will be) in 7 and 6.

The JVM picks up the wrong colors so we have to set them explicitly. We can create a class for the action menus and use that instead of JMenu:

Menubar with correct colors

Missing borders for dropdown menus and popups

This could be actually fixed in the theme file but remember, the application should work out-of-the-box, so let's fix it by tweaking the theme in Java. The method below changes the style for the popup menus and sets the xThickness and yThickness fields to a sensible non-zero value. We have to create a JPopupMenu instance for its style to get updated and then the original SynthStyleFactory can be restored.

You can see the result on the screenshots below. They were taken after applying the fix from the next section so besides the border they also show the dropdown shadow.
Dropdown menu with borders and dropdown shadow
Popup menu with borders and dropdown shadow

Missing dropdown shadow for dropdown menus and popups

A Swing popup can have different weight (light, medium, heavy). The heavyweight popup uses a window as backend and has dropdown shadow. Heavyweight popup is not the default so we have to force it.

The util class I created for getting/setting the popup type:

Missing menu separator

This one could also be fixed by changing the theme and doing it from Java is very similar to the solution for the missing borders. We just have to manipulate a different region and create a JPopupMenu.Separator instance for its style to get updated.

Horizontal menu separator


IntelliJ IDEA is the Java IDE I use on Ubuntu Linux and the community edition is open source. Hats off to the Jetbrains developers for finding or leading me to some of the workarounds I posted above.

Saturday, June 16, 2012

Shift+click on a launcher icon to open a new application instance

I just noticed that finally the Ubuntu Unity shift+click regression fix made it to the trunk:

  • Committer: Tarmac
  • Author(s): Thomi Richards, Miklos Juhasz
  • Date: 2012-06-05 04:49:29 UTC

I proposed my branch in April but unfortunately it did not make it into 12.04. It's not a crucial feature but I still like it and I am happy to have it back. A Unity package with this fix is in my ppa for 12.04.
Thank you, Thomi Richards, for writing the AutoPilot test!

Tuesday, April 17, 2012

Fix blue people on Flash videos without losing HW acceleration

Adobe announced that they abandon Flash on Linux and 11.2 will be the last release after which they will provide 5 years support limited to security issues only.

11.2 was released with a terrible bug: Flash Player sends U and V components in the wrong order which results in incorrect colors. They are not going to fix it - all reports are being closed as CannotReproduce or FeatureRemoved.

This bug affects only users with NVidia cards but still: I can hardly accept that such a bug slips through all the quality checks and that they refuse to fix it. Users have not much choice: earlier versions are full of security holes, the latest renders the videos unwatchable.

Since Flash Player is closed source and Adobe doesn't care, we can just workaround this problem, and libvdpau is one possible layer where we can do this. VDPAU is an open-source library which allows programs to offload portions of the video decoding process and video post-processing to the GPU video-hardware. This is the only open-source component I can think of in the chain so this will have to be abused. While this goes against all my software engineering principles here is the deal:

We modify the VDPAU trace library. It's meant for tracing VDPAU functionality and not for working around Adobe's incompetence but that's our best shot. In this library we can swap the U and V color components so they will be in the correct order. The patch is not mine, the credit goes to Pierre-Loup A. Griffais. The patch stubs out all the tracing routines and only does color swapping. If you need to trace the VDPAU functionality remember to use the unmodified version!

I applied the patch on the 10.04 (previous LTS version of Ubuntu), 11.10 (current stable version) and 12.04 (upcoming LTS release). All of them have been built on Launchpad and reside in my Personal Package Archive (PPA).

If you are reading this and eager to use the patched version you will know how to create your own PPA and copy my packages there. You can also add my PPA to your package manager (instructions on the PPA site) and upgrade the libvdpau1 package. Downloading the debs and installing manually is also an option. Be aware that my ppa has other packages which you may not want to upgrade so make sure you know what you are doing.

Having the patched VDPAU library is not enough: you have to activate it by enabling tracing. You can do so by using the VDPAU_TRACE=1 before starting your browser, e.g. you can modify the Exec line(s) in your firefox.desktop or google-chrome.desktop file to start with env VDPAU_TRACE=1

Not an easy ride. While it was possible to workaround this bug that's not always the case. Which proprietary, closed-source software comes next? Try to find the open alternatives, which, in case of flash, is using HTML5.

In the meantime, enjoy YouTube videos, on Linux, even with NVidia cards!

Update 1: I have backported the official fix for Precise (12.04) and it is in my ppa now. Other versions have been removed in order to reduce maintenance burden.

Update 2: The fixed version is in the official archive now - no need for 3rd party ppa's anymore.

Friday, February 24, 2012

BDSup2Sub 4.0.1

BDSup2Sub 4.0.1 has been released. This is a bugfix release.

  • Fixed: Flashing subtitles - if the end time of the current subtitle picture is beyond the start time of the next one the subtitle was set to zero length.
  • Fixed: Resolution incorrectly read from the dvd ifo file in cases other than 720x576/720x480 16:9.
  • Fixed: Incorrect (too short) subtitle duration for dvd subtitle frames using subtitle effects

WDTV Live users: I have also updated the version that has the workaround for the WDTV Live vobsub subtitle handling bug.

BDSup2Sub v4.0.1-wdtv (for WDTV Live media player owners)

Sunday, November 20, 2011

Device left behind after umount in Nautilus

Whenever I mount an iso via the loop device an entry representing the virtual drive shows up in Nautilus in Ubuntu. It has been like this for ages and it is correct. However in the Lucid cycle (10.04) I noticed that the drive icon is not always removed after unmounting the iso file.

This turned out to be a kernel issue: the loopback driver failes to emit the change uevent when auto releasing the device. Finally a patch made its way into the -mm branch but the behavior under the current Ubuntu release (11.10) is the same.

I am using a workaround which is emitting a change event manually after umount:

udevadm trigger --action=change --subsystem-match=block --sysname-match="$DEVICE_NAME"

I created a script which sets the $DEVICE_NAME based on the mount point. For my own convenience I packaged it into a deb file that can be installed under Ubuntu. Basically it's just two scripts: mount.loop and umount.loop

The deb can be downloaded from my ppa for Lucid.

Update: The fix made it into Oneiric as an update. Starting from kernel version 3.0.0-16.28 the workaround is not needed.

Friday, October 21, 2011

Rsync transfer hangs

SSH works fine but rsync or scp hangs after a while which makes file transfers impossible. It turned out to be an MTU issue: setting it from the default 1500 to 1492 fixed the rsync/scp issue.
It happened to me again and having taken the first bullet now I decided to figure out what's happening behind the scenes.

Every network link has a maximum packet size called the link's MTU (Maximum Transmission Unit). The full path from one computer to another may travel across many links with different MTUs. The smallest MTU for all the links in a path is the path MTU.

The MTU and the packet should be as large as possible, in other words, the least fragmented. I found an RFC from 1990 about Path MTU discovery which describes how we can make the packet as large as possible.

We have to find out the smallest MTU along the links. The basic procedure is simple: send the largest packet you can, and if it won't fit through some link will get back with a notification saying what size will fit.

Here comes the catch: the notifications arrive as ICMP (Internet Control Message Protocol) packets known as "fragmentation needed" ICMPs.
Some network and system administrators view all ICMPs as risky and block them all, disabling path MTU discovery, usually without even realizing it. Then if the default MTU happens to be larger than the path MTU we are out of luck unless we manually change it.