Staying in (version) control

As part of my job, I find myself writing lots of bits of code for people. Until quite recently, my version control system was renaming the files and commenting in the header to keep track of changes.

Not the tidiest system

I say “quite recently” as I started using git as my version control system and have not looked back. I’m by no means an expert, but in this post, I’m going to give an introduction to using git in the context of scripting.

This post is really aimed at people who have no experience with version control systems or have heard about git but have never really used it (or have tried and failed to get the hang of it as I did…twice).

Setup

I work largely on a windows system. GNU/Linux users get it easy as git is probably already installed, so you lot can skip over setup. Everyone seems to have a favourite client and I’ve heard good things about GitKraken, SourceTree and SmartGit but I like the flexibility of the command line, so I work with Git For Windows. Download and install and for the purposes of this tutorial enable Git Bash in the shell context menu. This puts an incredibly useful “Git Bash Here” link on the (right click) context menu that we’ll be using.

The Aim of the Game

In this tutorial, we’ll be writing a script to open a file, then apply some filtering. Boring I know but it’s just an example.

Behold the almighty Gaussian Blur, pinnacle of image processing techniques.

Step 1: Make a working folder

Git works by tracking changes to files within a directory (called a repository). To keep things simple, when I start a new Image Analysis project, I usually make a directory structure that looks something like this:

That way the script folder will become our repository and everything in there will be tracked. There are other ways of selecting which files are tracked but I’m not going to cover those here.

Step 2: Initialise a repository

Navigate into your script folder, right click and select “Git Bash Here” (assuming you selected shell extensions during installation). You should be presented with a nice command window that looks something like this:

Now we’re going to make a new repository in this directory with the following command:

git init

You’ll see the following:

That’s it! You now have a git repository in that folder (and there is now a hidden folder in your repo called “.git” that keeps all the repo info). Notice the cyan text at the end telling you that you’re on a branch called (master). This just means that any changes will be made to this version (called a “branch”) of the code.

Step 3: code and commit!

Right, let’s fire up Fiji and get working on our Opus. Open the script editor (we’ll be using IJ1 language so select this from the menu). You can open the recorder [ Plugins > Macro > Recorder ] or just copy the following.

run("Blobs (25K)");
run("Gaussian Blur...", "sigma=5");

Save your macro (with a .ijm extension) into the script folder. You can always issue the following command (git status) to get a helpful description of what’s going on so let’s do that now:

Nothing is tracked automatically, so the first thing to do is to start tracking (all of) the files in the repo. We do this with the “add” command.

git add .

Now let’s check the status again.

Great, so we’re tracking the new script file. Because we’ve not committed anything before, it appears as a “new file”. Let’s go ahead and make an initial commit:

Here, the ‘a’ parameter includes all files in the commit and the ‘m’ parameter indicates that we want to include the commit message that follows, in this case, ‘Open file and blur’. Commit messages are required and moreover helpful to remind you what you did.

Talking of which, let’s check git status again

Step 3.5: Taking a Shortcut

Let’s face it, we’re all busy and who has time for all that typing. Thankfully, git has a really versatile alias system that lets you make shortcuts!

Type the following to open your global git config file:

vi ~/.gitconfig

Add the following text to the bottom (note, in the vi editor, you will first need to hit ‘i’ to insert text).

[alias]
ac = !git add . && git commit -am
sl = status

Now hit escape to exit insert mode then type

:wq

which will write changes ( w ) and quit vi ( q ). You’re still in your repo so check the shortcuts are working:

The first alias we added (ac) simply combines the two commands we issued to start tracking all files and to commit all changes. We’ll use it the next time we commit, but let’s get back to Fiji.

Step 4: Parameterise!

Assuming you still have your IJ1 code open, pull up the window and add the awesomeness of Script Parameters.

// @Integer(label = "Sigma for Gaussian Blur") intSigma
run("Blobs (25K)");
run("Gaussian Blur...", "sigma="+intSigma);

Save this file (same name), and we can try out our new shortcuts:

Note that I still had to include a commit message in the second example but it’s a lot quicker now with the ‘ac’ shortcut.

Step 5: Blast from the past

For reasons I’ll come to later, you shouldn’t often need to roll back changes, but one of the reasons I bothered keeping all the old versions of my scripts (see intro) was that if I’d removed a block of code, at some point, I might want to get it back (commenting out will often work but can lead to very messy scripts).

With git, this is a pinch and it’s really the thing that made me understand how useful git can be (for more than just scripts, but again, more on that later). First off, let’s see everything we’ve done so far by issuing the following command:

git log --oneline --decorate --graph --all

I use this all the time so I made an alias. I use la (for Log All – without the –all it will only show you the log of your current branch which I’ve also aliased)

You can see the two commits we’ve made here and each line has a unique identifying string (actually just part of it). The HEAD -> marker indicates that the files in the repo currently represent the state of affairs at this point in the commit history.

Here’s where it gets cool. Use the checkout command to move back to the previous commit, before we added Script Parameters.

You’ll get some warning about the dangers of meddling in the past, but now all your files represent their state at the previous commit. If you click back into your script editor window (assuming you left it open), it will realise that the original file has changed and ask whether you want to reload.

Hit reload and your script will be back to how it was at commit 4c045fa.

Awesome! To switch back you can just issue:

git checkout master

or

git checkout a2c72b

Step 6a: Stuck on a branch

For me, being able to jump around your commit history and have the files immediately represent that point in time, is really cool. It’s what made me ‘get’ git, but for many the killer app (and it’s clear why) of version control is branching.

Let’s say that you love your script but really, you think there are better types of blur than Gaussian and you want to have a play with some other methods. You could do this by making a few commits on top of your current position but it’s harder to keep track of which one that is (also this becomes much more important when you start sharing code with others).

Instead, what we’re going to do is to branch off a copy of the code that we can work on without affecting our ‘master’ branch. One way to do that is by issuing:

git checkout -b filters

This is not exactly the canonical way to create a new branch, but for our purposes it’s a good option. The new branch is given the name ‘filters’ but can be almost anything. Using checkout (instead of branch) also switches us to the new branch immediately as shown by the cyan bit at the end of the current path.

Now we can start working on our script knowing that if things go horribly wrong (no matter how many commits we’ve made on the filters branch) we can always roll back to the master branch to make changes, start a new branch or even delete the current branch with:

git branch -D filters

Note the capital ‘D’ forces deletion even if you have code that isn’t saved anywhere so use with caution!

Where were we, right, filters. Make sure you’re on the filters branch then edit your script. What about a median filter?

// @Integer(label = "Sigma for Gaussian Blur") intSigma
run("Blobs (25K)");
run("Median...", "radius="+intSigma);

We’re pretty happy with that so let’s commit the change on the filters branch.

git ac 'Replaced with median filter'

Oh no! We forgot to change our script parameter label. How embarrassing. Let’s go back and edit the label so it’s correct.

// @Integer(label = "Radius for Median Filter") intSigma
run("Blobs (25K)");
run("Median...", "radius="+intSigma);

We have two options here. We can either run the same commit as before and end up with two commits for what is really one set of changes, or we can use the amend switch to git commit. The whole command looks like this (I alias this to ‘aca’)

git add . && git commit –amend -a

This doesn’t take a commit message, but will provide you with the vi editor and the commit message of the last commit (and a helpful display of which files have changed)

If you’re happy hit :q (we don’t need the ‘w’ unless you want to change the message) to amend the current changes into the previous commit.

Step 6b: Coming together

Make some more changes if you like then let’s call that project a success. If you look at your log it should look something like this:

We’re happy that the median filtering works as expected, so let’s merge the ‘filters’ branch back into the main project branch (‘master’). To do this we need to switch back to the master branch, then merge the filters branch. All of the commands are fairly intuitive, so you can do that with:

git checkout master

git merge filters

Note that in the above process you see the term “Fast-Forward”. This occurs because there have been no changes on the master branch since you created the ‘filters’ branch, so the history will appear to be linear, like all of the ‘filters’ branch commits were just applied to the master branch:

If you don’t want this, you can add the switch –no-ff to the merge command above and your log will look like this:

The big difference being that a new commit (called a merge commit) is created. Note that this is the default behaviour if there have been changes to the master branch. One final thing you may want to do is to remove the ‘filters’ branch now that it’s merged:

git branch -d filters

Note that because all of the changes now make up part of the master branch, deleting the filters branch is a completely safe option. It’s only when you have unmerged branches that you have to force with the ‘-D’ switch (because you can potentially lose code).

(Hopefully not) The end

So that wraps up an absolute beginner’s guide to how I use git in the context of image analysis scripts. Hopefully it will get someone to take the step into using version control. The notable absence here is the concept of a remote repository which is just a copy of your entire repos history, stored somewhere else.

It is definitely worth learning about remote repos as they are both a way to store a backup of your code but also the best route to share and collaborate on code. The de facto standard is github, but I also use bitbucket as it allows for free private repositories which are essential if you’re working on code that you’re not ready to release to the public.

These sites (and the links below) should help you to set up and utilise a remote repo once you have the hang of local version control but for now it’s beyond the scope of this post.

Epilogue: Version Control EVERYTHING

If you get the bug like I did, you’ll probably start to think about version controlling everything. This works well for a lot of things, and not so well for others. Remember that git compares line-by-line so anything that can be written as plain text plays well:

dotfiles: if you do any sort of server admin, you’ll have all your configuration set up just the way you like it. Why not put your config files into a repo so recovery (or cloning) is as easy as “git clone”. This is not a new idea, and there are plenty of examples of how to implement this.

micromanager config files: the config files for micromanager are just plain text. Great to have a backup and be able to see how your config changes over time.

websites: this is the one that really made my life easier. Websites are just HTML, CSS and javascript (which are all plaintext). Branching is particularly useful if you want to add a feature or test some code without breaking everything.

presentations: I’ve seen both the Open Microscopy group and ImageJ’s Curtis Rueden give presentations with Reveal.js which is awesome. You can put your presentation code in a git repo and you’re done! Great for slide reuse!

Further Reading

Below are some of the resources I’ve found useful:

A great starting point and reference, Pro Git covers just about everything you need to know and is available under CCBY-SA here or in hard copy on amazon.

The Bitbucket website has a really nice and clear set of tutorials you can work through. It also has what I think is the most helpful description of merging and rebasing.

Act like an IT professional and embrace google (or your favourite search engine). Anything you can think to search has probably already been answered on Stack Exchange or similar.

Play! The best way to learn is to play. Make a to-do list, set up a repo and mess around.

Advertisements

4 thoughts on “Staying in (version) control

  1. Jan Eglinger

    Great post, thanks for sharing! For those less command-line savvy, I also recommend one of the GUI options for using git: Atlassian SourceTree, or the GitHub desktop app.

    Oh, do you have your blog sources on github as well? I’d make a pull request then:

    — Fiji’s Curtis Reuden
    +++ ImageJ’s Curtis Rueden

    Jan

    Like

    Reply
    1. Dave Mason Post author

      Thanks for the comments Jan. Good catch, post updated.

      Odd though it sounds, I think one of the things I found a bit overwhelming with the GUI frontends (which are supposed to make things easier) is that you can see all the options. At least with the CLI (which I tend to be happier with anyway) you can learn what you need as you go. Just my experience, others will vary! I could probably navigate them fine now, but it would be hard to leave my aliases behind.

      Like

      Reply
  2. Pingback: Correcting the record | Post-Acquisition

  3. Pingback: How Revealing! | Post-Acquisition

Comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.