Planet

My new KOrganizer showing my old appointments.

I’m running Debian Sid, because I like to have a rolling distribution with the latest and greatest software as soon as it is released. Unfortunately, the KDE packagers in Debian lack (wo)men power and still ship 4 year old KDE PIM packages. Using their experimental KDE repository does not help.

Since I have multiple annoying issues with Kontact 4.4.11 and because I need it functional for work, I decided to take a rather extreme step: Compiling it myself from the source code repositories. Turns out that it isn’t all that difficult. I documented the process below for those who want to follow my example. It enables me to stay up to date with the development whenever I want and I receive bug fixes immediately after they have been pushed without having to wait for them to be packaged for my distribution. Also, in case I want something to be fixed that still isn’t in latest master, I can just do it myself now.

In order to get started I read KDE techbase which is quite a good resource even though it could still be improved. At first I had to add the following to my ~/.gitconfig file.

[url "git://anongit.kde.org/"]
        insteadOf = kde:
[url "git@git.kde.org:"]
        pushInsteadOf = kde:

Then I could start cloning the git repositories:

$ git clone git://anongit.kde.org/akonadi
$ git clone git://anongit.kde.org/kdepim
$ git clone git://anongit.kde.org/kdepimlibs.git
$ git clone git://anongit.kde.org/nepomuk-core
$ git clone git://anongit.kde.org/nepomuk-widgets
$ git clone kde:soprano.git
$ git clone git://anongit.kde.org/strigi

At first I tried to just clone kdepim, but I needed the others as dependencies. The Strigi repository uses git submodules, so I had to do the following:

$ cd strigi
$ bash change_to_development_git.sh

While I was trying to make everything compile without error, I needed to install the following packages. Please note that on your system you might need additional packages like build-essentials which I already had installed. Finding out which package is missing is quite easy from the cmake output.

$ sudo apt-get install libgrantlee-dev libsasl2-dev libprison-dev libical-dev libakonadi-dev libgpgme11-dev kdelibs5-dev libqtwebkit-dev libqt4-opengl-dev libiodbc2-dev libraptor2-dev librasqal3-dev librdf0-dev libbz2-dev libdbus-1-dev libfam-dev libclucene-dev libclucene-core1 libboost-all-dev libqjson-dev

The techbase told me to have some environment variables set, so I added this to my ~/.bashrc file.

# KDE
export KDEDIR=/opt/kde
export KDEDIRS=$KDEDIR:/usr
export XDG_DATA_DIRS=$XDG_DATA_DIRS:/usr/share:/opt/kde/share

Then I sourced the file, to have the changes right away.

$ source ~/.bashrc

Of course I created /opt/kde and gave my user the necessary rights to write into this directory. Then for each repository I cloned before, I changed into the directory of the repository and executed the following commands.

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/opt/kde
$ make -j 3
$ make install

In rare cases I got an error which I fixed by either installing a missing package or by editing some file. For example, in kdepim I needed to change build/kalarm/cmake_install.cmake because it was trying to install something to /usr where my user had no write access to. In strigi/strigidaemon/bin/daemon/eventlistener/eventlistenerqueue.cpp I had to add #include <unistd.h> in the beginning for it to build.

Once everything was build and installed, I removed my KDE PIM applications from Debian to be sure there are no conflicts. I couldn’t remove akonadi or some libraries because of dependencies.

$ sudo apt-get remove kontact kmail korganizer kaddressbook akregator

In the #kontact IRC channel I was told to execute the following command which allowed me to have access to the settings in the newly build applications.

$ kbuildsycoca4 --noincremental

The development version of KMail

After this is done, I would suggest a restart to be sure that no old libraries are loaded or old Akonadi is running.

Then you can start your new KDE PIM applications and enjoy them. I’m really impressed by the progress done over the last few years and especially enjoy IMAP idle for Inbox folders and the ability to run all PIM applications independent from each other. Kontact really is a professional full-fledged groupware client.

Have fun with your brand new PIM applications and consider contributing if you find the time!

While only a warning in setup-kolab, some people set up a Kolab Groupware server with with an improper system fully qualified domain name, with all sorts of undesirable effects. I'm actually seriously considering making this warning a fatal error.

Why does a system require a proper FQDN? Is it actually required?

I'll answer the second question first, since it has the shortest possible answer: No.

Of course a proper FQDN is not absolutely required. But it is strongly recommended. For example, the default "localhost" and "localhost.localdomain" names you'll find in /etc/hosts (with address 127.0.0.1) are not "fully qualified domain names".

Right after you install a system (unless you have a fancy network infrastructure with (dynamic) DNS and DHCP, or a prepared DNS entry for a static IP address), the only way a system knows how to refer to itself is using these names "localhost" and "localhost.localdomain".

With a Kolab Groupware setup, this is somewhat problematic. Not because there's a guarantee it won't work, but because it is an insensible default, and secundary components that install alongside Kolab Groupware won't work as expected.

So what is a proper FQDN? Well, it exists of at least two parts: the host's name, and a domain name. For example, if the host's name is 'kolab01', and the domain name is 'example.org', the FQDN would be 'kolab01.example.org'. Please note that the domain name itself 1) exists of the domain name you (probably) registered and the top-level domain name you registered it with - apologies to those in the know, for bastardizing the terminology, and 2) may actually be a sub-domain name, such as 'ch.kolabsys.com'.

When you set a system's FQDN it is important that it exists of at least these three parts divided by a dot (.) character. Why?

A domain name (for example, 'example.org') is used for three purposes: 1) its IN A is probably pointed at a webserver, so that people can navigate to http://example.org equally well as http://www.example.org, 2) its IN A is used as a fallback recipient SMTP server address should no IN MX records exist, 3) it contains the SOA and NS records (authority statements, signatures, and other things).

Because of 1), the server to which the IN A for the domain name is pointing, it is unlikely (for larger deployments) that this is also the server on which the groupware deployment runs. Furthermore, it is best practice to set your MX records, possibly to the same system, but MX records nonetheless, so that you may have a level of redundancy (by supplying more than one MX record).

That said, the Kolab Groupware component that is the 389 Directory server refers to itself using the configured system's FQDN. While this can work should the IN A for "example.org" never change, it is more problematic should the system only be using 'localhost' or 'localhost.localdomain' - your 389 graphical console won't be able to connect to the administrator server with that set.

Furthermore, it is just bad, bad practice. A domain like 'example.org' is like a particular forest, and one of your systems is one of the trees. Unless you had a very specific deployment in mind, I'm sure you'll want to be able to refer to a single tree rather than only being able to refer to the entire forest as a whole.

That said, this is not what setup-kolab is developed to handle either. It'll chop off the first part of the "FQDN", so using 'example.org' will lead the setup to conclude you're going to want to set up for domain 'org'. It'll use that domain to build you a standard root dn as well, which in this example case would become 'org'. Settings that relate to login realms, and other sorts of stuff all over the place will result in unexpected behaviour, not to say fail.

In our web client based on Roundcube, we have had a need to get very large LDAP directories to function as an address book. While "large" is a relatively subjective term, in terms of LDAP "large" simply means running out of time, or meeting the look-through or size limits. Additionally, of course, PHP itself is subject to memory size and maximum execution time restrictions.

We weren't the first to run into these problems, of course, but with a little help we came up with a very, very good solution. Using Virtual List View (VLV) control and Server-Side Sort (SSS) control, browsing very large address books page-by-page takes maybe a tenth of a second per page. Roundcube address book problem solved (thanks Thomas, thanks Robert!)

But are they complex?

VLV and SSS controls are both very complex (the former more than the latter), with configuration on the LDAP server required and client-side BER encoding magic involved. I am not a programmer, let alone traditionally educated, so to me the PHP code (or the code's capabilities, rather) resulting from this effort is simply awesome. To properly function, it requires a patch to PHP to be applied, which I've now submitted for upstream inclusion. Kolab Groupware ships its own PHP packages solely to include this patch. There's many other gotchas to take into account as well.

Gotcha!

To use server-side sorting, you have to create an index on the LDAP server with a single attribute containing a list of attribute names using which entries can be sorted. When using LDAP from a client, you have to specify that exact same sort order.

To use Virtual List View controls, you require a successful hit on the server-side sorting (as that is the index). You also have to configure the LDAP server with a search (for which it requires a base, filter and scope). When using LDAP from a client, in addition to hitting the server-side sorting, you have to hit those three exact same search parameters as well.

That may not sound so difficult...

Complications with the User Interface Context

However, an LDAP client like Roundcube lists and/or searches differently based on the context.

One context is the address book. Typically one would browse an addtress book sorted by "Surname, Givenname" or "Givenname Surname". These one or two attributes (displayName can be put to very good use here) would then be included in the server side sorting. But then again, who sensibly browses a very large address book page by page? A 1000 entries with a page size of 40 results in 25 pages already - address book sizes I speak of when I say "large" are literally hundreds of thousands of entries.

So the keyword is searching - but searching is a complex realm in and by itself. If you were allowed to search only on the attributes included in a SSS and VLV, searching is virtually useless (all puns intended). Searching would basically allow you to list VLV entries starting with the first result your search had found.

You would like to search by location, phone number, department, organization, email address and other attributes. This is difficult to do efficiently, but can be done. The show-case is in what I'm sort of announcing in this blog post.

After your search results are in, you would still like them sorted somehow. If you would like to know who lives in Narnia and search for it, your results should not look like so:

1. Narnia
2. Narnia

but like so (for example):

1. Doe, Jane
2. Doe, John

A second context is auto-complete. In it's very nature, auto-completion is always a search (and not simply a paginated list of all entries), similar to the search for people living in Narnia. You can imagine how a set of search results could need to be paginated if there were many people living in Narnia.

Furthermore, in the user interface context where you use auto-complete (the compose window, the ACL entry management on folders), no address book interface pop-up is available, as only a small list with hits is displayed. The pages are much smaller, increasing the change the result set needs pagination.

Naturally, auto-completion searches for a limited set of attributes - but also most likely different attributes than an address book listing. Most likely, a search for the purpose of auto-completion looks at a display name ("Doe, J. (Engineer)", perhaps), a given name ("Jane", "John"), a surname ("Doe"), any email address (mail, alias, mailalternateaddress, mailforwardingaddress attributes), etc.

Imagine these searches would occur without the help of pre-sorted indexes. A million-and-one LDAP entries may need to be searched, all results would need to be obtained, the results would then need to be sorted, to then return only the first 10-15 entries. Wouldn't it be great if you could search, sort and paginate with the help of pre-sorted indexes?

But... I did mention earlier, the three magic search parameters need to match the server-side VLV search configured. As it turns out you can actually specify additional search parameters, but only in a very specific way. More awesome stuff I'm announcing in this blog post.

And all of this is only in Roundcube?

Within the Kolab Groupware realm, we have more components that hook into LDAP and perform listings and searches. Two of them are the Kolab Web Administration Panel (or actually, its API) and Syncroton.

So I've had multiple agenda's to attend to; 1) avoid needing to do actual programming for the Web Administration Panel, 2) avoid requiring Roundcube libraries with both the WAP and Syncroton and 3) further enhance our LDAP capabilities for yet even more awesome stuff.

I give to you Net_LDAP3 - not yet a PHP PEAR module, because it's a work in progress, and I don't like their license requirements and I don't like their function naming conventions (camel-case, basically).

My target is to integrate all of the LDAP functionallity required with Roundcube, our Kolab plugins for Roundcube, the WAP and Syncroton.

I've already achieved the following milestones:

• Unless specifically configured, or specifically disabled, or not a supported control on the LDAP server, the Net_LDAP3 module automatically discovers available settings for VLV/SSS listings (method list_entries())
• A method search_entries() with an argument for additional search parameters that automatically uses VLV/SSS configured or discovered but with the additional filter settings,
• Validation of the desired sorting,
• Automatic selection of one sorting configuration out of available sorting configurations,
• Allow a method login() to find a user entry before binding as the user being logged in,
• The execution of a registered hook to obtain or set configuration items using a routine external to Net_LDAP3,
• The execution of a registered hook to use for logging,
• Trace and debug levels

Obviously there's still many things on my TODO list, which ends with patching Roundcube, the WAP and Syncroton to use this module.

Last week I was surprised to receive an email from Shane Coughlan inviting me to become a Fellow of the OpenForum Academy. The current Fellowship has some of my personal heroes in there and so it was exciting and humbling to receive such an offer.

From their website:

OpenForum Academy is a think tank with a broad aim to examine the paradigm shift towards openness in computing that is currently underway, and to explore how this trend is changing the role of computing in society.

and:

OpenForum Academy is an independent programme established by OpenForum Europe. It has created a link with academia in order to provide new input and insight into the key issues which impact the openness of the IT market. Central to the operation of OpenForum Academy are the Fellows, each selected as individual contributors to the work of OFA. A number of academic organisations have agreed to work with OFA, working both with the Fellows and within a network of contributors in support of developing research initiatives.

During Akademy last week I was very impressed by the keynote presentation given by Will Schroeder of Kitware. At the core of his talk was a concern that research should be open by default, but it isn’t: peer review is a black box and publishers charge a fortune for access to the final paper. Over dinner we spoke at length on this matter.

I am lucky in that my career is no-longer tied to my publication record. This allows me the freedom to do my research when and where I want. Typically this means I treat all my research as “work in progress”. As and when I make little steps forward, I publish what I have done in my blog and gather feedback.

I have now taken the decision that whenever I have a complete piece of work I will publish:

• in an open-access journal;
• my data and the method I used to gather it;
• any tools I used to process that data.

GIMP does not drive me to the airport if my flight is before 10am.

So recently Jeroen van Meeuwen asked me to take a look at Cyrus IMAP. He had been involved in their switch from CVS to GIT and was curious to see what the results looked like. Let’s start with the usual green blobs:

Cyrus IMAP: Full History in Green Blobs (Click to Enlarge)

So, since I do not know precisely when the switch from CVS to GIT was made, I’m using Jeroen’s start date in the project (2010-09) as a rough guideline. Looking at the green blobs it is pretty clear that something happened after he joined the project. But let’s start by looking at what was going on before he joined.

Between 1993-07 and 2010-09 there were 25 accounts in CVS. Note: accounts and not contributors; clearly some of these accounts belong to the same person. For much of these first 7 years the project is also displaying what I refer to as “token-based development”; that is, in many weeks there is only one contributor (as if you had to hold a shared token to commit). I first noticed this phenomenon when studying Evince during my PhD and I have seen it only a couple of times since. I wish I could explain it.

Now, since 2010-09 we can see that 27 new accounts have contributed to the project; most are only around for one week (if we look deeper, I bet for only one or two commits) never to be seen again. Perhaps one of the effects of switching to GIT is that it is simply a lot easier for people to contribute? No brainer.

But I think there is slightly more at play here. How did Cyrus IMAP manage to get to its 17th anniversary and then basically double the size of its developer community just because of a switch to GIT? A project of such importance surely must have been attracting more folk before the switch to GIT… It is not like activity has increased significantly since the switch, right? Right?

Let’s take a look at some simple measures… Commit and Committers per month:

In the days after 2010-09 we can clearly see that commit rate (commits/day) has increased and so has the effort density (there is more activity on more days). But we are not seeing any significant increase in the number of contributors per day. So commit rate is up (not surprising, GIT encourages this and it is almost always exposed except where pushes are squashed) and committer rate is about the same.

Hang on! What about all those new folk we saw appear since the switch to GIT? Well, a typical week still only sees one or two contributors and these new contributors are mostly only hanging around for a few pushes (so they easily account for the days where the commuter rate increases to 2 or 3).

Here is what I think is going on… The switch from CVS to GIT has almost certainly made it easier for people to join and take part in the Cyrus IMAP community (even if only briefly). This accounts for some of the increased participation of new contributors. However, I suspect Cyrus IMAP in its first 17 years had way more contributors that the 25 we can see. My guess is that plenty people were submitting patches and that one of the 25 were applying them, thus losing the identity of the patch submitted in the CVS log.

So whilst switching to GIT has probably caused some growth, I think it is not as much growth as it may seem because the old CVS data hides the true scale of the community.

So what does the Cyrus IMAP community look like these days? This graph shows which contributors worked together (i.e. committed in at least one file together) during October 2010, the month after the switch. Everyone say “Hi” to the Cyrus IMAP community:

As the primary data store for Kolab data, Cyrus IMAP is a project close to my heart (but one that I really play no direct role in). If you would like to learn more and take part in the community check them out at their website or join in the discussion at #cyrus on Freenode.