Armin's personal Blog

We have released libinfinity 0.4.0 and Gobby 0.4.93, the latest version of our collaborative text editor. If you ask me, the killer feature in this release is with the user list - see for yourself:

Before: After:

However, there have been other major changes as well, including:

• Chat: The chat known from the stable Gobby 0.4.x series has been reintroduced. It now also features a backlog.
• HTML Export: Documents can be exported to HTML, keeping both user colors and syntax highlighting.
• Color Reset: User Colors in a document can be reset via Shift+Ctrl+C, or via the menu. This allows to get an overview of most recent changes when you want the user colors to be an indicator of activity, rather than of history. Colors are reset locally on the client-side only.
• Password Protection: infinoted 0.4.0, the dedicated server, allows to set a simple global password for the server. More sophisticated authentication mechanisms are planned for the next release, libinfinity 0.5 and Gobby 0.4.94.
• UI Enhancements: The user interface is nicer at many points: The browser list is now sorted alphabetically, the file chooser allows to open multiple files at once, the direct connection entry has a history to easily connect to hosts to which one connected before already, on a failed connection one can easily retry the connection via double clicking the host in the browser list, user colors adapt to the background color of the text view (read: text is readable with dark themes) and several more enhancements and bug fixes.

The very rough plan for the future is to add authentication support in 0.4.94, and displaying carets and selections of remote users (all protocol code is already there, all that is left to do is to draw them into the TextView) and self-hosting for 0.4.95 and 0.4.96.

The downside of this release is that it breaks compatibility with earlier releases, especially libinfinity 0.3.0/Gobby 0.4.92. I am sorry; I know this is annoying, but maintaining backwards compatibility is as well, and my excuse is that Gobby 0.4.9x is still experimental. It allows me to concentrate more on the interesting stuff. However, if it is not too much trouble then I will try to keep compatibility for the next release. For Windows users it should be very easy to upgrade since an all-in-one installer is provided. Philipp Kern promised to get it soon into Debian and Ubuntu, so I hope the compatibility break will not cause too much trouble. Our public server on gobby.0x539.de runs 0.4.0 already. None of us is using Mac OS X (anymore), so we can't provide binaries for it, unfortunately. If anyone creates a nice bundle, we would be happy to distribute it on our website.

I also want to express my thanks to Gabríel A. Pétursson and Benjamin Herr who made significant contributions to this release. Beginning from January, Benjamin Herr will be paid by the KIT's department of computer science to implement SVN integration and LDAP authentication, so these are likely to be available in one of the future releases as well.

In a related note, we have also launched http://infinote.org, a Wiki to host information on the infinote protocol used by Gobby via libinfinity's implementation. I plan to migrate our protocol site to it, and to add more detailed and more structured documentation there. Also, Kobby, Gobby's KDE counterpart, has had another release recently. It still uses libinfinity 0.3.0, but I expect that to change rather soon. Simon Veith has implemented the adOPTed algorithm and the transformation functions as used by infinote in Javascript, in the scope of the jinfinote project. It passes all of libinfinity's tests, and he is now working on a web-based editor which can talk to Gobby and Kobby. An experimental demo is available already, however it does not interfere with Gobby yet.

In another related note, the Gobby IRC channel moved to #infinote on freenode.org a few weeks ago. Actually we wanted #gobby which was already registered earlier though, however #gobby was registered earlier already even though it is unused nowadays. We try to get through the FreeNode bureaucracy to get the channel dropped, but it seems that this may take several months, if not years.

So I arrived back in Germany on Sunday morning. I had a great time in the US where I finished two projects at Duke: I investigated Super-K's capabilities to detect supernovae (with special regard to correlation with a Gravitational Wave signal), and I investigated whether and under which circumstances a supernova location can be determined from neutrino oscillation in Earth matter. There is a report for the former and a paper (my first \o/) for the second. I learned about and used the ROOT data analysis toolkit for these. It's very powerful, but also very ugly API-wise.

I also travelled a bit. Major visits include the Great Smokies, Kitty Hawk at the Outer Banks, Raleigh, Norfolk, Washington D.C and Pleasure Island/Wilmington. Until I find a good place to put the photos on (I tried flickr, though I probably need to resize the pictures and make a selection to get things up there in one month or two because of the 100MB/month limit, and I'm also not sure how this 200 Pictures/Photostream limitation actually works) you can also have a look at some albums of Ruggero Golini who, amogst others, accompanied me on some of the trips.

Looking forward, my seventh semester at the KIT (formerly Universität Karlsruhe) has begun. I take courses on statistics and data analysis, experimental particle physics and cosmology. Luckily there was nothing on Monday so I could recover from jetlag. Being in the US also meant I was not able to take any exams at that time but I'm still pretty confident I can take all three remaining exams (Experimental physics, Theoretical physics and another more specialized physical major) during the next 10 or so months so I can start with my diploma thesis in Fall 2010.

I am at Duke for more than a month now. Things turn out to be fun. My work still involves a lot of programming, mostly even using free software. The difference, however, is what it is for. This is not about making life more convenient for computer users, but about creating new knowledge. This is a huge source of motivation for me. I think the last time I had a similar commitment was when I was figuring out how to do Undo/Redo properly in Gobby. In contrast, I just can't get into "man-made" problems such as economy or law; this somehow feels articifal to me. The actual project I am working on is still secret, though. I hope I can say more about this in a month or two.

My impressions of America, or at least the part of it that I have seen so far, are somewhat twofold. On the one hand there are inconveniencies such as prices always shown without tax, no timetables at most bus stops (interesting problem here btw - given a frequency f a bus runs a certain route and a time interval Δt you save by taking the bus instead of walking, how long do you wait at the bus stop before starting to walk when you don't know the exact time the bus arrives?), getting charged for incoming calls and text messages and food being more expensive than in Europe generally. On the other hand, however, I like the way of living of the people here (In Germany, I have yet to be asked whether I need a ride when waiting at a bus stop), and I like the landscape very much, including the huge Duke campus, and how green everything is when you look around. This comes with the price that without a car you are pretty much screwed if you want to go anywhere. Public transportation is available, but not until late in the night, and further limited on Sundays. Fortunately, car rental is somewhat affordable when you are enough people to split the costs. As a non-alcoholic I could also get used to the fact that there is mostly no alcohol in public places, and that other drinks are much cheaper than beer or whatever.

Despite all, I have not given up on other software development. I have not done too much on Libinfinity/Gobby recently (it's a bit sad that I am still the only person working on it, despite many offers to help on the mailing list - but all these people disappeared soon afterwards. Maybe I should try Jono's book to find out whether I am doing something fundamentally wrong in that regard...). However, I worked a bit more than usual on Clonk recently (which became FLOSS a while ago btw, as the OpenClonk project). I spent some time on realtime 3D model rendering. This allows much smoother animations, and transitions inbetwen them (bone system is already implemented, but only in software for now). Support is far from complete, but it is usable already (see the mesh branch in the repos).

I have left Openismus at the End of June as I will be doing a summer internship at Duke during the next three months for which I have received a DAAD scholarship. I will be heading to the States on July 13, and I'm not going to attend GCDS therefore (even though it wouldn't overlap, I don't feel comfortable with such few time inbetween. Plus I would miss even more University stuff here at Karlsruhe).

At Duke, I will be working with the HEP Neutrino group (supervised by Kate Scholberg) by taking part in simulation studies for next-generation Water Cherenkov neutrino detectors, with special regard to Supernova neutrinos. The goal is to choose detector design parameters so that we gain a maximum of information from the detector, for example about Neutrino oscillation. This will be the first time that I will actually take part in modern research, and actually be applying (some of) the stuff I have learned at University. After having finished the internship, I also hope to have gained insights into whether I want to write my Diploma thesis (starting in a bit more than a year, if everything goes well) in the field of theoretical or experimental (Particle) Physics.

I would like to thank everyone at Openismus. Earning money for working on Open Source Software is a great thing. This helped me a lot to afford my studies. At the end of october, there is a scholarship holder meeting in Berlin. I hope I can visit the Openismus Office then.

A small Openismus task I recently carried out was to investigate whether there is a generic timeline widget for GTK+ out there, for example to show photos associated with the day they were taken. The main feature this is supposed to have is that the items it contains are grouped by time periods, such as day, week or month, and that it should be possible to view multiple periods at once, or to allow easy browsing between them.

The closest thing I found is Gnome Zeitgeist, an application which takes up Federico's Journal Idea. Gnome Zeitgeist is still under heavy development (I also had a short look at it yesterday, and when I updated the branch today it looked totally different), but the timeline widget looks promising already. It shows recently used files ordered by the day they have been used. By default, it shows three days, but when filtering by tags, it can also show more (scrolling horizontally if necessary), omitting days with no entries at all. This is implemented using multiple GtkTreeViews, therefore I guess it would be easy to generalize the widget to show arbitrary data by allowing to pack own cell renderers into it.

Then, there is Nemo, a program which aims at making file management easier by showing all documents (not only recently used ones) in a calendar type of view. There are views for a day, a week, a month or a year. However, when there are more items than there is space in a calendar cell, then it simply shows "+ 309 more". Gnome Zeitgeist allows scrolling instead, which is more handy when for example previewing photos.

Other projects I had a look at include Gnome-Shell (which shows recently used icons ordered, but not grouped), Wizbit (which has a timeline widget for the revision graph, but Wizbit has slightly different usecases), Paperbox and tracker-search-tool (both of which don't group items by time periods).

The Gnome Zeitgeist widget looks really promising, though. Although it is not a stand-alone widget, I don't think it would be too much work to separate it and make it more generic.

Are there more solutions to this kind of problem in the GTK+ world? Any application I did not have a look at, although I should in this context? If so, please tell me in the comments.

Over the last few months I implemented SQLite support in Glom for Openismus. When creating a new document, Glom now allows to save the data in a SQLite database instead of a postgresql database. This is more light-weight, and there are generally fewer things that can go wrong as everything is performed within the same process. However, there are some obvious limitations: As SQLite does not support authentication, Glom documents using an SQLite database can be opened without access restriction either. Also, the Glom session is not announced on the network since the SQLite database can not be accessed from remote anyway.

When implementing this, I tried to get rid of most of the postgresql-specific code which was in quite a few places throughout Glom, and putting it into a single place, and using libgda instead of hand-written SQL when possible. This should make it relatively easy to add more database backends in the future, if people need any. There is only some code dealing with users and privileges that may be postgres-specific and which I didn't touch, simply because it is not used with SQLite anyway.

One of the interesting parts was to map the different glom data types to SQLite data types. The switch to libgda-4 helped with this, since libgda-4 allows storing dates and timestamps in SQLite databases, by converting them to standardized strings. However, SQLite does not support a "numeric" type, but only integer and real. Currently, Glom maps its numeric type always to an SQLite real, which worked surprisingly well so far, though I'm still not 100% happy with it, mostly due to possible floating point inaccuracies.

Another problem was changing a table, meaning adding, modifying or removing columns, or changing the primary key column. As SQLite only supports adding non-primary key columns, but not changing existing columns or removing columns, Glom recreates the whole table in that case and moves the data from the old table into the new one. The tricky part of this was to get all possible type conversions right. For example, when changing the type from Image to something else, then the conversion failed with a "Database table is locked" error only if at least one row contained an actual (non-NULL) image. It took me some time to find out that SQLite does not allow dropping a table when a recordset from a SELECT query is still in use, and that Glom was indeed still holding such a recordset to display records from the current table. The problem went away when I released that recordset before changing the field type, though I wonder why there was no problem when there were no images in that column, or for other field type changes not involving images.

This work also yielded some bugs in the upcoming libgda-4, but Vivien has been very responsive so that they have been fixed very quickly. Kudos!

The SQLite version feels much faster and more responsive than the postgresql one when creating a new database or populating the list view. I wonder whether this can be explained with the overhead that comes with postgresql (talking to another process via TCP/IP) or there is a bottleneck elsewhere, which, if fixed, would make (postgresql) glom more enjoyable to use.

The current tarball version, 1.9.1, already supports SQLite, but still has some problems with changing field types which have been fixed in SVN. A 1.9.2 will hopefully be released soon. The next stable version of Glom, 1.10, will also contain SQLite support.

Sorry, no screenshots this time, since there wouldn't be anything to see that would be different from a postgresql-based Glom, except maybe an option to create a SQLite database when creating a new Glom document.

Look at this screenshot of the recently released Gobby 0.4.90. See that nice yellow arrow in the toolbar? Yes, this means we finally have Undo/Redo support. It was a really good feeling having closed bug #39

Gobby 0.4.90 is the first release in the 0.5 series, and 99% of the code have been rewritten from scratch. It is not yet 0.5.0 since it does not yet contain all the features that previous versions had (for example, it does not yet support self-hosting or password protection). However, removing the integrated chat was a design decision, since there are so many other possibilities for communication, including VoIP. As it supports removing documents, only use it with people you trust enough not to delete all your data.

Gobby 0.4.90 does no longer use obby or net6 as backend libraries doing the hard work, but libinfinity, implementing the Infinote protocol. libinfinity is not yet stable API and ABI wise, and probably will not be for still some time. libinfinity contains a dedicated server, called infinoted that Gobby 0.4.90 can connect to.

You can safely install the new version since it is parallel-installable to previous Gobby versions. But please consider everything as very experimental. Things may crash or otherwise behave badly. Report bugs if they do. We set up a playground server on dalaran.0x539.de if you want to test the new version without having to install server yourself.

I was curious where most time in libinfinity, Gobby's infinote implementation, is spent. Not because it was too slow or something, but just because I wanted to know. So I used valgrind's callgrind tool to gain some profile data while running libinfinity's main test for the concurrncy control algorithm. When I displayed the result in kcachegrind (there is still no GNOME equivalent, no?) everything looked as I would have expected, until I noticed this line:

This means that 14.5% of the overall program time is spent in g_type_instance_get_private(). This is because libinfinity uses g_type_class_add_private for every class to store its members into, so it is easy to extend functionality later without breaking ABI. However, this also means that for every function that needs to access a member, g_type_instance_get_private() is called. I didn't expect such a huge impact, though.

To fix this, I added a simple gpointer to the public instance struct, and let it point to the private field in the instance_init function. So this means one call to g_type_instance_get_private() per instance, instead of roughly one per call to a function operating on an instance of the class in question. After doing this for the five classes suffering most from this, g_type_instance_get_private does no longer show up anywhere near relevant in kcachegrind.

I'd like it if improving performance was always as easy as this.

With the release of gtkmm 2.14 I created new installers for Microsoft Windows; the last ones were only for 2.10. We also wrote down instructions on how to use the installers, and how to recreate them (in case anybody feels the need...). They now contain everything needed for gtkmm to run on Windows, including GTK+ itself (from the GTK+ all-in-one bundle).

I used the NSIS installer script from Cédric Gustin, the previous gtkmm Windows maintainer. It is now in gtkmm SVN. Comparing with InnoSetup (which I used for Glom's Windows installer) I must say that InnoSetup scripts are easier to write and maintain (especially for a few common tasks such as creating StartMenu icons or creating an uninstaller). The gtkmm NSIS script has around 1700 lines while the Glom InnoSetup script has only around 200. I know that the gtkmm installer is more complex, but that's still a huge difference.

Gregory Haynes is working on Kobby featuring collaborative text editing in your favourite KDE text editor as a GSoC project. He started to wrap the (still unreleased!) libinfinity (my infinote implementation) API for C++ for this task. I'm really excited to see other collaborative text editors coming up that work together with Gobby, since this is exactly one of the reasons why I took the burden to write the library in GObject-based C.

Speaking of which; I didn't blog on it for a long time. We have moved to git in the meantime. I'm currently working on porting Gobby to use libinfinity instead of obby, heading towards a first usable release, so that the hard work pays off. There are still some things in libinfinity that need to be fixed (performance issues, a few known bugs and way more testing) which are probably easier to fix when a usable Gobby version using it exists. I hope to have these done in a few months, though the most difficult part is always to get the little details right, which is probably why I'm not good at meeting (self-set) deadlines.

I updated the installer. It now includes jpeg62.dll and python25.dll which were missing (thanks to the reporters on my initial post). I also included the glom and gda python modules. If I do understand things right, then installing Python and pygtk makes Python support in Glom automatically work with this. I tried this out by reinstalling Python into a different directory, which worked.

Again, if there are still problems with it, please tell me.

Here it is, Glom's installer for Windows. Thanks to the opinions on my last blog entry. I set up an all-in-one installer, so Glom should run out of the box after installing (please tell me if not). It includes GTK+ from SVN with my patch from bug #506062 (hint, hint) so that recent files filtering works.

Since most people seem to use NSIS for Windows installers (comment on this blog entry, gtk-win.sourceforge.net) I also had a look at it. However, the scripting language reminds me more of assembly than anything else, and I think these days are over. So I used InnoSetup which has Pascal scripting for the rare cases you need it.

The only thing that is not going to work is python scripts for buttons and calculated fields. Ideally, one would just install Python and PyGtk, and having Glom use that installation automatically when present. This shouldn't be too hard, it just needs to tell python where to find the own gda and glom python modules. I am going to tackle this after my physics exam on Thursday.

With added support for self-hosting, Glom for Windows now supports all the major features that the Linux version does (apart from Service Publishing, which is probably way more effort since Avahi is not available on Windows). It's time to think about how to package it, and what dependencies to include, especially whether to ship with GTK+ or not.

All the major GTK+-using projects such as Inkscape, Pidgin and Wireshark include GTK+. Even the GIMP does so since version 2.4. This actually generates a lot of duplication, which is why we (or, rather, I) did decide not to ship it with Gobby. Of course we are getting complaints such as "Hey, this doesn't work because libgtk-win32-2-0-0.dll was not found!!1" from time to time, but if people are not able to read the instructions right above the download link, well, then I'm sorry. Do others think the added duplication is worth that the user needs to install GTK+ manually once? Or, maybe, is this just because there is no "official" GTK+ installer?

Then, there is python, and I have not so much experience here. Glom links statically against libpython, so it should even run without python being installed. However, buttons and calculated fields won't work then, unless we ship with at least pyGTK. Perhaps the best thing is to state that for button scripts and calculated fields to work, python and pyGTK have to be installed. We just have to make sure that Python finds the glom and pygda modules then, even if Python was installed after Glom (so it had no chance to install them into the standard location for such modules), but probably Python has some API for this.

Last but not least, we have gtkmm which also has an installer that is hosted on ftp.gnome.org even. I think we should either ship both GTK+ and gtkmm, or none of them.

If you have any thoughts on this, recommendations and suggestions are welcome.