1. Setup the UI, start listening in socket.
2. Connect objects to react to: user manipulating the UI, connection happening in socket.
3. Calls app.exec().

That’s it. Then it’s job of app to act like a dispatcher, waiting for external world events to happen and propagating them to the application objects. Qt uses QEvent to propagate those events.

Well, this is common in other frameworks. In some of them the word mainloop or eventloop is used to describe the job that app in Qt is doing. For example: EFL has the ecore mainloop, glib has a mainloop and Node.js has this concept built in as well.

In this article I’ll focus mostly on Qt machinery to work with mainloops.

Underneath app

One interesting aspect of the Qt implementation is that the concept of mainloop is abstracted, so other parts of Qt don’t heavily depend on how the mainloop is implemented. This supports Qt’s goal of being crossplatform. So the real class that implements a mainloop in Qt is not the QApplication, but another platform dependent class that implements the interface defined by QAbstractEventDispatcher.

This abstract class is the minimum interface that QApplication needs to provide the event-driven programming for its users. This allows different implementations for each platform, and even multiple implementations on the same platform.

In Unix, for example, besides the default dispatcher implemented using POSIX API, you can use an event dispatcher implemented on top of glib mainloop. And this gives you something very nice: your program can make use of libraries that rely on the glib mainloop as well. This kind of integration has been done in other libraries as well, for example with ecore on top of glib.

When you instantiate QApplication, Qt will automatically create a proper event dispatcher for the platform and use it. However if you instantiate one before, QApplication will use it. But before we play with that, let’s look at event dispatchers a little bit more.

What can an event dispatcher do?

Well, if we look at QAbstractEventDispatcher, we’ll see the following groups of methods:

• Timer: to register/unregister timers, so the mainloop will send a QTimerEvent to an object after certain time.
• Socket notifier: in Unix-world this would be registering/unregistering file descriptors to watch. The mainloop will send a QSockAct type of event to the socket notifier object.
• Control: to run one iteration of the mainloop, to enable another thread wake the mainloop if it is idle waiting for new external events.

If you’re familiar with the select() function from POSIX, you can imagine how this functionality can be implemented. Running an iteration of the mainloop would be the select() call for example. Actually, that’s the way the default unix event dispatcher works in Qt.

With such an interface, it’s easy to see that Qt could run on top of most mainloops, and for this article I wrote a simple QEventDispatcherEcore, which implements this interface on top of EFL’s ecore mainloop.

Show me the code!

The code is in my playground repository, and the important file for now is the dispatcher.cpp. You’ll see that it is all marked with ### comments, which indicates points of the code that are incomplete: this is just a simple example of the concept, be warned!

I won’t go step by step in the code, because it should be not so hard to understand. The Ecore library gives us enough functionality to implement all Qt needs. I’ll highlight two pieces that I think deserve special explanation

First, how the wakeUp() call was implemented: since ecore mainloop didn’t have a similar functionality, I used the same trick as the Unix event dispatcher. The trick is to create a pipe and watch one of the ends inside the mainloop. Then, to wake up we simply write on the other end of the pipe.

A second important point is: the processEvents() function takes a flags argument, which indicates to the event dispatcher important parameters about the iteration it is going to run: whether it should ignore UI events or whether it should block waiting for the next event if there’s no event right now.

The block until event show up is a very important functionality, since we don’t want to spend all CPU cycles calling processEvents() again and again with no side effect, no new event being delivered, and avoid busy loop.

In the code for the event dispatcher example, what’s tricky is that while Ecore does have a iterate() primitive it doesn’t have a iterate_and_wait(). The rationale is that part of the work that Qt is doing here inside QApplication code

    // Simplified version of what QApplication do
    while (!quitLoop)
        processEvents(flagToWait);

is something that usually libraries encapsulate for the user. In our case has a mainloop_begin() function to encapsulate the while-loop.

To the event dispatcher example, what I did was to simulate the functionality I wanted by making use of Ecore’s idle_exiter mechanism, which allow functions to be registered to be called after a iteration the mainloop

    // Simplified version of what Ecore do...
    while (!quitLoop) {
        // ...
        processEvents(flagToWait);
        // ...
        runIdleExiters();
    }

The trick was to register an idle_exiter that caused the mainloop to quit. After inspecting Ecore implementation, the result of doing this be exactly what I wanted to achieve with an iterate_and_wait() primitive without changing the API.

Hmmm, what’s the guidispatcher?

If you followed the repository link, you’ll see there’s another dispatcher there in the guidispatcher.cpp file, that’s the QEventDispatcherEcoreGUI. To understand it we need to remember the difference between a QCoreApplication and a QApplication: the former doesn’t depend on GUI code, and the latter is the subclass with GUI functionality.

This separation also happens in event dispatchers. Using the case of Unix again, in Qt the GUI for Unix is based on X11. So a QApplication needs to create a X11 connection and watch for events, translate them to QEvents and send them to the proper objects.

This job is exactly what an event dispatcher is for. So in Unix, QCoreApplication uses the Unix’ select() implementation we mentioned before, while QApplication uses a subclass of it, which also handles X11 events.

Letting the UI event handling inside the event dispatcher allows the existence of a flag to skip handling of UI events.

Back on our example. This new dispatcher is GUI-enabled, it watches for X11 events and translates them to QEvents for delivery with help of the function QApplication::x11ProcessEvent(). Note that since this is just a proof-of-concept, we are not checking for all the flags passed.

Inside the directory ‘qapplication’, there is one example using this dispatcher. It mixes a QWidget on the screen (using QApplication for the GUI) and an EFL window showing Evas in other. They are in the same process but with different X connections.

Last but not least: the QEventLoop

If you look the Qt documentation, you can see there’s a QEventLoop class. But, does this class do if the mainloop is already in the event dispatcher?

Simply put: QEventLoop is a helper class that provides convenient access to the dispatcher’s processEvents() call. For example, QEventLoop has an exec() method, that looks like this

    // Simplified version of QEventLoop::exec()
    while (!eventLoopQuit)
        dispatcher->processEvents();

One usage of this pattern is to make a function blocks but still running the mainloop (so things like UI still work). This enable APIs like the ones used in some QDialogs

    QMessageBox msgBox;
    msgBox.setText("The document has been modified.");
    msgBox.setInformativeText("Do you want to save your changes?");
    msgBox.setStandardButtons(QMessageBox::Save | QMessageBox::Discard | QMessageBox::Cancel);
    msgBox.setDefaultButton(QMessageBox::Save);

    // Blocks until a button is clicked in the dialog, ret will contain information
    // to identify that button.
    int ret = msgBox.exec();

To see some downsides of this kind of API in dialogs, I recommend reading the article Unpredictable exec() from Qt Labs blogs.

Further reading

For those who are interested in digging further, there are many ways to go from here:

• Event-driven programming: there are plenty of resources on the net, but I would like to point out two:
  • A talk about Node.js (pdf and video). Node.js provides an environment for server-side programming in JavaScript based on events. The talk particularly highlights the advantages of using event-driven programming.
  • A talk from the Plan 9 conference in 2007 called Threads without Locks. The author shows the relation between usage of threads and event loops and how they can be used together to avoid locks for syncronization. These ideas appear also in Go programming language.
• QThreads: I didn’t mentioned threads, but in Qt, if your thread runs a loop, it’ll create a new event dispatcher. As of today you still can’t use custom dispatchers for the QThreads, only for the main thread.
• Evas in Qt: the example showed Evas being used in a Qt application. While this was one way to use Evas in Qt, there are other possibilities. You could just use the Evas library directly with Qt, without Ecore/EcoreEvas. But in that case you’ll need to do the job of feeding Evas with events yourself.

That’s all folks.

What is even better is that, not only the author of the D-Bus bindings in Emacs, Michael Albinus, commented on the post, but also implemented functionality that I had to workaround before (thanks, Michael!). This means the code is smaller and the bindings are more complete.

In particular, the bindings now support

• registering properties, and the binding will automatically implement the org.freedesktop.DBus.Properties interface (i.e. the Get, GetAll and Set methods).
• unregistering objects now will release the service name if there are no more methods/properties registered for that service.

Also during that time, the folks working with the protocol, kept reviewing it and the name of protocol changed, now it’s called StatusNotifier.

The new code is available here and should work with recent development versions of Emacs.

By the way, it’s far from being a complete implementation of the protocol, but it’s a nice start and was an excellent way to learn both DBus and Emacs :-).

A few weeks ago we had to do some work related to D-Bus, the IPC solution from FreeDesktop. I took the chance to understand it better, reading a little bit about it and doing some experimenting.

There are some very useful tools to debug and experiment with D-Bus: I used a lot of dbus-send and qdbus (part of Qt library) to call methods of existing objects. Also used d-feet to explore the object interfaces. To monitor interactions in the bus, I used dbus-monitor.

However, one of the APIs we were interested in was the Notification Item, which came from KDE but it is now in the path to become a freedesktop blessed specification. It aims to replace the system tray mechanism based on XEmbed.

The Notification Item is a little bit more complex to play with because it was based on the interaction of different services. You have a daemon to track items and hosts, and hosts will ask items about their properties. So, it wasn’t a simple matter of sending messages, it was necessary to get a service up with certain properties to see it work, i.e. just dbus-send and other tools were not enough.

Even though Qt has very nice tools for binding and implementing D-Bus APIs (for the real work we used those), I was aiming for something more “dynamic”, in the sense that I could build a notification item part by part, and see the interactions.

And then I ended up writing a small, but non-trivial, example of D-Bus usage in Emacs, using the brand new bindings available in Emacs 23. It implements parts of the Notification Item. In a KDE desktop this means you can make an icon appear in the notification area on the panel and interact with Emacs through that icon.

The code is available here, I would be really glad to hear about different ways of doing things. In particular the usage of lexical-let.

The documentation of D-Bus bindings for Emacs is very nice, so I recommend it. Another take on understanding how D-Bus work is looking at KDE’s tutorials, they also explain how to use Qt D-Bus bindings in C++.

I will not go into a line by line detailed description of the code. Instead, I’ll comment on things that I think made the example non-trivial: the usage of lexical-let to create handler (callback) functions, defining D-Bus properties and unregistering a service created in Emacs.

Note that this post is not meant to downplay the Emacs bindings, but to point out my difficulties in a real use, as well as how I overcame them. It really rocked to have those bindings available, without them this post wouldn’t exist and would lose a chance to learn a lot of things about both D-Bus and Emacs Lisp.

The lexical-let

In Emacs, to react to methods being called on a certain object in a service, you have to use the function

(dbus-register-method BUS SERVICE PATH INTERFACE METHOD HANDLER)

so that when that method is called on your object in the given service, the function handler will be called.

In the Notification Item, one of such methods that items have to react to is Activate. In KDE, items are usually represented by icons in the panel, and the action of clicking triggers Activate. There is a SecondaryActivate, which in KDE is triggered by the middle click.

When coding the example, I made the choice that, instead of creating one Notification Item object for the entire session, it would be possible to create different items, with different handlers. And while each method has two integer parameters describing a position where the icon was activated, I wanted to have only one handler function with one parameter describing whether the activation was primary or secondary.

My make-notification-item function takes, among other things, the handler function I described. The code for actually registering the methods is (without the docstring)

(defun ni-register-activate-handlers (service activate)
  (lexical-let ((act activate))
    (list (dbus-register-method :session service "/NotificationItem"
                                "org.kde.NotificationItem" "Activate"
                                (lambda (x y) (funcall act t)))
          (dbus-register-method :session service "/NotificationItem"
                                "org.kde.NotificationItem" "SecondaryActivate"
                                (lambda (x y) (funcall act nil))))))

so I actually registered a lambda, to filter the discarded parameters and add the primary/secondary parameter. Note that this is going to call the function act, which is not defined in the lambda’s scope.

In many programming languages, what would happen is that the context where act is defined would be saved and attached to the lambda that is being passed around. This feature is what people call lexical binding (or lexical scoping), and the “lambda plus context” is called a closure.

But by default Emacs Lisp doesn’t support ‘closures’ but instead will look into (what I will call) the global context for the act variable. So in my example, I could just use a (setq 'act ...) to change what action would take place when some item was activated. This is called dynamic binding (or dynamic binding). Actually what happens is that when the handler is called, act is not defined, so we get an error.

The code works because of the lexical-let. What it does, roughly binding. Behind the scenes, it is a macro that creates a unique symbol for act in what I called the global context and uses it in the code. So it works in dynamic binding mode. If we somehow found out the name of that generated symbol, we could mess with it just like we could if we were not using lexical-let.

While dynamic scoping is less common, it has some benefits. For a more detailed discussion on that topic, see the Emacs Wiki page DynamicBindingVsLexicalBinding.

The lexical-let is also used when defining the handler functions
that respond to D-Bus queries on properties of the item object.

It seems to me that lexical-let was essential to allow multiple items to be created at the same time. This happens because of the way the bindings are done. The handler function is called without passing any information regarding what D-Bus service/object was called. And at register time we have no other way to pass additional information that would be sent as argument when calling the handler.

If either of those were different, I could have a dispatch table mapping objects to real handlers, or some arbitrary data to real handlers.

D-Bus Properties

While writing the example, I discovered that D-Bus itself has a very low-level mechanism of message passing (or method calling) and many other features are built on top of that.

One of these high level concepts are the D-Bus Properties. In practice the idea is very simple, and could have appeared in different APIs based on D-Bus (maybe that’s the reason they are there…). Instead of having a series of methods for getting data associated with an object, you could have only one, and use an identifier to know which data you are talking about.

For example, in Notification Item, items have among other stuff: a category, a title and a status. Instead of having three getter methods, what the specification of Notification Item does is say that an item have those three properties. Of course, there’s a well defined interface for properties (it is the org.freedesktop.DBus.Properties).

However these high level features are not always available in the bindings. As of the time of this post, the Emacs binding only support reading other objects properties, but not registering new ones. So if you want to do that, you have to implement by hand org.freedesktop.DBus.Properties for your object.

In my code, that’s implemented in my-dbus-register-properties and its auxiliary functions. And that’s another place I’ve used lexical-let, this time to create different handlers for the Get method (part of D-Bus properties interface)

(defun my-dbus-register-get (bus service object prophash)
  (lexical-let ((ph prophash))
    (dbus-register-method
     bus service object dbus-interface-properties "Get"
     (lambda (iname pname)
       (list `(:variant ,(gethash pname ph)))))))

in the code, I stored the properties of an item in a hash.

How to make a service disappear?

One last issue that I’ve tackled was the fact that the service continued to be visible to D-Bus clients after I unregistered all the methods.

After some research, I’ve found out that what really was happening was that when registering a method for a named service for the first time, the binding requested this name to the server by sending a message to the server object (org.freedesktop.DBus) this is inside the C code of the binding, and uses a wrapper function instead of explicitly building and sending a message.

Inspecting the binding code, I didn’t found the equivalent code for releasing this name in the server. So, even with no methods registered, the named still existed and pointed to the Emacs process.

This was an issue for me because the Notification Item API specifies that to unregister an item, you have to unregister (release) the associated name.

While the solution was simple enough

(defun my-dbus-release-name (service)
 (dbus-call-method
  :session dbus-service-dbus dbus-path-dbus dbus-interface-dbus
  "ReleaseName" service))

this is something I think could be part of the Emacs bindings. Either for doing this automatically (as the requesting names is), or at least to error check when someone is trying to release a name which still have methods registered.

Final thought

It was rewarding to spend time understanding both D-Bus and Elisp. The knowledge helped me understand the D-Bus APIs that we were working at the time, including the Notification Item.

I hope that someone could pick up where I left and try to actually implement a feature in Emacs taking advantage of Notification Item. One obvious feature would be some alert system for the IRC clients in Emacs.

At first I was using it just for playing at home, and kept using Vim at work, since my project was running at high speed and I was a little bit insecure — specially because I wanted to use a version from CVS that supported fancy Truetype fonts in the X11 environment. After some time, I got confident enough to use it full-time, bootstrapped my .emacs file with Barbieri’s .emacs plus minor modifications — funny thing is, for some reason I kept using Vim to edit my .emacs file for a few weeks.

Inside Emacs’ M-x info library, there is a nice document, also available on the web, Programming in Emacs Lisp. Nice because it explains Elisp but also how Emacs works at the same time — strongly recommended for those who are starting. Another good resource is Yegge’s Emergency Lisp, which is a good practical summary of the language.

During this process I kept talking to my friends about the new editor (we were all Vim users) and trying to push Emacs into them, too. It worked for at least one of them. He tried a bit, but missed some features from the editor he used to work, Textmate. We both immediatly concluded: hey, Emacs should be able to do everything that Textmate did, and more.

So he found out about yasnippet that allowed the creation of mini-templates that are triggered by short strings and then using TAB (yes, the guy has two blogs, don’t ask me why). I ended up trying to solve another problem, which is to automatically insert the closing parenthesis when you insert the opening (abstract this to use with other interesting pairs of characters). It was just using the existing skeleton-pair feature available in Emacs, with a few twists.

Show me the code

I’ll comment on my solution, but you can also grab the raw file here. By the way, I’ve used M-x htmlize-buffer to create colorized pieces of code.

;; Paren experiment
(setq skeleton-pair t)
(defvar my-skeleton-pair-alist
  '((?\) . ?\()
    (?\] . ?\[)
    (?} . ?{)
    (?" . ?")))

First we needed to activate skeleton-pair. It’s part of skeleton-mode. It allows you to type ( and get () with the point (also known as cursor) positioned inside the parens. Also allows you to selected a region and type (, and the region will be inside the parens. In this block we also create a helper association list, that we’ll be using later.

(defun my-skeleton-pair-end (arg)
  "Skip the char if it is an ending, otherwise insert it."
  (interactive "*p")
  (let ((char last-command-char))
    (if (and (assq char my-skeleton-pair-alist)
             (eq char (following-char)))
        (forward-char)
      (self-insert-command (prefix-numeric-value arg)))))

First real thing is here. We defined a function that accepts one argument. The interactive tells us how we fill the arguments when the function is called via some key or key combination interactively.

At this moment it’s nice to know that everything you type in Emacs will end up triggering some function. Even simple characters. When you type a, Emacs will look in a table and see what function it should call. Normal letters like a are bound to the function self-insert-command.

Back to the function: it’s body is very simple, if the character we typed (given by the global last-command-char) is one closing char present in our association list and the next character is the same, we just forward-char, otherwise, we add the character as usual. The idea is to bind this function to the closing chars like ) and }. That’s what we do next:

(dolist (pair my-skeleton-pair-alist)
  (global-set-key (char-to-string (first pair))
                  'my-skeleton-pair-end)
  ;; If the char for begin and end is the same,
  ;; use the original skeleton
  (global-set-key (char-to-string (rest pair))
                  'skeleton-pair-insert-maybe))

So when you type ( the function skeleton-pair-insert-maybe takes care, when you type ) our function will take care of the ending. This gives the first feature over pure skeleton-pair: when you type ( and then ) you’ll end up with the right thing () instead of ()) that the plain skeleton pair would give you.

After testing a bit, and showing to others, we realize: if I press ( and then backspace I would still have the ) hanging around. Solutions? Well, let’s rebind Backspace to be smart enough about that, right? Well, that’s what I did at first: do my thing and call the original bind for backspace backward-delete-char-untabify. It worked…

…except when some language mode rebinds backspace, like python-mode because it uses backspace also to walk thru the possible indentations in the code and we would break their binding. And guess what? Lately I was coding mostly in Python. After some research, a solution came up:

(defadvice backward-delete-char-untabify
  (before my-skeleton-backspace activate)
  "When deleting the beginning of a pair, and the ending is next char, delete it too."
  (let ((pair (assq (following-char) my-skeleton-pair-alist)))
    (and pair
         (eq (preceding-char) (rest pair))
         (delete-char 1))))

Short explanation: we say that this piece of code will run before backward-delete-char-untabify runs. It does what we need: if we are calling backspace to delete an opening paren and the closing paren is right after, we delete the closing one. After we run, the normal function will run and delete the opening one. Now python-mode or any other mode can rebind backspace as they want, everytime they use the original function, our code will run.

And that’s how we got the second feature (smart backspace). And that’s it. Actually after that, experimenting with other modes, like the cc-mode (default for C/C++ files), I discovered that the opening paren (as other opening chars) are binded to c-electric-paren to do indentation as you type.

I didn’t found an easy way to workaround this like in backspace, because the original function for ( is self-insert-char but making an advice for this function would make our code run lots and lots of times unnecessarily. The other solution would be to create advices for the c-electric-* functions. But since I’m not using this smart parens when using C, I didn’t investigate further, so if you discover another clean way to do this, please tell me.

Conclusion

I hope this post could teach you something about Emacs. The code I present here is not very important, but the concepts around it are, for example: the binding of keys to functions in Emacs, or how to improve functions by using defadvice. Just knowing these concepts is already useful: later when you need something similar, you’ll know where to look. Even better, when solving a problem, you’ll have new ideas on how to solve it.

I’m still using this code, not sure for how long. I can say it’s useful when I’m coding Python — with a few exception cases. In other modes — like when typing normal text — the smart parens are not very useful for me, and in erc (one way to use IRC inside Emacs) they are even annoying, but that’s just me. If you try it, comments are welcome. Also are welcome new ideas on how to deal with these problems.

I had a previous attempt, at blogspot, started almost two years ago to talk about my Summer of Code project. And the last post is from one year ago. I was very excited about our Free Software group in Unicamp (my university).

Since then, a lot of changes have taken place. I was hired by INdT (Instituto Nokia de Tecnologia) to work as a researcher on the OpenBossa labs in Recife/PE. This was about 8 months ago, when I joined the amazing people in the Concepts team and with them helped build Canola2 (and contributed to lots of open source libraries to make it work), a media center for Nokia’s Internet tablets. After much work we have a great product for the tablets.

Well, that’s it for now. I have some other posts still in draft about other changes — the most dramatic being the one about leaving Vim to use Emacs. Let’s see if this time I keep a good posting rate :-)