Slixmpp is a python 3 XMPP library, based on SleekXMPP. I don’t know how you would name it, but “fork” is not appropriate because future changes from SleekXMPP that apply to slixmpp will be merged into slixmpp’s code. Maybe “branch”.

Description

To sum up:

• Do not use any thread
• Support only python >= 3.2
• Remove all compatibility with old API version of SleekXMPP

Basically, for users, slixmpp should be almost usable like SleekXMPP without any change, except for some features that are available only through threads (for example res = iq.send() which looks simple and intuitive but is actually a nightmare, or add_handler(…, threaded=True). The main goal is to get rid of threads, and to cleanup the code by removing a few hacks that are only used for python 2 support or backward compatibility. All changes are made at the core of the library and everything else (for example the stanza objects, or the plugins) should remain almost identical.

Removing threads

As you may have noticed, I don’t like threads. I really like SleekXMPP, its way to provides stanzas in very useful objects, the way plugins are made and how easy it is to implement a new XEP.

At first, I was skeptical when I learned that everything was handled through threads: there are 3 default threads (one for reading on the socket, one for writing on the socket, and one to handle timed events) and one additional thread is spawned for each callback triggered by a received stanza. This is, in my opinion, not the best design for a networking library. This is Ok for a bot or a server or any software where SleekXMPP is the only source of events, but from the moment you need to handle other events, for example other network sockets (an MPD socket for example) or (more commonly) user inputs, this becomes a mess. The current solution to do that in SleekXMPP is to run SleekXMPP process in its own thread (through client.process(block=False)) and do the rest in the main thread (read stdin or other sockets, or handle gtk events, or whatever). There is no solution to do something like

def main_loop():
    while True:
          sleekxmpp_client.process(timeout=0.1)
          gtk.main_iteration()
          mpd_socket.handle_events()

We’re stuck with one thread for sleekxmpp events and one other thread for everything else:

def main_loop():
    sleekxmpp_client.process(threaded=True)
    while True:
          gtk.main_iteration()
          mpd_socket.handle_events()

Now what happens when you receive a stanza that triggers an event, that will write something on your screen, while the user enters a command that will write something on the screen? Hell yeah, a race condition!

How do you avoid that? By surrounding every bit of your code that could be changed by both threads with locks (mutex, semaphores…).

That’s what we’ve tried to do in poezio to avoid garbage on the screen when curses’ screen is touched by two threads at the same time. But actually we should do that for almost all poezio code, and that’s not done. Hence the (rare) segmentation faults (yes, the python interpreter) and other unreproducible wrong behaviour that some user sometimes encounter (this may also be a bug in Poezio itself, but as long as we have threads, I’ll blame threads).

How could we fix this? How could we read on fifos, on an MPD socket, on a ZMQ IPC socket? Run each thing in its own thread and add more lock everywhere…?

Also it makes everything complicated, impossible to diagnose, debug or even reproduce. A good software should (at least) be predictible. Threads make everything unpredictable.

My solution is to entirely remove every bit of threads from SleekXMPP. I’m more the “one single select/poll loop for everything” kind of guy, and here is how I see it:

• slixmpp internally uses one single poll/epoll/select to do its writing/reading on the socket AND the timed events thing (that’s easy, just retrieve the time of the next timed event, pass that delay in poll()’s timeout, and you’re done), everytime it receives a stanza, it triggers the callbacks one by one
• If you want to monitor a socket, or anything file-like (a ZMQ or tcp socket, stdin, stdout…), you can just pass to slixmpp an object containing your file object, an fd to monitor and a callback, and slixmpp will register that socket into its uniq poller and will trigger your personal callback when it’s time to
• If you want to handle other events, just pass a short timeout to sleekxmpp_client.process() and do something like the code example above

With this simple process, it becomes possible (and actually easier than running your own select) to monitor everything without any thread, avoiding a lot of complexity in XMPP clients code and most importantly: no potential race condition.