Delta XMPP Project

It may not be widely known that Psi (as well as QCA and Qt) can be built and used “in-place”. This means you configure and make, but there is no ‘make install’ step. This can be convenient if you don’t want to install these packages system-wide.

Of course, you can always just set a prefix within your home directory (or other user-writable area) when building sources if you don’t wish to install system-wide. However, in-place builds are extra convenient for the developer, as it saves the trouble of having to do ‘make install’ every time something is changed.

So while this is not anything new for Psi development, and nor is the idea particularly revolutionary, it might be something you didn’t know you could do and so I’m documenting it here. Personally, I develop everything in-place.

Qt:

Qt 3 is/was always built in-place. You extract it where you want it to end up, configure and make. There is no ‘make install’ step. With Qt 4, they changed it to behave more like other packages. Now the prefix defaults to somewhere in /usr/local, and there’s a ‘make install’. However, the old way is still supported if you pass the directory of where you extracted Qt to -prefix. That is:

$ tar zxvf qt-x11-opensource-src-[version].tar.gz
$ cd qt-x11-opensource-src-[version]
$ ./configure -prefix $PWD
$ make

This is not a hack or a way to trick Qt. It is actually a supported feature, and when Qt is done configuring you’ll notice it says:

Qt is now configured for building. Just run 'make'.
Once everything is built, Qt is installed.
You should not run 'make install'.

Another feature of Qt 4 is that qmake is aware of the installation location. This allows it to locate the necessary files it needs without QTDIR. Additionally, all subtools (e.g. moc) are invoked using their full path. This means you don’t even need the Qt 4 bin directory in your PATH. Finally, if you’re using gcc, qmake also uses rpath when linking. This means you don’t need Qt 4’s lib directory to be specified anywhere either. No need for /etc/ld.so.conf or LD_LIBRARY_PATH. You simply build Qt, go into your project directory, and run qmake.

$ cd /path/to/myproject
$ /path/to/qt/bin/qmake
$ make

QCA:

Like Qt, QCA (as of 2.0 beta2, I believe) supports in-place builds by pointing prefix to $PWD.

$ ./configure --prefix=$PWD

Note: If you’re building against an in-place Qt, and you didn’t put Qt’s bin directory in your PATH, then you’re going to have to tell configure where Qt is. Do this with –qtdir. For example:

$ ./configure --prefix=$PWD --qtdir=/path/to/qt

Unlike Qt, QCA doesn’t use rpath. This may change in the future, but for now you will need to put QCA’s lib directory in your LD_LIBRARY_PATH if you want to use it in-place:

$ export LD_LIBRARY_PATH=$PWD/lib

There’s actually one hidden secret you must do next, and it is sort of cheating. You need to copy crypto.prf into your qmake feature directory. This is so projects that need QCA can find QCA.

$ cp crypto.prf /path/to/qt/mkspecs/features

Fortunately, this file doesn’t ever change, so if you later modify QCA in-place, there’s no need to copy this file again.

QConf:

With the latest development version (as of October 22nd, 2006), qconf can be run without doing ‘make install’. In fact, you can run it from another location, and it locates its data files relative to the executable. This allows it to work similar to qmake, in that you can invoke it by specifying the full path. Unlike qmake, however, this is not dependent on the prefix. That is, you can configure qconf with the default prefix of /usr/local, and you can still run qconf from anywhere without installing it.

$ ./configure --qtdir=/path/to/qt
$ make
$ cd /path/to/myproject
$ /path/to/qconf
$ ./configure ...

Psi:

Psi can be built and used in-place. Once compiled, just run it.

$ ./configure --qtdir=/path/to/qt
$ make
$ ./psi

That’s it!

You can build and use all the Psi tools without mucking with your system, and without ever doing ‘make install’ either.

QMetaObject::invokeMethod uses QArgument to store arguments and return values. But wouldn’t it be cool if we could use QVariant instead of QArgument? QArgument is just a thin wrapper around a pointer, and using it in a signal is certainly dangerous. On the other hand, putting argument values inside QVariants means we would have real objects that we can marshall around, which would give us great flexibility in performing dynamic function calls.

We already talked about how to dynamically obtain return types. The getReturnType function, as well as what we learned from it, should allow us to make a function for invoking methods that uses QVariant instead of QArgument.

Okay, so here’s getReturnType:

static QByteArray getReturnType(const QMetaObject *obj,
        const QByteArray &method, const QList<QByteArray> argTypes)
{
        for(int n = 0; n < obj->methodCount(); ++n)
        {
                QMetaMethod m = obj->method(n);
                QByteArray sig = m.signature();
                int offset = sig.indexOf('(');
                if(offset == -1)
                        continue;
                QByteArray name = sig.mid(0, offset);
                if(name != method)
                        continue;
                if(m.parameterTypes() != argTypes)
                        continue;

                return m.typeName();
        }
        return QByteArray();
}

Now for the new stuff:

bool invokeMethodWithVariants(QObject *obj,
        const QByteArray &method, const QVariantList &args,
        QVariant *ret, Qt::ConnectionType type = Qt::AutoConnection)
{
        // QMetaObject::invokeMethod() has a 10 argument maximum
        if(args.count() > 10)
                return false;

        QList<QByteArray> argTypes;
        for(int n = 0; n < args.count(); ++n)
                argTypes += args[n].typeName();

        // get return type
        int metatype = 0;
        QByteArray retTypeName = getReturnType(obj->metaObject(),
                method, argTypes);
        if(!retTypeName.isEmpty())
        {
                metatype = QMetaType::type(retTypeName.data());
                if(metatype == 0) // lookup failed
                        return false;
        }

        QGenericArgument arg[10];
        for(int n = 0; n < args.count(); ++n)
                arg[n] = QGenericArgument(args[n].typeName(),
                        args[n].constData());

        QGenericReturnArgument retarg;
        QVariant retval;
        if(metatype != 0)
        {
                retval = QVariant(metatype, (const void *)0);
                retarg = QGenericReturnArgument(retval.typeName(),
                        retval.data());
        }

        if(!QMetaObject::invokeMethod(obj, method.data(), type, retarg,
                arg[0], arg[1], arg[2], arg[3], arg[4],
                arg[5], arg[6], arg[7], arg[8], arg[9]))
        {
                return false;
        }

        if(retval.isValid() && ret)
                *ret = retval;
        return true;
}

Now let’s compare QMetaObject::invokeMethod to invokeMethodWithVariants. Well, here’s the example from invokeMethod’s documentation:

QString retVal;
QMetaObject::invokeMethod(obj, "compute", Qt::DirectConnection,
        Q_RETURN_ARG(QString, retVal),
        Q_ARG(QString, "sqrt"),
        Q_ARG(int, 42),
        Q_ARG(double, 9.7));

Let’s call the same function, using invokeMethodWithVariants:

QVariantList args;
QVariant retVal;
args << "sqrt" << 42 << 9.7;
invokeMethodWithVariants(obj, "compute", args, &retVal,
  Qt::DirectConnection);
// assuming successful invoke, retVal should contain a QString

Now you can call any method dynamically. Yay!

This trick is used in Leapfrog’s Qt-to-Cocoa object bridge. We’ll also make use of it in the Synchronized Threads series.

Delayed signals are a design mechanism for reducing errors by people using your object (similar in spirit to Signal-Safety). This is a difficult topic, because up until this writing I still haven’t been able to formally define a universal rule for when you should use delayed signals. I’ll attempt that now, and show some code to illustrate.

Rule: When requesting a result from an object by first calling a member function and then obtaining the result via a signal, the signal should not be emitted until after the application has returned to the event loop.

To explain just what the heck that means, consider how signal usage can fall into one of two groups: 1) as a means of returning or indicating a result of a requested operation, or 2) as a means of announcing general status changes, independent of a request. #1 is used in worker objects, for example QTcpSocket or QProcess, where there is a request/response expectation. #2 is used predominantly in GUI objects (widgets), where the signals that occur are not requested nor required to operate.

My rule only pertains to #1. One good way to ask yourself if your signal falls under the rule is if it appears to play a part in returning an answer. Synchronous methods will use the return value to deliver an answer. Asynchronous methods don’t use the return value for the answer, and instead give or indicate the answer via a signal. If this is you, then you should use delayed signals.

Now, with that out of the way, you might ask: why use delayed signals?

Well, first of all, you probably wouldn’t be using a signal if you didn’t expect your response to take time to generate. If your object uses the event loop to do its processing before signalling a result, then your signals will already be “delayed” in the sense I’m talking about.

The problem arises when your object knows the result early, and therefore the object does not need to run the event loop in order to generate the result. If the signal is emitted before returning to the event loop (for example, if it is emitted during the “request” member function), then this would not qualify as delayed.

If your object is unpredictable, that is it sometimes emits delayed and other times non-delayed, then the user of your object may have to write extra code to handle both possibilities. The whole point of using a signal is to be able to provide an asynchronous reply, so if we want to pick just one pattern then “delayed” is our answer. It is the more natural pattern for signals anyway.

If you aren’t convinced, the best explanation is with code. Suppose there’s an object that downloads a file from a URL:

class Downloader : public QObject
{
        Q_OBJECT
public:
        void start(const QString &url);
        void stop();

signals:
        void finished(bool success);
};

Simple enough. It will emit finished(true) if the file download succeeded, or finished(false) if there is a problem.

Now let’s make a simple app that tries to download two files, and immediately quits if there is a problem:

class App : public QObject
{
        Q_OBJECT
public:
        Downloader file1, file2;
        int done;

        void start()
        {
                connect(&file1, SIGNAL(finished(bool)), SLOT(dl_finished(bool)));
                connect(&file2, SIGNAL(finished(bool)), SLOT(dl_finished(bool)));

                printf("Starting downloads\\n");
                done = 0;
                file1.start("http://example.com/document.txt");
                file2.start("http://example.com/otherDocument.txt");
        }

signals:
        void quit();

private slots:
        void dl_finished(bool success)
        {
                if(success)
                {
                        ++done;
                        if(done == 2)
                        {
                                printf("Downloads success.\\n");
                                emit quit();
                        }
                }
                else
                {
                        file1.stop();
                        file2.stop();

                        printf("Error during downloads!\\n");
                        emit quit();
                }
        }
};

Looks good, yes?

Actually, no, it isn’t quite right. We assumed that signals would always be emitted to us delayed. If Downloader doesn’t emit delayed 100% of the time, then we have bugs.

Let’s assume Downloader may or may not always emit delayed. We’ll write extra code then, to handle all situations:

class App : public QObject
{
        Q_OBJECT
public:
        Downloader file1, file2;
        int done;
        bool error;

        void start()
        {
                connect(&file1, SIGNAL(finished(bool)), SLOT(dl_finished(bool)));
                connect(&file2, SIGNAL(finished(bool)), SLOT(dl_finished(bool)));

                printf("Starting downloads\\n");
                done = 0;
                error = false;
                file1.start("http://example.com/document.txt");
                if(!error)
                        file2.start("http://example.com/otherDocument.txt");
        }

signals:
        void quit();

private slots:
        void dl_finished(bool success)
        {
                if(success)
                {
                        ++done;
                        if(done == 2)
                        {
                                printf("Downloads success.\\n");
                                emit quit();
                        }
                }
                else
                {
                        file1.stop();
                        file2.stop();
                        error = true;

                        printf("Error during downloads!\\n");
                        emit quit();
                        QTimer::singleShot(0, this, SIGNAL(quit()));
                }
        }
};

If finished(false) is emitted non-delayed by file1.start(), then this means we will perform our deinitialization process before App::start() completes. We use the ‘error’ bool to track if this happens, so that we can skip the calling of file2.start() as necessary. Since file2.stop() would occur before the call to file2.start() (had we not fixed the code), this would most likely mean that the second file transfer wouldn’t be stopped at all.

Finally, if we do a normal emit of quit() during a non-delayed call to finished(), then we lose signal-safety. If App is deleted during the slot that listens to quit(), the code following file1.start() will cause a crash when App’s members are accessed (such as the ‘error’ bool). We can give ourselves signal-safety by doing a delayed emit of quit() with QTimer::singleShot.

Bottom line: there’s less code if we can assume that Downloader always uses delayed signals.

QMetaObject::invokeMethod allows calling methods on Qt objects dynamically. The method name is passed as a string, and the arguments are put together with QGenericArgument. If you haven’t read the documentation for the invokeMethod function, do so. It will open your eyes about Qt.

There’s just one problem: how do you handle a dynamic return value? QGenericReturnValue has to be set in advance with the type of value being returned. If we want to have a fully dynamic call, with even the return value type determined at runtime, then we need a way of inspecting a method to obtain its return type before we call it. After that, we can set up our QGenericReturnValue properly and call the method.

So, how do you get the return value type? Like this:

QByteArray getReturnType(const QObject *obj,
        const QByteArray &method,
        const QList<QByteArray> argTypes)
{
        const QMetaObject *mo = obj->metaObject();
        for(int n = 0; n < mo->methodCount(); ++n)
        {
                QMetaMethod m = mo->method(n);
                QByteArray sig = m.signature();
                int n = sig.indexOf('(');
                if(n == -1)
                        continue;
                QByteArray name = sig.mid(0, n);
                if(name != method)
                        continue;
                if(m.parameterTypes() != argTypes)
                        continue;

                return m.typeName();
        }
        return QByteArray();
}

Now you can have some code:

class MyObject : public QObject
{
        Q_OBJECT
        ...
public slots:
        QString intToString(int x)
        {
                return QString::number(x);
        }
};

MyObject *obj = ...
QByteArray methodName = "intToString";
QList<QByteArray> argTypes;
argTypes += "int";
QByteArray retType = getReturnType(obj, methodName, argTypes);
// the value of retType is now "QString"

At this point, we have the method name, the argument types, and the return type. For homework: figure out how to use this information in order to make an invokeMethod() call. I’ll explain how in a later article.

The astute observer will notice a problem in the code I provided for Synchronized Threads (Part 1), which is that the eventloop begins only after we unlock the mutex.

Why is that bad? Well, this means that the calling thread will resume before the eventloop in the SyncThread has started. If the calling thread then calls stop() right away, it is possible that the “quit” method of the QEventLoop object will be delivered too early. If this happens, then the eventloop won’t actually quit, and the calling thread will be stuck forever. Additionally, code within atStart() occurs without an eventloop running. These two problems are nearly identical to those covered in the startmyappLater article, which is why I published it before this one.

Correcting this problem in SyncThread takes a little bit of effort. We are going to need a helper QObject that runs in our thread, which I’ve called SyncThreadAgent.

class SyncThread : public QThread
{
        Q_OBJECT
private:
        QMutex m;
        QWaitCondition w;
        QEventLoop *loop;
        SyncThreadAgent *agent;

public:
        SyncThread(QObject *parent = 0);
        ~SyncThread();

        void start();
        void stop();

protected:
        virtual void run();
        virtual void atStart() = 0;
        virtual void atEnd() = 0;

private slots:
        void agent_started();
};

class SyncThreadAgent : public QObject
{
        Q_OBJECT
public:
        SyncThreadAgent(QObject *parent = 0) : QObject(parent)
        {
                QMetaObject::invokeMethod(this, "started", Qt::QueuedConnection);
        }

signals:
        void started();
};

SyncThread::SyncThread(QObject *parent)
:QThread(parent)
{
        loop = 0;
        agent = 0;
}

SyncThread::~SyncThread()
{
        stop();
}

void SyncThread::start()
{
        QMutexLocker locker(&m);
        Q_ASSERT(!loop);
        QThread::start();
        w.wait(&m);
}

void SyncThread::stop()
{
        QMutexLocker locker(&m);
        if(!loop)
                return;
        QMetaObject::invokeMethod(loop, "quit");
        w.wait(&m);
        wait();
}

void SyncThread::run()
{
        m.lock();
        loop = new QEventLoop;
        atStart();
        w.wakeOne();
        m.unlock();
        agent = new SyncThreadAgent;
        connect(agent, SIGNAL(started()), SLOT(agent_started()), Qt::DirectConnection);
        loop->exec ();
        m.lock();
        atEnd();
        delete agent;
        agent = 0;
        delete loop;
        loop = 0;
        w.wakeOne();
        m.unlock();
}

void SyncThread::agent_started()
{
        atStart();
        w.wakeOne();
        m.unlock();
}

Now everything should be correct.

Stay tuned, we will do more with SyncThreadAgent in Part 3.

Here is a simple trick you should use when starting up your Qt program. If you have read just about any of my “main.cpp” files, you may have seen it.

Ok, so say you have this:

class App : public QObject
{
        Q_OBJECT
public:
        App()
        {
                // init, do stuff, whatever
        }

signals:
        void quit();
};

int main(int argc, char **argv)
{
        QApplication a(argc, argv);
        App app;
        QObject::connect(&app, SIGNAL(quit()), &a, SLOT(quit()));
        a.exec ();
        return 0;
}

Here’s a basic application object. It can emit quit() to end the program. The constructor does the initialization, makes QWidgets, etc, and off you go.

However, there is a subtle problem. The Qt eventloop is not active during App’s constructor! The eventloop begins only once exec is called. Here are the snags of this pre-eventloop phase:

1) You can’t call any functions that require the eventloop to already exist. For example, I believe QObject::deleteLater() does nothing if the loop hasn’t started. I don’t know if you can use QTimer objects either (singleshots, however, are allowed. see below).

2) You can’t exit via QCoreApplication::quit(). This means if you decide during your initialization that you want to exit, you can’t. You need to wait for the eventloop to begin before you can do that.

We can fix this with a QTimer singleshot, an operation that fortunately does work in this phase. We simply create a start() slot in App, and set a singleshot on it from main().

class App : public QObject
{
        Q_OBJECT
public slots:
        void start()
        {
                // init, do stuff, whatever
        }

signals:
        void quit();
};

int main(int argc, char **argv)
{
        QApplication a(argc, argv);
        App app;
        QObject::connect(&app, SIGNAL(quit()), &a, SLOT(quit()));
        QTimer::singleShot(0, &app, SLOT(start()));
        a.exec ();
        return 0;
}

That’s it! Now you can do your major initialization inside start(), with the assurance that all code inside there will work as expected.

Suppose you want to do some work in another thread. Qt gives you the right tools — QThread, QMutex, QWaitCondition, and even signals and slots across threads — but there is still some tedious and error-prone work left even for basic situations.

First of all, there’s no “simple” way to know when the services of a remote thread can be referenced. QThread has a started() signal, but I find it to be nearly useless. Just because some thread has started does not mean you can start calling methods on objects in that thread. How do you know the object pointers? Have the objects even been made yet? Is there any initialization that needs to be done in the remote objects before you can start shooting signals all over the place? What we really want is a signal that is emitted not only after the thread starts, but after some of our own initialization occurs within that thread.

That said, I also consider an asynchronous initialization notification to be overkill in the general case. Starting a thread should take no time at all, nor should the initialization of some objects. Synchronizing the startup of the thread greatly decreases the complexity of the program. Similarly, I think it is worthwhile to synchronize shutdown.

Well, here we go:

class SyncThread : public QThread
{
        Q_OBJECT
public:
        QMutex m;
        QWaitCondition w;
        QEventLoop *loop;

        SyncThread(QObject *parent = 0) : QThread(parent)
        {
                loop = 0;
        }

        ~SyncThread()
        {
                stop();
        }

        void start()
        {
                QMutexLocker locker(&m);
                QThread::start();
                w.wait(&m);
        }

        void stop()
        {
                QMutexLocker locker(&m);
                if(!loop)
                        return;
                QMetaObject::invokeMethod(loop, "quit");
                w.wait(&m);
                wait();
        }

protected:
        virtual void run()
        {
                m.lock();
                loop = new QEventLoop;
                atStart();
                w.wakeOne();
                m.unlock();
                loop->exec ();
                m.lock();
                atEnd();
                delete loop;
                loop = 0;
                w.wakeOne();
                m.unlock();
        }

        virtual void atStart() = 0;
        virtual void atEnd() = 0;
};

So, what does this all mean? Well, now, instead of inheriting QThread and implementing the run() function, you can inherit SyncThread and implement the atStart() and atEnd() functions.

Suppose you make a class called MyThread that inherits SyncThread:

class MyThread : public SyncThread
{
        Q_OBJECT
public:
        SomeObject *obj;

        MyThread(QObject *parent) : SyncThread(parent)
        {
                obj = 0;
        }

protected:
        void atStart()
        {
                obj = new SomeObject;
        }

        void atEnd()
        {
                delete obj;
        }
};

Now, you can call start() to begin the thread, and SomeObject is guaranteed to be accessible when start() returns.

MyThread *thread = new MyThread;
thread->start();
thread->obj->foo();
delete thread;

Stopping the thread is also cleanly synchronous. The eventloop of the other thread is stopped, atEnd is called, and the thread itself stops, all before the delete finishes.

If you keep your atStart/atEnd reasonable, that is, you don’t put more code in them than you’d normally put into one pass of the event loop, the performance trade-off should be negligable.

Look above at MyThread again just to see how simple it is. There isn’t even a mutex! Stick to signals and slots when interacting with SomeObject, and you can have safe multithreaded code without even trying.

Ok, so we can do this now:

That’s presence using Link-Local Messaging (a.k.a. Apple’s Bonjour Chat protocol, now documented in JEP-0174). Presence is delivered via local DNS Service Discovery. Chat is done via peer-to-peer Jabber.

Doing Bonjour presence is not all that new actually. The JDNS subsystem has been able to do it since last year. What is notable this time around is that I’m doing it via the new Iris DNS API, which is an abstraction layer, and JDNS is just one possible backend. And there’s also an avatar / buddy icon. That’s good old icon_32.png from Psi.

If you want to try it yourself, have a look at the irisnet command-line tool (now committed to the CVS). Quite a few rough edges remain, but it should work. Here is what I ran:

./irisnet pserv justin@ubuntu _presence._tcp 5298
"txtvers=1,1st=Justin,port.p2pj=5298,status=dnd,
msg=coding on iris stuff,
phsh=b12eebe7b5a2f2673aba2f6eb0f3264939a78f67"
-a null:icon_32.png

Yeah, it’s pretty long. It is supposed to be a single line, but I’ve broken it up for readability. “irisnet pserv” means to publish a service. Then it goes: JID, service type, port, list of attributes. The -a means to add an extra DNS record, in this case the avatar image (yeah, iChat delivers avatars over DNS, don’t ask questions).

The value of the phsh attribute is the SHA1 hash of the avatar image in hexidecimal. I used the ’sha1sum’ program to do this, which is present on most Linux installations at least:

$ sha1sum icon_32.png
b12eebe7b5a2f2673aba2f6eb0f3264939a78f67  icon_32.png

I don’t know when we’ll see Link-Local Messaging in Psi, but it shouldn’t be too hard to do at this point.

There are three ways Psi might locate a Jabber service. ServiceResolver covers all three, on all platforms, and makes it easy.

To demonstrate how ServiceResolver works, below you can see some output of my ‘irisnet’ commandline tool (which still in development).

The main way to find a Jabber server is via SRV. ServiceResolver looks up the SRV records of a given type on a given domain, as well as the address (A/AAAA) records for each:

$ ./irisnet rservd sapo.pt _xmpp-client._tcp
[213.13.146.7] port=5222
[213.13.146.8] port=5222

You can also lookup non-SRV (”plain”) services. This is done by pre-supplying the port to use, which is returned with each address record found:

$ ./irisnet rservp gmail.com 5222
[216.239.57.107] port=5222
[64.233.161.107] port=5222
[64.233.171.107] port=5222

Lastly, suppose there’s a JEP-0174 / iChat service instance on the LAN called someguy@macbox. In this case, DNS-SD is used:

$ ./irisnet rservi someguy@macbox _presence._tcp
[192.168.1.156] port=5298

There may be three different input methods, but there is only one output method: a list of address & port pairs. The list is always sorted in the order in which the addresses should be tried. Priorities and weights are taken care of for you.

With cross-platform capability, pluggable backends, and seamless Bonjour support, the Iris DNS system is shaping up to be one of the most simple and capable such systems in existence. I suppose I don’t feel so bad about it taking forever to finish, because we really have something cool here.

#ifdef QT_GUI_LIB
# include <QApplication>
#endif

inline bool is_gui_app()
{
#ifdef QT_GUI_LIB
        return (QApplication::type() != QApplication::Tty);
#else
        return false;
#endif
}

This is enough if you don’t plan to actually touch the Qt GUI API. Otherwise, you’ll have to drag that QT_GUI_LIB ifdef all over your code.

(I’m going to use this to determine whether or not I should set up the Windows console control handler.)