I recently learned that Snow Leopard includes a (public!) framework called IOSurface that allows for the sharing of basically OpenGL textures between processes. That neatly solves the problem with having your graphics device running in a separate process for the GUI. Previously, I've shipped bitmaps around between the GUI and the backing process (which is headless) but this could definitely be a performance improvement both in update speed and in memory. I'll have to look into it.
I suspect even RGL could be made to operate in this mode though you'd need some help from the package since it expects to run an event loop for handling mouse interaction with the scene graph.