Statistical Computing & Software

Thoughts on an iPad R (or similar device)

2010-07-18T20:36:00.000-07:00

This has come up a few times on various lists I read and I have a strong opinion, so I figured I'd jot down some notes. The topic of discussion is the notion of R running on an Apple iPad, the inevitable Android Pads or the HP Slate (assuming the thing ever makes it out the door). Maybe even some sort of Windows 7 Mobile device.

There are a couple of immediate hurdles that come to mind:

* Limited RAM. The iPad has 256MB, the iPhone 4 has 512MB. A standard R compute box sports, what, 4GB at a minimum these days? Sure, back when I started using R I had maybe 512MB or 1GB of ram on my laptop, but I also had a lot less data back then.

* Battery. It doesn't really matter how fast the processors in these things are, the big problem is battery. Batteries aren't really getting better, the 10 hour battery life in modern laptops comes from a) bigger batteries (this is real reason you can't remove the battery from an Apple laptop anymore---doing that lets you wedge a bigger battery into the box without weight gain) and b) aggressive power management for the CPU.

* Apple prohibits interpreters outside of very specific use cases (games, Javascript). This is an Apple specific thing, but it's worth mentioning as there are now a lot of iPads out there. (Also Fortran is an issue under the new rules.)

For those reasons, I don't think there should be an R that runs directly on your Pad device the same way it does on your desktop. You don't have enough RAM and if you crank up the processor enough you'll destroy your battery life. Just look at what your favorite game does to your battery and remember that it gets to use specialized hardware for its processing.

So what should we do instead? Those who've known me for a while (they probably don't read this, but whatever) will remember StatPaper and the "Kernel" R hobby projects I've had at various times. Both operated on the premise that the R engine itself should run in a separate process from the UI and interact with the user through some sort of remote connection. This was inspired by Mathematica in the case of StatPaper and by the desire to use the multiple cores found in most machines through a single GUI in the case of Kernel R (which let you spin up multiple R consoles and transfer data between them).

I don't really work on either anymore, I have a day job and don't really program recreationally anymore (not that I dislike programming, it's just that I have work problems that provide a focus for learning new technologies and noodling around so I tend to go there first). But I do still like the idea of them and I think they could work for R. Here's I think what looks like a plan.

* Move away from the notion of R on a filesystem. R should really be living in something like a database, probably an object database. One option might be some of the new-ish document-oriented databases like CouchDB or MongoDB. Both would let you serialize R objects into documents using something that looked a lot like a standard environment.

* These databases should also live in a cloud environment (Cloudy-R! Eh? I know you like it). Having some sort of standard serialization that wasn't R specific could also let you have other applications updating data and using results. This lets R focus one what it's good at without being forced to do something it isn't good at (like web UIs).

* Use the same serialization format as the command format for a remote R UI. The serialization format should ALSO encode the graphics device.

Once you have those things you have the core of what you need for your Pad R. You have an R that exists in an always running state (because you could serialize the entire R environment to your database when the environment is idle. If you build it carefully you could probably even eliminate the need for a sticky session but that would likely involve some rummaging about in the internals).

On the client side, you can start simple with the familiar console interface then you can start to experiment with other interface modalities. Perhaps some sort of direct manipulation of a graphics object for exploratory data analysis.

Fascinating Captain...

2009-11-04T21:46:00.001-08:00

I recently learned that Snow Leopard includes a (public!) framework called IOSurface that allows for the sharing of basically OpenGL textures between processes. That neatly solves the problem with having your graphics device running in a separate process for the GUI. Previously, I've shipped bitmaps around between the GUI and the backing process (which is headless) but this could definitely be a performance improvement both in update speed and in memory. I'll have to look into it.

I suspect even RGL could be made to operate in this mode though you'd need some help from the package since it expects to run an event loop for handling mouse interaction with the scene graph.

Aw, nuts.

2009-09-07T19:25:00.001-07:00

Upgraded to Snow Leopard and so it is time once again to make some changes to my personal R GUI--if for no other reason than to test cool stuff like Blocks and Grand Central Dispatch. It also looks like there's some interesting developments in Pasteboards and a few other under the hood places.

First off are Blocks. They look like function pointers but they definitely aren't. They're actually Objective-C classes that you can treat like functions. Very Smalltalk, complete with similar tricks to stuff their data onto the stack you explicitly move them to the heap (they're objects so you can copy them).

Unfortunately, they are not interchangeable with C function pointers so you can't just use them as callback functions. A shame really, since right now I have a libffi-based binding between the R callback functions (ReadConsole, WriteConsole, etc) to thunk to Objective-C methods. All hope is not lost, the R binding is not so complicated so writing a Block compatible version shouldn't be too hard.

Seen in a report about the G1...

2008-12-19T00:00:00.000-08:00

Says Tim Bray about the G1's browser (compared to the iPhone's Safari browser):

All they need to catch up is some fit-and-finish and that little pinch/unpinch trick for zooming.

Yes, well... Horseshoes, hand grenades and thermonuclear devices and all that. Seriously, the only thing that really sets one operating system apart from another IS fit and finish. Something the Linux On The Desktop (or laptop for that matter) have never really figured out. Aping pieces of Windows and OS X (I notice modern distros seem to have BOTH a start bar AND a global menu bar... why?) will never get you fit and finish. Some poor bastard (bastards most likely) sitting down and smoothing the hell out of the rough interface edges is what's going to get you there.

Ah, interesting....

2008-08-10T23:12:00.000-07:00

It looks like someone is finally doing a commercial R of some sort (and congrats on the series A to anyone who may work there) over at REvolution Computing (and thanks to Ben for sending me Google alert). I always sort of wondered why Insightful didn't try to wire up their GUI stuff from S-PLUS to the R engine, aside from maybe some sort of licensing concern. Though it looks like a lot of people involved are ex-Insightful the website seems to indicate more of a focus on support rather than front end--I'll be very curious to see how that plays out.

No official Mac support though, so it's unlikely to affect my life for the time being. ;-)

Excellent

2008-03-25T23:05:00.001-07:00

The feature freeze for R 2.7.0 is in place with an expected April 22nd release. I've noticed some of the locale stuff has changed recently (LC_CTYPE and the like) so I'll need to fix those warning messages. Right now I'm setting up the IKImageBrowserView-backed graphics device interface for the GUI, though the built-in quartz device has been serving me well so far. There are a couple of reasons for the view set up, which uses bitmaps to represent the graphics device on the backend. After the page is completed, the bitmap is converted to a more efficient representation (e.g. PNG) for temporary storage and the display list recorded into the device. We'll also continue to cheat and provide a Cover Flow mode (no, I don't care that it's a private API).

Another interesting feature is that, unlike the current Mac GUI, the graphics device is considered a View on the same Model (the interpreter). We'll see how it plays out, but that might help with some of the fancy footwork usually required when dealing with the NSDocumentController.

Implementing XCode 3 parenthesis highlighting

2008-03-18T20:43:00.000-07:00

One of the things I like about XCode 3 is the parenthesis matching. It seems very obtrusive at first, but I find it works better than the more subtle Emacs-style matching once you get used to it. For the R GUI I wanted to implement the same thing, which turned out to be fairly simple.

The highlighter is implemented using the new [NSTextView showFindIndicatorInRange:] so there's literally no work to do there.

The first thing you need to know is that adding this functionality in textView:shouldChangeTextInRange:replacementString: won't work because the text is added after the method executes and that removes the find indicator. Fortunately, we just implemented a custom keybinding in the last post so all we need to do is bind the various close delimiters to special selectors (fancyCloseParen: for example). Then we can do things in doCommandBySelector:


// ...
} else if(@selector(fancyCloseParen:) == aSelector) {
  [self insertDelimiter:')' withMatch:'(' inTextView:aTextView atPosition:aRange.location];
  return YES;
} // ...

next all we have to do is find the delimiter (which we do with a category on NSString) and then insert the find indicator:


- (void)insertDelimiter:(unichar)aDelim withMatch:(unichar)aMatch inTextView:(NSTextView*)aTextView atPosition:(NSInteger)aPos {
  [aTextView insertText:[NSString stringWithFormat:@"%c",aDelim]];
  NSRange where = [[aTextView string] rangeOfDelimiter:aMatch matching:aDelim inRange:NSMakeRange(lastPrompt,aPos-lastPrompt)];
  if(where.location == NSNotFound)
   NSBeep();
  else
   [aTextView showFindIndicatorForRange:where];
}

Easy as pie.

On Key Bindings

2008-03-15T19:00:00.000-07:00

For my R GUI I want user customizable key bindings. I know for a fact that there are customizable key bindings built into the Cocoa text system, but they aren't user accessible and the only response from Apple was a message posted to a mailing list about 2 years ago to the effect of "I have something." Two years later, well, surprise, surprise. So what to do?

Well, for the first pass (found in the R-Multi git repository) I class-dumped AppKit and found NSKeyBindingManager. It turns out that this class has a shared initializer and a means for setting a new key binding dictionary. So, I did the simple thing and just changed that dictionary during app initialization to support some new keybindings similar to the ones found in TextMate (which doesn't use NSTextView as far as I know):


NSMutableDictionary *newBindings = [[NSMutableDictionary alloc] 
    initWithDictionary:origBindings];
[[NSKeyBindingManager sharedKeyBindingManager] setDictionary:newBindings];
[newBindings release];

About as simple as you can possibly get. Unfortunately, this approach has several drawbacks primarily due to the fact that it is a global change to the entire application (which is more useful that the SYSTEM global version mostly recommended). Can we do better? Let's find out.

For a clue, let's look at one of the R GUI's own crash traces:


[NSTextView keyDown:]
...[NSView interpretKeyEvents:]
......[NSTSMInputContext interpretKeyEvents:]
.........[NSKeyBindingManager(NSKeyBindingManager_MultiClients) flushTextForClient:]

Excellent. So it looks like we'll be needing to subclass and hook into interpretKeyEvents: to get things to work properly in our system. However, we cannot simply use an NSKeyBindingManager because the manager doesn't return a BOOL so we can't tell when we haven't handled a binding. Or does it?

So, it turns out there is a MultiClient category on NSKeyBindingManager that lets us do just that using interpretEventAsCommand:forClient:

So, my interpretKeyEvents now looks like:


- (void)interpretKeyEvents:(id)sender {
 if([(NSArray*)sender count] == 1) {
  if(YES == [manager interpretEventAsCommand:[(NSArray*)sender objectAtIndex:0] 
                   forClient:self]) return;
 }
 [super interpretKeyEvents:sender];
}

So far it seems to work in testing, but I'll have to do more testing with it over time. For example, I don't know if I'll ever see more than one event in that array.

Dear iPhone SDK haters,

2008-03-14T22:26:00.000-07:00

Nobody is forcing you to develop for the iPhone. If you don't like it, by all means, go develop for 'Android' and be sure to let me know how that works out for you.

A little FFI goes a long way

2008-03-11T22:05:00.000-07:00

As my two faithful readers have surely noted, one of my little projects is a Leopard-specific GUI for R that tries to leverage as many of the new features as possible. As it happens, one of those features is Garbage Collection, which I can't use because of the way the R framework is built. Perhaps I'll be able to use Simon's method for building self-contained R applications with the framework built in the appropriate way.

What I'd really like to talk about though is the fact that having PyObjC and whatever the Ruby bridge is called means that I can be positive that a copy of libffi is available on every Leopard system. This let me finally build something I've wanted for a long time: an elegant C<->Cocoa thunk for the R function pointers. To wit:


ptr_R_WriteConsoleEx = ffi_bind(self,@selector(writeConsole:length:type:),&ffi_type_void,
  4,&ffi_type_pointer,&ffi_type_sint,&ffi_type_sint);

So, how do we do it? Well, first we need a little bit of context to keep a binding between a function pointer and a specific object/method combination. We also keep the types of arguments around as well as the return type. If we were doing a complete libffi bridge we would introspect this from the class at runtime rather than the static method I use here. In the case of R this would be overkill since the R function pointers are very much fixed entities. In any case, here's the little structure I use:


typedef struct {
 id  obj;
 SEL sel;
 IMP fn;
 int nargs;
 ffi_type *retval;
 ffi_cif  *cif;
 ffi_type *types[0];
} ffi_call_struct;

Next, we need to write a little function to use as a closure and move the C function call to the ObjC function call. All libffi closure handlers look have the same function declaration:


void handler(ffi_cif *cif,void *retval,void *args[],void *user_data) {

First we build a place to hold our arguments


 int i;
 ffi_call_struct *call = (ffi_call_struct*)user_data;
 void **values = (void**)malloc(sizeof(void*)*(call->nargs+2));

and simply transfer the arguments accross, making sure that the first two are the target object and target selector (Obj-C methods have two implicit arguments):


 for(i=0;inargs;i++) { values[2+i] = args[i]; }
 values[0] = &(call->obj);
 values[1] = &(call->sel);

Then all we do is use ffi_call to make a call. Note that we use the NSAutoreleasePool method, which wouldn't be necessary if we could enable garbage collection:


 NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
 ffi_call(call->cif,(void (*)(void))call->fn,retval,values);
 [pool release];
 free(values);
}

Next we need a little function to build an appropriate closure function to use as the R function pointer:


void *ffi_bind(id obj,SEL selector,ffi_type *retval,int nargs,...) {
 int i;
 ffi_cif *cif;
 ffi_closure *closure;
 va_list ap;

We use vargs so that we can define the argument lists inline. First, we allocate our context structure and transfer the argument types


 //Allocate the structure and copy the call information into the structure.
 ffi_call_struct *call = (ffi_call_struct*)malloc(sizeof(ffi_call_struct)+(nargs+2)*sizeof(ffi_type*));
 call->obj = obj;
 call->sel = selector;
 call->fn  = [obj methodForSelector:selector];
 call->retval = retval;
 call->nargs = nargs;

 call->cif = (ffi_cif*)malloc(sizeof(ffi_cif));
 cif = (ffi_cif*)malloc(sizeof(ffi_cif));
 
 call->types[0] = &ffi_type_pointer;
 call->types[1] = &ffi_type_pointer;
 va_start(ap,nargs);
 for(i=0;itypes[2+i] = va_arg(ap,ffi_type*);

Next we define the two ffi_cifs. The first one is the ffi_cif of the Obj-C method call and the second is the one for the function pointer we're trying to create. Finally, we create the closure and return it


 ffi_prep_cif(call->cif,FFI_DEFAULT_ABI,nargs+2,call->retval,call->types); //The CIF for the ObjC Method
 ffi_prep_cif(cif,FFI_DEFAULT_ABI,nargs,call->retval,&(call->types[2]));   //The CIF for the function call
 closure = (ffi_closure*)malloc(sizeof(ffi_closure));
 ffi_prep_closure(closure,cif,handler,call);
 return closure;
}

and that's pretty much all there is to it. There are two minor caveats, the first is that anything that employs method swizzling on this class is likely to fail since we cache the function pointer at bind time. Fortunately, swizzling is relatively rare these days so it generally doesn't happen. It may also technically be slower than the hand-coded method, but there are a couple of areas where we could probably speed up the thunking operation---the allocation of

values

in particular could be moved to the context object.

And the bloggers chime in...

2008-03-06T23:03:00.000-08:00

So, the iPhone SDK was announced (good luck with actually downloading it) and I'm looking forward to playing around with it (no, I'm not going to try to get R running. Let's face it, the iPhone UI is more like, say, Data Desk than anything else). I am getting annoyed with assorted bloggers and pundits bitching about perceived shortcomings.

First, the 70/30 split. That seems to be the biggest complaint. In the words of Dr. Evil, "quite typical, really." Jobs is probably right when he says that it's just enough to cover expenses. Credit card processing ain't free, kids. They might see a little net revenue, but it's in the noise relative to the rest of their business. Basically, apps are a way to sell iPhones, and it'll work.

I also saw someone complain about not being able to change the iPhone UI L&F. Well, yeah. Duh. Apple has seen X11 and knows exactly what happens when you allow that sort of control. Basically, rip-offs of the Windows 95 UI (No, don't expect me to be impressed that you can map Windows 95 on to the sides of a cube and spin it around. At the end of the day you've still got Windows 95 with uglier controls).

Speaking of UI and complete non sequiturs (well, not completely since Salesforce presented in the iPhone thing). I ran across an O'Reilly book this evening at Barnes & Noble called "Designing Dashboards," except that the title was longer. For statisticians with at least a mild interest in presenting results there's nothing there that comes as a surprise (and several areas that could have used some help, color palette selection for example). Clearly the author has read Tufte's books (sparklines even made an appearance). Mostly, though, I'm struck by how absolutely horrible the types of dashboard produced by things like Business Objects and Salesforce (there are others, but these are really popular). The outputs are very expensive in terms of real estate (lots of round things, pie charts and dials and such. Usually in some godawful 3d rendering) and deliver very little, usually a single number. Even when they become more traditional/less junky, the little things really start to stand out. There's a lot of "stop light color" usage, which seems like a good idea but really just induces fatigue (I tried this recently in my own workflow tool and could only stand to use the tool for about 10 minutes. The same dimension is now represented by a much more subtle change in glyph size, which is surprisingly easy to assess at a glance). Even when they move away from stop-light colors there's no rhyme or reason to the color choices nor any apparent internal consistency.

Anyway, I'll probably write more on what I'm learning about visual display of complex information from the workflow tool I've been developing as a place to hang my modeling hat for the last couple of months. I've got a stable core set of users and I use the tool myself literally daily (usually the development version which varies somewhat from the 'production' version). I've also got some things to say about R and databases and workflow in that environment as well.

Long time, no post.

2008-01-30T11:06:00.000-08:00

So, a comment to my Cover Flow post yesterday reminded me that I haven't posted anything here in quite some time despite making a lot of progress on an R GUI over the Summer and the release of Leopard at the end of October. I have an excuse though! See, in mid-October I left academia and started working for a company in San Francisco called AdBrite. I'm pretty happy with things so far, I get to do statistics and program basically as much as I want. There were a bunch of factors that contributed to the switch from academics, most of which I'm not going to discuss on a public blog, though I'm happy to correspond via email. Long story short, AdBrite made me an offer I couldn't refuse (and it had nothing to do with the Nolan Lab, Garry and all the folks in the lab are great). The whole NIH obligation thing could become a problem (this is an area where cross discipline researchers really get screwed), but I'll deal with it as it comes, I try to think of it as a really ill-advised student loan.

Anyway, this means I haven't had much time for the R Mac GUI in the last several months as most of the work I've been doing has been more server-side rather than client-side so I haven't seen much of the R GUI and when I do see it, it happens to be on a Windows box---we have SAS and SAS notably is not available for OS X. I suppose I could Parallels it, but I'd still be running Windows and the Win64 box was already here.

I'd like to get back to it, but we'll have to see. I think at the very least I'll get the code off my backup drive and put the source code up somewhere so it isn't lost to the ages. On the bright side, a lot of the graphics drawing code made it in the core R release for 2.7.0 and the API Simon chose to adopt is fairly similar so a bunch of the graphics code can actually be removed. Simon also put paging into the core graphics device, though it's not as fun as Cover Flow or the Image Browser version (though those are Leopard specific so he can't really put it into the core R).

I think there's some useful stuff in the Console implementation, particularly the clipboard support and the improved completion/hinting support. I like Simon's idea of having a built-in console with the ability to spawn external consoles at will, there are some things (RGL comes to mind) that didn't like the RExecServer setup very much.

I'll try to post more often and get that source up somewhere. :-)

R 2.7.0 Quartz graphics device clipboard support

2007-10-05T16:09:00.000-07:00

Tom Elliot recently posted a request to the R-SIG-Mac list looking for a way to programmatically put graphics onto the clipboard under OS X. Right now this is pretty hard, but I was inspired to patch R-devel's (the future 2.7.0) new Quartz device to allow for clipboard output.

After applying the patch, you can specify file="clipboard://" and when the device is closed (yes, you MUST close the graphics device to see output. It would probably be a good idea to write the file on a NewPage as well, come to think of it). Then you can wrap up a simple function to get copying of the current device to the clipboard (or you can simply open it directly).


copy.to.clipboard = function(dpi=300) { dev.copy(device=quartz,type="png",file="clipboard://",dpi=dpi);dev.close(); }

I've attached the patch for the brave souls willing to try it out:


Index: qdBitmap.c
===================================================================
--- qdBitmap.c  (revision 43081)
+++ qdBitmap.c  (working copy)
@@ -49,16 +49,45 @@
         /* On 10.4+ we can employ the CGImageDestination API to create a
            variety of different bitmap formats */
 #if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_4
-        CFURLRef    path  = CFURLCreateFromFileSystemRepresentation(kCFAllocatorDefault,(const UInt8*)qbd->path,strlen(qbd->path),FALSE);
-        CFStringRef type  = CFStringCreateWithBytes(kCFAllocatorDefault,(UInt8*)qbd->uti,strlen(qbd->uti),kCFStringEncodingUTF8,FALSE);
-        CGImageDestinationRef dest = CGImageDestinationCreateWithURL(path,type,1,NULL);
-        CGImageRef image = CGBitmapContextCreateImage(qbd->bitmap);
-        CGImageDestinationAddImage(dest,image,NULL);
-        CGImageDestinationFinalize(dest);
-        CFRelease(image);
-        CFRelease(dest);
-        CFRelease(type);
+        CFStringRef pathString = CFStringCreateWithBytes(kCFAllocatorDefault,(UInt8*)qbd->path,strlen(qbd->path),kCFStringEncodingUTF8,FALSE);
+               CFURLRef path;
+               if(CFStringFind(pathString,CFSTR("://"),0).location != kCFNotFound) {
+               CFStringRef pathEscaped= CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault,pathString,NULL,NULL,kCFStringEncodingUTF8);
+                       path = CFURLCreateWithString(kCFAllocatorDefault,pathEscaped,NULL);
+                       CFRelease(pathEscaped);
+               } else {
+                       path = CFURLCreateFromFileSystemRepresentation(kCFAllocatorDefault,(const UInt8*)qbd->path,strlen(qbd->path),FALSE);
+               }
+               CFRelease(pathString);
+               
+               CFStringRef scheme = CFURLCopyScheme(path);
+               CFStringRef type  = CFStringCreateWithBytes(kCFAllocatorDefault,(UInt8*)qbd->uti,strlen(qbd->uti),kCFStringEncodingUTF8,FALSE);
+       CGImageRef image = CGBitmapContextCreateImage(qbd->bitmap);
+               if(CFStringCompare(scheme,CFSTR("file"),0) == 0) {
+               CGImageDestinationRef dest = CGImageDestinationCreateWithURL(path,type,1,NULL);
+               CGImageDestinationAddImage(dest,image,NULL);
+               CGImageDestinationFinalize(dest);
+               CFRelease(dest);
+               } else if(CFStringCompare(scheme,CFSTR("clipboard"),0) == 0) {
+                       //Copy our image into data
+                       CFMutableDataRef      data = CFDataCreateMutable(kCFAllocatorDefault,0);
+                       CGImageDestinationRef dest = CGImageDestinationCreateWithData(data,type,1,NULL);
+                       CGImageDestinationAddImage(dest,image,NULL);
+                       CGImageDestinationFinalize(dest);
+                       CFRelease(dest);
+                       PasteboardRef pb = NULL;
+                       if(noErr == PasteboardCreate(kPasteboardClipboard,&pb)) {
+                               PasteboardClear(pb);
+                               PasteboardSyncFlags syncFlags = PasteboardSynchronize(pb);
+                               PasteboardPutItemFlavor(pb,(PasteboardItemID)1,type,data,0);
+                       }
+                       CFRelease(data);
+               } else
+                       warning("Not a supported scheme, no image data written.");
+               CFRelease(scheme);
+               CFRelease(type);
         CFRelease(path);
+               CFRelease(image);
 #endif
     }
     /* Free ourselves */

Those of your tracking R-devel may have noticed...

2007-09-21T15:07:00.000-07:00

...that there was recently a big change to the way the Quartz device works. Turns out, Simon and I had roughly the same idea about a Quartz 2D implementation of the Quartz device for reasons of speed and the combined result (I ran "patch" on my R-devel tree and then Simon did all of the hard work :-) ) is in there now. This means the RExecServer and other R GUIs can now all use the same drawing backend for a variety of sources. I believe, for people who don't want to use RExecServer, that Simon also put in a CGLayer-based Cocoa target that runs an event loop in much the same way as RExecServer to give you interactive graphics natively in v2.7 (I'm guessing, since the feature freeze has passed). It also means native access to all of the Quartz 2D bitmap generation goodness and a lot more speed for ALL of the graphics device implementations.

Simon also went through and added some other features as well, including support for non-square pixels and some scaling things that I never got working correctly.

So, things have been moving along even though it may appear that I dropped off the face of the planet for a while. I'm going to be updating RExecServer soon, which means a dependence on R-devel, to use the converged device and it's new API (which is also available to any GUI developer).

Not Dead

2007-09-04T00:55:00.000-07:00

Just busy trying to get some software sorted out before feature freeze. Turns out to be a lot of work when you break R. :-)

Cover Flow!

2007-08-17T22:40:00.000-07:00

Yup. Figured it out. One of two ways to browse display plots. There is a more useful mode, using an iPhoto-like Browser that also lets you edit the list of plots. The arrow keys also let you move between plots.

iWork '08 Spelunking.

2007-08-16T16:03:00.000-07:00

Fiddling around with iWork '08 a bit and Numbers support is trivial to add to the R GUI. Basically, it exports a tab delimited version as text, much like Excel, though it uses the more modern pasteboard type (where Excel exports a much less useful string). We can detect that it is numbers from the metadata and "native" pasteboard types and throw up the same dialog as for the Excel pasting. Yay!

I'm thinking, for symmetry, that I should let you right click on a data frame in the workspace and copy as a tab delimited type for pasting into Excel/Numbers as well.

OLPC Musing

2007-08-10T14:25:00.000-07:00

So, I've been playing with the OLPC a bit (I got a chance to play with a prototype in person at a party a while back) and I wonder: where's the science/mathematics? Authoring languages and EToys, which seem to be the focus presently (well, that and Tetris of course) are all well and good, but it would be really nice to see some more focus on the physical sciences, which leads naturally into topics like data analysis and statistics (collecting data is just the first step after all).

The minimal implementation would seem to be to start with something like the TILE/StatDocs projects that Duncan Temple Lang (Duncan, I hearby dub you DTL for the remainder of my posts. Sort of like DHH, the Ruby on Rails guy) and Deb Nolan have been doing for a while now (there used to be a StatDocs.org website, but it seems to have disappeared). They give you an organizational unit of a "Lab" or an "Experiment," with some data and ways of interacting with the data (plotting and so on). These'd probably be hosted on the school's server and brought across the network dynamically. Another interesting tack would be to let the server also COLLECT data from the kids' experiments to be combined together into larger datasets via surveys or actual data collection. An ad hoc version of this is pretty common in introductory statistics courses for example.

This leads to the question of what to use as the analytical environment. Being biased, I would tend towards using R + GTK + Gecko, with a UI designed for the more modal experience of the OLPC. Alternatively, Python could be used for the implementation, though I'm not convinced that Python would be a particularly good interactive data language (things like R/S's formula DSL are really quite powerful for example). I tend to dislike point-n-click on principal, especially in an environment that stressed authoring languages.

Beyond that, you want kids to be able to collect their own data. I remember having this thing for my Atari 800 when I was a kid that had a number of different probe options for things like temperature and pH and rainfall and such. The software it had was pretty primitive, but it was lots of fun (for me anyway) playing with the temperature probe and such. It seems like the USB ports could be used to build data acquisition systems pretty cheaply these days---you wouldn't need really high performance for the most part and that lets kids collect their own data, which will always be more compelling than some prepackaged dataset since they "own" the data. There's at least one, possibly more, Open Source hardware project for building lots of these sorts of sensors for DIY weather monitoring that can probably be leveraged for that sort of thing.

RExecServer can load help

2007-08-04T19:11:00.000-07:00

It's not the best implementation in the world---I still think that should take place at the R level not in the GUI implementation---but on the plane to Seattle I gave RExecServer the ability to load help via the pager instead of just complaining.

Using RExecServer from ESS

2007-08-03T13:06:00.001-07:00

In the long term I hope that most of this can be configured by an Installer of some sort, but for those wanting to use RExecServer with ESS, here's how my setup is currently configured:

When I use Emacs at all (which is very rarely these days, I mostly use TextMate) I use Aquamacs, which ships with ESS installed. Hopefully the instructions won't be all that different for people using Carbon Emacs and what have you (and if they are, perhaps people would be so kind as to post their changes). I'm also not a super-sophisticated ESS user so there might be better ways of doing the ESS side of things.

First, though, we need to get ourselves easy access to RExecServer from the command line. I like to do this via a symbolic link in /usr/local/bin (/usr/bin would also work). You'd think that we could just symlink RExecServer.app/Contents/MacOS/RExecServer and be done with it, but the way OS X starts applications is... uh... strange and doing that will cause all sorts of problems because the bundle loader won't be able to find any resources and promptly crash out. It's not pretty. What we need to do is actually set a special environment variable called CFProcessPath to point to the application bundle rather than /usr/local or whatever.

Fortunately, I made this into a little script so you won't have to do it yourself. It lives in the Resources folder of the application bundle. So, from Terminal we would do something like:


$ cd /usr/local/bin
$ sudo ln -s /Applications/RExecServer.app/Contents/Resources/RExecServer.sh R-exec

if you have put RExecServer.app somewhere other than Applications, use that path instead of "/Applications." That's pretty much all there is to it. Of course, "/usr/local/bin" should be in your path


$ R-exec

R version 2.6.0 Under development (unstable) (2007-07-14 r42234)
Copyright (C) 2007 The R Foundation for Statistical Computing
ISBN 3-900051-07-0

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>

You should be able to use R-exec from Terminal just like normal R. For ESS, I have a line in my .emacs file:


(require 'ess-site)
(setq inferior-R-program-name "R-exec")

that starts my RExecServer version of R instead of the normal version. Once upon a time, I think I read that ESS is smart enough to detect multiple installed versions of R with specially named symlinks, but I don't know how to get it to work (if an ESS expert knows how to do this, please chime in).

In any case, that's pretty much all there is to it. Hopefully that helps, but if you have questions please email me or (better yet so others can see) leave a comment.

Oh, yeah.

2007-08-02T14:39:00.000-07:00

I did figure out one thing in an idle moment on vacation last week. How to paste from Excel to the R console. Thought y'all might like something like that. I should probably do another featurecast soon to show some of the things that have been fleshed out a bit. These include...

Multi-level hinting. If you have "plot(foo()," the hinting mechanism will show you "plot," rather than nothing

Drag-n-drop from the workspace to the console gets a summary() on those objects

Excel (or Excel-like things I suppose. Anything that posts VALU or NSTabularPboardType) pasting

Device recording, with the option to deactivate as well as clear plot listing

Clear the console

Copying text from the console optionally does so in a "source()-able" manner. i.e. prompts and output are removed.

Keybindings for popular things like Command-= to give you "<-"

TextMate-style brace handling. i.e. select some text and type "(" (or ",`,',{,[,Command-[) and it will put the appropriate bracket around the selected text.

An editor. :-)

Preferences :-)

Any requests?

I'm sure there's other stuff I've forgotten at this point, but I'm still unburying myself otherwise and getting ready to head to BioC'07 in a couple of days.

Quick Post

2007-07-31T13:44:00.001-07:00

Back from vacation, got a request for instructions on how to set up ESS to work with RExecServer. Consider me on that... right after I dig myself out of this email hole. :-)

And We're Back!

2007-07-19T20:17:00.000-07:00

Okay, apparently Blogger decided that this was a spam blog for a while there and I didn't notice because I had other things to do this week. All sorted out now it seems. To celebrate, I updated the leopard branch of the RExecServer repository to activate bitmap graphics output to files, thusly:


quartz("public.jpeg:Rplot.jpg")

Supported UTIs are public.jpg,public.tiff,public.png and public.jpeg-2000. public.pdf will be coming along in the not too distance future. This is actually a side effect of moving from using raw bitmaps in the GUI to using png images. Your average statistical graphic is usually pretty easy to compress so you can get from ~500K for a raw image down to ~20K pretty easily. On the server side it doesn't matter much because we only keep one bitmap for each device, but on the GUI we're keeping a potentially large number of pages that can eat up RAM.

To people tracking the repository, the leopard branch is really no different from the master branch. I just push work done in my Leopard partition there before migrating to the master branch with a test run in Tiger. The leopard branch will work in Tiger unless I've made a mistake.

In other news, posting will be spotty of the next week. I'm heading down to Los Angeles tomorrow afternoon and then down to San Diego on Wednesday and Internet access will come and go.

RExecServer git repository update

2007-07-15T13:29:00.001-07:00

The master branch of the git repository is now up-to-date with the leopard branch, minus any leopard specific code (of which is there none at present). R-devel is required to get things to compile. The GUI code isn't available yet, I'd like to get things to a truly usable stage first. I haven't had as much time this week, other things have taken priority, but some minor stuff was added such as the beginnings of the preference panes (and the infrastructure to support them) as well as the start of the action menus. I also changed the graphics device plot selector to take advantage of available space. Perhaps I'll post a screencast of that soon.

In the meantime, people who aren't using OS X, but want to consider a similar GUI type of set up might want to look at the Omegahat CORBA stuff, which did something very similar a long time ago. The page hasn't even been updated since before OS X 10.0 was released. :-)

Just a couple minor updates

2007-07-10T23:33:00.001-07:00

Doing some other things today so only a couple of minor updates watching The Tour this evening:

Removed the mouse-sensitive pages menu. This seems to be pretty disruptive without putting some extra delay into the transition, but the delay makes it too slow. Instead, I'm now using the toolbar lozenge to show and hide. I think I'll also hook it up to middle-click.

Implemented some display transformers for the Workspace. There's a (localized) "n objects" on the bottom now. The class and size information for objects is now shown (could probably be prettier!) as well.
Updated: Oh, yeah. You can also clear a console with Cmd-L now (the current prompt is restored). I saw that feature request on R-SIG-Mac a couple of weeks ago and it was something like 5 lines of code so I tossed it in.

R GUI Screencast

2007-07-09T23:53:00.000-07:00

Some of the new GUI features are hard to get from screenshots so I decided to make a little screencast to show off a couple of the new features.

Starting to think about help...

2007-07-07T00:11:00.000-07:00

You know what would be awesome? If the HTML manpages generated by R were actually microformat-enabled HTML. I was just thinking about how it would be nice if you could identify example code and whatnot from the eventual help page. I wonder how scary that code is? IIRC the HTML generated by R's manpage generator was pretty much HTML3 (so not even easy to style) so it could be pretty scary. It may even be one of the Perl bits.

Okay, I need to get some sleep now.

Making Progress on the Console

2007-07-06T22:48:00.000-07:00

Making some progress on the GUI's Console implementation. You can see that we now indicate WHICH server we're talking to at the moment. I've added an attached "workspace" inspector in the form of a source list such as the ones you see in Mail and iTunes. This holds information about the objects for inspection or dragging and dropping between servers. The thin black line is a splitter bar and is user selectable (double clicking collapses it entirely).

Both sides now have a status bar (of course I made sure the shading matched the action button... why do you ask?) with an action menu. As well, the hint is back and is now being powered from a new object inspection infrastructure in RExecServer (I have some other code to work on, but I will hopefully mind a few minutes to finish things and do a git push). We're now parsing out the function information rather than capturing string output so I think we can do completion in the style of Xcode or TextMate where, say, lm completion would result in:


lm(<#formula>,<#data>,<#subset>,<#weights>,<#na.action>)

I think Ctrl-/ should advance and that TAB (perhaps, I'm open to suggestion) should complete the editing. In other words, imagine we had...


lm(X ~ Y,mydata,<#subset>,<#weights>,<#na.action>)

then TAB would clean the other three arguments and jump us out of the function. You can't see it, but the console is actually doing a lot of invisible markup that we can use to detect the completion regions. We're presently also using text attributes to track input, prompt, error and output regions. This means we can easily cut and paste text WITHOUT prompts (which has been a major feature requests) and without ambiguity. We can also save a console script as a sourceable .R file for later use (sort of like dribble) saving the previous output as comments or not as all.

Oh Good, Performance Doesn't Totally Suck.

2007-07-05T22:24:00.000-07:00

Been doing a bit of performance testing this evening on the R GUI with RExecServer. Qualitatively, on my year old MacBook Pro, it seems that nothing has changed. At least, the overhead incurred by drawing the graphs or outputting the text seems to outweigh the communication overhead by quite a bit. This may change when hinting gets going, but thus far there doesn't appear to be a problem. Additionally, we have plenty of space for optimization in the distributed objects interface since we're not defining protocols yet. Good good good.

One thing does worry me: shipping bitmaps across the wire. I *think* that NSData it is actually optimized to use shared memory, but I'm not positive. In any case, the bitmap size of a full screen graph is pretty huge. That said, we could always do some timing (hey, we're statisticians, right? Staunch empiricists.) and then compress bitmaps above a certain area for transport via libz or something suitably fast. Hell, RLE would probably be a big win in most plots. I know you're thinking "use PDF," but PDF rendering performance is really awful relative to compositing to a bitmap. (Witness complex scatterplot in your Keynote presentation. There's a reason it takes 30 seconds for your graph to come up.)

Ha Ha! Success!

2007-07-04T20:17:00.000-07:00

After several hours of missing one very important line of code in RExecServer, which was preventing it from operating in purely vended mode---it was a delegate problem and the TerminalDelegate would replace it---I've gotten RExecServer executing and communicating with an R GUI.

The GUI is Leopard specific, but I can offer some details so far:

We use a normal NSDocument with no modifications to NSDocumentController or NSApplication or any of the questionable things I had to do for the pre-RExecServer GUI implementation

Devices are considered to be views of the document so devices and the console are explicitly intertwined now. In much the same way that the main Interface Builder window and the associated UI views are linked. Hopefully this means an end to the infamous "hanging" windows.

Scripts won't be, but we'll need a UI for specifying their target console...

Apparently I'm on a tear...

2007-07-03T15:38:00.001-07:00

Rob Goedman gets the Official Tester Award for RExecServer, reporting several issues with the graphics subsystem and my inability to appropriately use version control systems. There are a bunch of changes in the git repository:

All the files you need are actually there

Clipping rects are restored on subsequent updates so some things are working better

Terminal support has been refactored into a pseudo-GUI to help make sure I have coverage on the things I'll need for a true front end. Its probably fractionally slower than a true Terminal version of R, but it has some more flexibility

Device windows close when devices go away. A good thing too because they'd crash if you resized them after the device disappeared.

I've been playing with making the RGUI_Type not be AQUA, which restores all of the help file functionality, but causes an annoying (and untrue) complaint from quartz() among other things like trying to use X11 for select.list(). I wish there was a way to selectively deactivate things like that in R instead of the blanket situation we have now. There are a number of places where we have


if(.Platform$GUI == "windows" | .Platform$GUI == "AQUA") ... else ...

that are just irritating. Personally, I think we should be doing dispatch (S3, S4, I don't care) where .Platform$GUI becomes an _object_ so that I can define functions for my particular GUI and where .default is the stuff on the RHS of the else.

RExecServer object vending

2007-07-02T16:50:00.000-07:00

The RExecServer started getting basic object vending today. To test it out I implemented some simple object copying using Distributed Objects. This is checked into the public git repository, but will require R-devel to run because of changes to function export under R-devel. Behold!

First we start ourselves a couple of copies of R. Then,


> x = 1:100
> .Call("RES_CopyObject","x","R Execution Server 2")
NULL
> .Call("RES_ServerName")
[1] "R Execution Server 1"
>

Afterwards, we can take a look at the other execution server:


> ls()
[1] "x"
> .Call("RES_ServerName")
[1] "R Execution Server 2"
> x
  [1]   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
 [19]  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36
 [37]  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54
 [55]  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72
 [73]  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90
 [91]  91  92  93  94  95  96  97  98  99 100
>

Ta Da! There's no big trick to it--we're just using the standard serialization routines to read and write NSData objects on the Cocoa side and then using the usual NSDistantObject routines to actually transmit the data. There's no real error checking at the moment, but that will come.

Also, with the help of some scripts from Rob Goedman I think I've tracked down the last of the clipping and state stacking errors in this latest checkin.

RExecServer available as a git repository

2007-07-02T09:53:00.000-07:00

The RExecServer source is now available as a git repository. Once I figure out what exactly they mean I will be "mobbing" it to allow for patch submission. The binary hasn't been updated, but the source fixes a few reported problems with the pager() and some graphics issues (clipping mostly). It also adds the ability to specify the usual command-line options except for the gui related ones (for obvious reasons).

RExecServer Preview

2007-06-30T14:30:00.001-07:00

Posted last night to R-Sig-MAC:

I've mentioned this little project to a few people off-list as something I'd like to do for Leopard, but it occurred to me that there is nothing particularly Leopard specific in this particular piece of code. So, here we go:

RExecServer

This project mostly comes out of a conversation I had with Stefano about using multiple cores in the R GUI and Xgrid. The RExecServer is a first step in that direction. It provides a true Cocoa application that runs as a background server (so no Dock icon or menubar). The user (that's you) communicates via either a normal stdio connection (i.e. Terminal or ESS) or using Distributed Objects (for the GUI). In this initial implementation only the stdio access is working.

To use it I recommend symlinking the shell script RExecServer.app/Contents/Resources/RExecServer.sh to something handy in either /usr/bin or /usr/local/bin so that you can get at it from ESS or the Terminal. Then, just use as normal.

Things you get:

Mostly a working, fully responsive, Quartz device. This Quartz implementation is actually completely new so you may notice that certain things are different. Particularly, the font metric calculations are now improved---note the location of elements in plotmath (particularly sum and product). Right now aliasing is turned off, but that will be an option (I was experimenting with something). It also doesn't update the screen until it's done processing so while it feels slower it is actually much faster. There might be a few clipping bugs, but we'll sort those out.

A normal readline-based interface that can be used from ESS or Terminal. You can also start multiple copies, though it presently complains about Services. This is harmless though.

Very low CPU usage when idle. I'm forced to use polling with readline, but it doesn't appear to use very much. The event loop works differently in this version so there is no need for a timer or anything.

I'm not sure how much time I'll have, but here's what the design buys me:

We can pipe bitmap and pdf output through the quartz device. This means no more X11 required. Right now this isn't working, but the infrastructure is in place.

We can separate the GUI and R itself. This has pros and cons but I think it will be a long term advantage, especially as we get more cores.

Things I'd really like to do (again, time):

Copy-n-paste objects between Servers. Using serialize/deserialize and Distant Object or NSPasteboard connections

Quicktime movie output device. This might wait for Leopard.

If you poke around the link above you might find some other ideas. :-)

What this isn't:

Intended as a complete GUI. That's mostly for the front-end implementation which is a separate application. The graphics device is intended to be very minimal for ESS users who want something better
than the old Aqua device.

Anywhere near complete. You don't get lots of things right now. Like command-line options. Any options at all really. Lots of safety things aren't wired up either.

Let me know what you think and if you run into major trouble. The build is Universal so it should also work on PPC.

An improved flowSet idiom?

2007-06-28T23:20:00.000-07:00

In flowCore, a flowSet is associated with an AnnotatedDataFrame that contains ancillary information about the frame. This seems really useful, but it's really only used by the flowViz package---there really isn't anything remotely resembling a useful interactive idiom. Now, we could use subset(), assuming we could ever get the generic working properly, but we already have Subset in flowCore and the opportunity for confusion is high. One idea that has occurred to me is to take advantage of the ellipsis argument in [ and [[ to let us say things like

patient[[CellType="B Cells"]]

to extract the flowFrame identified by the CellType column.

If I Only Had The Time....

2007-06-26T22:53:00.000-07:00

Other things I'd like to do with R if I had the time (if someone else needs a project, I won't mind :-) )

1. A JITer for R to extend Luke's Bytecode stuff. I would say something along the lines of a binding so, say GNU Lightning. Now that function pointers can exist as R objects for use in .Call you'd create an EXTPTRSXP that protects the RAWSXP holding the generated code.

2. Finish up my libffi interface for R. I wrote one of these just after DSC2003. It probably still even works since libffi doesn't drift very much. It would probably be nicer if it was integrated with TypeInfo though. It also let you write R functions that appear to be C function pointers (for use as callbacks for example), though this has issues in multithreaded environments.

3. A centralized object database. One of the things I actually like about S-PLUS is the persistent database notion. I often have little pieces of code (for example my alpha function) that I stash away in scripts and places and then promptly lose. It would be nice to have a little database, perhaps with versioning, that you could easily tag and search. Hell, it could even sync with something online.

4. GData for R. I think it would be cool to be able to access Google Spreadsheets from R. It could be a pretty slick way of distributing data easily. If there was a way to hook it up to Google Docs for documentation and description through the help system that would also be cool. You'd probably have to use RCurl as the back end to get https support. I've started this one a couple of times but I don't really have a pressing need so I end up putting it on the back burner.

5. A complete dbxml interface. Again, I have chunks of this one, but never finished it (I ended up just using pipe() and the command-line tool). DBXML is pretty handy if you have a massive XML file (say a FlowJo workspace) and you only need a teensy tiny little chunk (like the gating strategy).

6. R on Rails! Okay, that might be a little silly (though I wonder how I would do R mixins...)

7. GPU backend. Actually, with the advent of CUDA you could probably do this pretty easily a la the Matrix package.

8. R/Flash (or Flex) interface. Plots as SWF anyone? Do it right and you could use it to serve up things in Flex/Apollo apps. I suppose you could also do a XAML one?

Old Things New Again: Multiple Evaluators for R Under OS X

2007-06-25T22:00:00.000-07:00

This post is mostly the result of a conversation I had with Stefano Iacus a couple of weeks ago at WWDC. He was making the observation that a) he would like to be able to run multiple copies of R from the R GUI and b) that he would really really love to run R evaluators over XGrid.

The second one might be harder, but I think the first one can be solved. Ideally, we would simply be able to spawn off multiple R evaluators in separate threads within the R GUI and apart from synchronization problems in the GUI we would be good to go. However, I rate the chances of R becoming thread-safe (let alone supporting multiple evaluators) any time soon as "slim to none." Of course, I'm not on R Core so I could be wrong, but from what it would take (every function in every package would need an extra argument for starters) it seems unlikely. The way around this? Spawn off separate R processes and connect them up within the R GUI. This is basically what people do when they use R from Terminal so there is no real disadvantage compared to the current methods and a lot of potential advantages.

So, the plan:

1. Implement RExecServer as an LSUIElement application. As much as I'd like to use Leopard-specific features here (garbage collection in particular), people using Tiger have multiple processors too. So, we're stuck with autorelease pools for now.
a. Vend an interface as a NSDistantObject that can be picked up by the GUI
b. Provide a threaded stdin reader (if TERM is set) using the "traditional" R reader. This is to provide ESS support. I think we can actually vend the object and allow the stdin reader at the same time. Er, this could be a cool feature for something we'll talk about in October (if all goes well, this isn't my day job so it only gets implemented when I have some spare (hah) time) :-).
c. Theoretically, we could vend to/from different machines. Using Bonjour you could publish your R session. We'd have to work out some sort of security model. Not sure how that's normally handled by NSDistantObject.

2. Change the graphics device a bit. Mostly I don't think we want to ship around the graphics list. In general, we can ship a bitmap that is appropriate for the display. We can also ship back a PDF if so desired, but performance with a bitmap is likely to be higher with no discernible quality difference except in special circumstances. We can just have a [Device dataInFormat:] that sends back an NSData of the appropriate format (RGBA bitmap, PDF, etc). The nice part is that both the client and server are running OS X and both have access to the Font metrics so there doesn't need to be communication there.
a. On the ESS thing again, we can provide a simple shim device window to give ESS users a decent graphics device with full interaction. It won't be as cool as the one in the GUI, but that's what you get for using ESS ;-).

3. The GUI now maintains connections to any number of GUI console/device windows.
a. Stefano suggested being able to copy and paste between environments. I think this can be done using private Pasteboards and serializing objects to RAWSXP types, converting them to NSData and then transferring them over.
b. The GUI itself never need become unresponsive. You could even force kill a runaway server.

Now, certain things get more difficult. Certain GUI toolkits, like my own Mojave, won't be running in the GUI process anymore making them difficult to write GUIs. Of course, nobody that I know of writes GUIs using Mojave (and you'd think I'd know), but this also rules out everything else. Personally, I think the way around this is some sort of Dashboard-style interface where the front-end is implemented in HTML and Javascript with hooks back into R using a special protocol handler (which is apparently an SDK these days...) or a Javascript proxy to allow execution. I tend to favor the protocol handler.

Interactive Gating with flowCore

2007-05-25T16:32:00.000-07:00

One of the biggest hurdles for flowCore adoption is probably the lack of interactive gating. People really like it and it can be useful even to us statistician types that are suspicious of the entire exercise. R and flowCore, unfortunately, aren't particularly good at interactive graphics so this presents something of a problem.

To help with this, here's a quick function for getting a 2D polygon gate with an optional transform:


interactiveGate = function(fcs,x,y,filterId="Picked", trans=NULL,...) {
  if(is.function(trans)) {
   tnf = structure(list(trans,trans),names=c(x,y))
   trans = do.call("transform",tnf)
  }
 plot(exprs(if(is.null(trans)) fcs else trans %on% fcs)[,c(x,y)],
  pch=20,cex=.5,main=filterId,...)
 points = locator(type="n")
 polygon(points)
 points = cbind(points$x,points$y)
 colnames(points) = c(x,y)
 r = polygonGate(filterId,points)
 if(is.null(trans)) r else r %on% trans
}

The fcs parameter is a flowFrame and the x and y arguments should be pretty obvious. The trans argument can be one of three things: NULL, a function, or a transform object. If it is a transform object (created by transform(...)) then it is simply applied. If it's a function, a transform will be created for both parameters and also applied to the subsequent gate.

e.g.


interactiveGate(fcs,"APC-Cy7-A","PacOrange-565-A","Control",function(x) asinh(x/32))

Just a little thing, but pretty useful.

Is it time for Q?

2007-04-30T17:05:00.000-07:00

There was an interesting post over at Jim Hugunin's blog today announcing the Dynamic Language Runtime, which appears to be a refactoring and standardization of the IronPython and Javascript implementations on the CLR. The stated goal is to support the development of dynamic languages atop the CLR, freeing implementors from garbage collection issues and whatnot.

Considering Duncan Temple Lang's recent talks about system interoperability (challenges, advantages, etc) it seems like this might be a nice way to have an integrated statistical environment that can be bound to the sorts of things software engineering projects have come to expect but happen to be pretty tedious to implement (a decent network library, etc).

As a side note, we probably can't call it Q since a language of that name already exists in active development.