cgwalters

Jython about to hit 2.5

Thu, 26 Jun 2008 17:47:46 GMT

An excellent update from one of the Jython hackers (and followup). I particularly liked this:

>>> from __future__ import GIL
Traceback (most recent call last):
(no code object) at line 0
File "", line 0
SyntaxError: Never going to happen!

For me the most useful thing is that Jython SVN now supports decorators. I use decorators fairly extensively in my Python code, and they were the main thing blocking me from using Jython more widely.

Some people may wonder - why Jython? One reason is that the tooling around the JVM is awesome. JDWP (and clients like Eclipse's debugger) blow away gdb. For a while now I've been trying to debug a periodic lockup in a multithreaded Python process, and it's been immensely painful with gdb. There are a lot of other extremely useful FOSS tools for heap analysis, profiling, etc. Even if your entire world currently runs on Python, and you aren't interested in easily taking advantage of some of the other awesome JVM software out there like Hadoop, running on Jython makes sense alone just for OpenJDK being among the best Free VMs out there.

The forehead mashing method

Wed, 25 Jun 2008 23:03:18 GMT

I liked this blog entry from Planet KDE. That entry reminded me to talk about one way I like to think about how to improve the user experience with software - and that is by what I call the "forehead mashing" method.

This is where as you're developing software, write out a few problems that you're trying to solve. Then, for every step the user is required to make - for every dialog that pops up, for every few seconds of keystrokes, every dialog to click, every checkbox to check...lower your head onto the desk and hit your forehead.

You don't have to do this with a lot of force. Just enough so that you feel it.

This calls for several head-desk hits

One of the top things that will quickly make your forehead sore are Wizards/Assistants. The way I think of these is that the programmer is so proud of their work and they think it's so awesome and complicated that they want you to acknowledge it.

What if programmers inflicted these kinds of things on ourselves? Say in gcc.

If you all keep making wizards, I'll create this patch for GCC

It's not just wizards of course that are the source of pointless user time spent reading text and clicking buttons; random dialogs are another source. Often with some thought, it's really not hard to reduce and simplify.

This is davidz's theme, don't blame me

So I promised in the comments in my previous post that I would do something to reduce the pain from the notification bubble I added; to keep the universe in balance as it were. So I'm happy to say that in Fedora rawhide now, the above dialog is no more. It was just a bad idea, and you will never see it again.

Some small tweaks

Tue, 27 May 2008 00:08:40 GMT

One of the things I've been looking at again recently is the initial experience for the desktop. Previously we investigated reducing the number of steps by starting the browser by default; incidentally I have plan for a more refined version of that kicking around that isn't quite ready but should hopefully be soon.

In the context of the Online Desktop work we want to get the user logged in online. However, this requires a network connection. Pretend for a moment you're new to the system. Look at a default desktop, say one from the Phoronix Fedora 9 review, without the obligatory window to 1970 in the middle.

How do you know what to do to get online? If you're using a wired connection then you don't have to know anything; NetworkManager does it automatically. However a much more common case now is wireless networks, and you would have to know to click on the "two computer and broken X" icon in the top right. A simple solution is to have a notification pointing to it:

How to connect to a wireless network

Cooking up the patch for that, I quickly ran into the problem though of notifications pointing to the wrong place. If you've used GNOME for long enough you probably know what I mean; on login sometimes getting a notification in the top left instead of the top right, or it could just be off by 10 pixels. This usually wasn't too bad because the notification was more about the content, but in this case a notification pointing to the wrong thing was actually worse than nothing.

Fixing this right ended up taking me though a number of layers in the stack, from the application displaying the notification, to libnotify, to GtkStatusIcon, to the notification protocol, and finally to the notification daemon, which needed a fairly big patch. However after spending a bit of time this weekend I finally got it to work:

Link to Theora video

Not earth shattering, but it has been a pretty longstanding bug in the desktop. It was actually sort of fun to get back to grubbing around with the low level of Xlib.

Polyglot programming

Sat, 10 May 2008 19:25:49 GMT

As you may have gathered from previous entries, I've recently become interested in programming languages again.
I'm almost done (though skimming some parts) with my copy of the aforementioned Ruby book. and though I haven't done any substantial Ruby programming (just playing around in jirb while reading), I think I have a good idea now why so many people love the language.

Closures

Most of my real code over the last number of years has been in C, Java, and Python, and I know those languages and their runtime libraries pretty well, but reading the Ruby book I was struck by how really useful closures can be in an Algol-family language (i.e. not Lisp). Well, Ruby calls them "blocks" and has infrastructure on top in the form of yield, etc., but that's fundamentally what they are. C/Java/Python all lack them (no, Python's single-line lambdas are too restrictive to count).
Closures...environment, get it?

Closures are incredibly powerful, in fact you might say they're the ultimate language construct. Neal Gafter has a good description of the kinds of things you can do with them, going from a language that doesn't currently have them.

Polyglot programming

While I was thinking about this blog entry, I reread one of Steve Yegge's great blog posts, and decided to look up more about his reference to the author of a design pattern book "leaving Java to go to Ruby". After going to Martin's home page, I found on his wiki he has a good entry which is pretty close to what I wanted to talk about. That is basically: it makes sense for large software systems to have multiple layers in different languages.

Now, if you're thinking "Wow, that's obvious", that's good; but there is more to the story here. So let's look at some rationales. If your app is large enough, you probably have parts which need to be fast. And you probably have other parts which cry out for a domain-specific language.

So for speed, you'll want a lower layer which is usually characterized by manifest typing and direct vtable function dispatch. Read: C++/Java/C#.

But it makes sense often to have a higher layer which is agile. It is better for the parts of your program which change rapidly - this could be user interface bits you're prototyping, or rapidly creating test cases. This layer is usually characterized by implicit typing (possibly with type inference), metaprogramming capabilities, and (ideally) good integration with the lower level language. Read: Groovy,JavaScript,Python,Ruby

There is a lot of software out there split in exactly this way; in fact, you're almost certainly reading this blog entry in one of them, Firefox, where the answer is C++ and JavaScript. A lot of computer games are built in this way too - for Civilization 4, the answer is C++ and Python, and for World of Warcraft it's C++ and Lua. If you're familiar with Java, just think about JSP and Ant - they're really DSLs. If you mostly know Python or Ruby, think about how much of the underlying platform is actually written in C/Java/.NET.

So it's fairly easy to dismiss anyone who says something like "everything must be written in language X", for values of X like C,Ruby,Python,Java. Which reminds me to say: Eclipse really needs to embrace Eclipse Monkey.

The impedance mismatch

So we accept that it makes sense to have multiple languages with different characteristics. One important issue then becomes - how similar are our two different layers? Taking the example of C++ and Python as in Civilization 4. The gap is enormous. C++ containers are not the same as Python containers. C++ strings are not the same as Python strings. C++ objects and Python objects are wildly different. The answer to this problem is to create a special glue layer; in Mozilla, it's called XPCOM. In GNOME, it's called pygobject. These layers are very painful to create and maintain.

An interesting question is - what if our two languages shared more? Do we really need to have separate container types just to get agility and dynamism? The answer turns out to be - no, which we'll get to in a minute. As we know, the fact that there are a lot of things that every modern language shares lower level components (like garbage collection, JIT compilation) led Microsoft to brand .NET as a multi-language runtime (as an aside, plenty of languages ran on the JVM long before .NET was created; for example Kawa, which dates to 1996). Now here's the thing, though. Running on .NET does not make Python objects same as .NET objects, nor does it make their containers the same.

What is an object?

Let's briefly take a look at what a Python object is. It's a fairly illustrative example of just how different languages can be. In Python, every object instance is by default a dictionary (hash table), with data stored in the __dict__ member. Every property lookup or method call has to in general traverse a chain of hash table lookups. At any point, some other code can come along and add a new entry in an object's dictionary:

#!/usr/bin/python

class Test(object):
  def __init__(self, a):
    self.a = a

t = Test("hello")
t.b = 42
print t.b

Supporting this level of dynamism is expensive, both in time and space, again because every object instance carries along a mutable hash table under the covers. It means you can't share very much between processes. It makes multi-threading much slower because everything has to be synchronized on that dictionary. Besides being expensive, it's almost never what you actually want, at least by default. You usually want t.b to be an error. This is by far my biggest issue with Python. In fairness to Python, it predates almost every other language discussed here.

Stealing and language evolution

Languages are clearly stealing things from each other, and evolving together. In the Ruby book they often mention how certain parts were taken from other languages. Java and C# are stealing ideas from each other. ECMAScript 4 is clearly rebuilding itself on a more JVM/.NET like class model.

What I've been looking at lately is a new dynamic language that has clearly stolen a lot of the good ideas from Ruby and Python, but is a lot more "native" to a modern runtime (in this case, the JVM): Groovy.

#!/usr/bin/env groovy

class Test {
  String a
}

def t = new Test(a: "hello")
t.b = 42
println b

This results in an exception about a missing b, because its idea of a class is exactly the same as the underlying JVM platform, where objects are much more static by default (this is also true of .NET). Note we can even declare types if we like (or we can just use def). I really like how default constructors work - it's even less typing than both Python and Ruby! It has useful closures, regular expression and hash table literals. Pretty cool. For a more complex example, here's an example of a fairly typical scripting task of log file processing I wrote a few days ago. I'm fairly sold so far, but there is still more to learn. I spent a bit of spare time poking at getting it packaged, but ran into some Maven bootstrapping issues.

More on languages

One random link: An awesome feature of Python is Generators, and if you aren't familiar with them and think of yourself as a "systems programmer", check out this very good slide set.

Second to last: some recent additions to my Google Reader feed: Charles Oliver Nutter, John Rose, Lambda the Ultimate.

As an aside that's not directly language related, but also new to my feed list is Why, who is like a great artist-programmer churning out amazing works like Shoes. Does anyone else have the feeling that for Why all of these code projects are just what he does in his idle time, and in the next few years he'll emerge from his underground hideout with an army of giant robots and take over the earth?

An even better OpenSSH interface

Fri, 09 May 2008 18:01:56 GMT

As I've mentioned before, in my spare time I've been working on a fun project to replace the Unix terminal and shell with native graphics and a high-level language runtime. One thing that I quickly found out though was that certain applications require a real VT100 terminal emulator; for example, less is a highly optimized program, and while replacing it with a proper X11 app would be possible, it would not be trivial and would largely be a distraction from my goals anyways. Another situation where you need a terminal emulator is connecting to remote Unix hosts; while I have complete control over the software on my laptop, it's hard to change all the existing servers out there.

SSH in particular is something that really cries out for a nice application around it, rather than launching it primitively from an existing terminal emulator. Earlier I mentioned the result of a few hours of PyGTK hacking. Over the last week or two I've been working on improving it. Today David Ascher of Thunderbird fame asked for it, so I should probably kick it out the door (though HotSSH is primarily for X11/Unix and not the Mac).

New connection with history

As you can see, the new version of HotSSH now remembers what you do, automatically. For example:

If you connect to a host and use a custom username, the next time you open HotSSH and type the host again, it defaults to that custom username, without you having to do anything

It has pretty good completion on the Host entry, based both on ~/.ssh/known_hosts as well as its own history

Open windows and tabs are saved automatically, Firefox style

The only dependencies are CPython, GTK+, DBus, and OpenSSH. I've broken it out of where it previously lived in the Hotwire tarball into a separate source tarball. You can currently get it from the Hotwire downloads page, though my long term plan is to import it into GNOME SVN and ideally have it come with GNOME out of the box. The code is currently here, and bug reports here. It works well enough for me now; about all it might make sense to add is some sort of color scheme (we currently just pull from the gnome-terminal defaults). To run, you can just unpack the tar and python ui/hotwire-ssh. If you install it, it should create a default bash alias for ssh, but this is something that package distributors will need to make sure integrates well.

How to share state with applications for Free Unix-based desktops

Sat, 26 Apr 2008 17:44:00 GMT

So you're writing an application. At some point, you have a problem where you want to be able to talk to an instance of your program while it's running. It turns out, this is fiendishly difficult for Free Unix-based desktops, and many programs get it wrong. On my Fedora 9 system, looking in /tmp, I see listings like gconfd-walters (GConf), hsperfdata_walters (OpenJDK), pulse-walters (Pulseaudio), emacs500 (GNU Emacs). All of these programs are broken. The root cause of this is the failure of Unix to provide a standard per-user, per-machine temporary directory. If we had that, things would be dramatically simpler, but we don't.

You can't use /tmp because it's not per-user. Any time you rely on a well-known name in /tmp (like the above list), your application is subject to denial-of-service attacks if another user creates that file before you log in. Security aside, it's also just harder to deal with because you have to check for whether or not the data in your temporary directory is stale or not (is something listening on that socket, or is it left over from a session crash?), and that's yet more painful code.

You can't use $HOME because it's not per-machine (think of all the sites that subject users to the pain of NFS), and won't always support Unix domain socket files.

Now, there are other solutions; you can use the X server, which is what Firefox does I believe. If your application is single-instance this isn't too bad, but there is an easier way:

Enter solution: DBus

The best solution is to use DBus. The key feature here is that the DBus session bus provides exactly what we want: a per-user, per-machine namespace, as well as a fairly sane IPC system we can use to pass messages. Here's how you use it.

Decide on a service name for your application. For the purposes of this discussion, we'll call it org.openssh.Ssh, for no particular reason.

In your application startup, try to acquire this name on the session bus, using DBus. If you're successful, your application can now provide a service. To pick a completely random example, this service might be multiplexing SSH connections. If something else already owns the service, then you can act as a client; you might send the message org.openssh.GetControlMasterSocketPath to return the path to another Unix-domain socket, because your application has a non-DBus protocol for communication.

In this example, we just used DBus to retrieve the path to another local, randomly-named Unix domain socket. That's perfectly fine - DBus is not the one true IPC system. Programs like Pulseaudio and OpenSSH already have defined protocols, and there is nothing wrong with that, we don't need to replace them with DBus. However, DBus is the best candidate for a one true atomic session namespace.

Again, all examples used in this blog post are completely hypothetical, and have no relation to the silliness of requiring users to try to hack around this issue on their own computers with variables like ControlMaster and ControlPath to enable a useful feature of your application.

OK Internet, I concede

Mon, 21 Apr 2008 16:44:00 GMT

I'm going to learn Ruby. My copy of The Ruby Programming Language arrives today.

Cute..

Tue, 15 Apr 2008 21:47:10 GMT

I see from this history meme that a lot of people are still using prehistoric shells! Low numbers like 303 or even 24 total times for top execution? Now, my shell knows how to store data properly, so for example it doesn't toss away your entire session if you don't exit it cleanly, and has scaled so far with near-instant searches for a 20000+ history size.

My code:

import os,sys,sqlite3
c = sqlite3.connect(os.path.expanduser('~/.hotwire/state/history.sqlite'))
cmds = map(lambda x: x[0].split(' ', 1)[0], c.execute('SELECT cmd FROM Commands'))
stats = {}
for cmd in cmds:
  count = stats.get(cmd, 0)
  stats[cmd] = count+1
cmds = stats.keys()
cmds.sort(lambda a,b: cmp(stats[b],stats[a]))
for cmd in cmds:
  print "%s: %s" % (cmd, stats[cmd])

Output:

cd: 2280
ls: 2003
ed: 1823
git: 1362
python: 1187
make: 817
svn: 751
sudo: 669
py: 543
cat: 442
rpm: 380
rm: 370
less: 366
cvs: 306

Various

Fri, 11 Apr 2008 15:44:02 GMT

Mailing lists

Benjamin discovers that yes, pretty much no one except hardcore hackers use mailing lists (my thoughts on them here). I don't see the disconnect between normal users and developers as "scaling" - it's not a good thing. I'll try to look at Fedora Forum more...must train self to ignore the flashing "NEW" icon.

New journal

I spent a little bit of free time a few days ago to make the Firefox Journal extension work again. As mentioned before, you'll need to create an AMO account to install because it's still in the "sandbox".

Now it needs some CSS love for sure. But, it works and I find it pretty useful for a few tasks, such as "go back to that new bug I was looking at 10 minutes ago".

Speaking of extensions...Password Maker is awesome. I've now redone most of my web passwords using it.

Account system

I've been thinking lately about how to improve the online.gnome.org account system. If you're curious, my proposal is linked from here

Languages

Clojure definitely shows that there is a lot of room left for innovation in programming languages. I'd only briefly heard of transactional memory before; this is something to learn.

ECMAScript 4 looks like a massive improvement over the old JavaScript. I, for one, will not miss the old bizarre prototype system.

A good post by one of the PHP authors on dynamic vs static languages (though he phrases it as Ruby/Python/PHP vs Java). I think my feelings on this lately are that if I were going to work on a website frontend process, it makes total sense to use Ruby or Python. But there's a lot more to the world than website frontends. For example, I don't think I would choose a dynamic language to write Hadoop or HDFS. The dynamic vs static debate is of course a very old one, but what is new is the dramatic progress of dynamic languages on VMs designed for static languages, which I was playing with before.

What I think people in these debates greatly underestimate though is the importance and power of the Free Software ecosystem, which could also be called the "default factor". /usr/bin/ruby has always been a simple yum/apt-get install away on Free operating systems if it wasn't already there; no license clickthrough web pages. That matters.

It's the community, stupid

Speaking of technology though, a lot of us sysadmins, programmers and free software hackers can get easily get caught up in the implementation details of our systems. Now for sure, some things are just technically better than others; for example, the JVM is years ahead of CRuby's green threads and basic bytecode interpreter. But what ultimately matters is the people behind the technology. It's easy to get confused because the technology names are staring us in the face; when you want to run your program, you type /usr/bin/ruby, and not /usr/bin/a-language-from-matz-and-and-now-a-global-community.

A key example where I see this happen often is in "packaging systems"; many people confuse the program apt-get, which is just an overgrown wget, with all the hard work that goes into packaging and maintaining software. In the beginning, people had to go through and make sure it worked. For certain types of software, it is actually hard and there's no getting around it; an example is JNA which deals with some lower-level JVM access. In that case, a lot of the work was done by two Fedora community members. And we even sent the patch for our work upstream, so that all Free operating systems can benefit.

Anyways, that's enough for a Friday pre-lunch update...

ToolKit

Wed, 09 Apr 2008 16:27:10 GMT

"What it lacks is an extensive screen reader like GNOME's Orca": Do you really think our time is best spent spending time reimplementing Orca just so we have something in Qt? Maybe someday someone will, but right now we have better things to do. This kind of mentality of "my toolkit or DIE!" is a disease. It makes us waste more effort and time when it isn't specifically needed. In this case, just use Orca with KDE4 apps.
-- Aaron Segio.

Agreed! -- Colin, who is currently using the official Last.fm client written in Qt because, hey it worked, and it's Free Software.

On standards

Wed, 02 Apr 2008 14:36:42 GMT

Very good post I saw while scanning Planet KDE. Which reminds me - besides keeping up with what's happening in KDE via their planet, other recent additions to my feed list are Stevey's Blog Rants and the Hadoop blog.

A perhaps less controversial plan for creating a better VM

Thu, 27 Mar 2008 17:51:17 GMT

The previous post spawned a lot of discussion, a lot of which was surprisingly technical and on-topic. But after talking with some people I realized that OpenJDK can do a lot just on its own. Here's my wishlist:

OpenJDK->OpenVM plan

Split up source distribution into OpenVM core, place things like Swing into separate source project

OS-specific integration in core; e.g. javax.unix namespace (e.g. Unix domain sockets), javax.windows (similar to python-win32), javax.osx; and allow interested operating system vendors to innovate there. The operating system does matter.

Commit to longer-term VM improvements necessary to allow compilation of C# into extended JVM bytecode (not CIL)

Commit to VM improvements necessary to make Jython/JRuby work well

Stay on top of Linux distribution integration, make sure packagers aren't carrying patches (this includes JSR 277 work)

Together with the above, branding as OpenVM or something similar to express willingness to be more than just the old "Java" which was JVM+Java language+Swing

An Open Letter to Jonathan Schwartz and Miguel de Icaza

Tue, 25 Mar 2008 17:25:53 GMT

Jonathan, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called OpenJDK.

Miguel, you are leading the development of a Free Software, high-quality, multi-language VM runtime with an extensive class library, called Mono.

How about a merge? We'll call the new project "OpenVM", for convenience in this letter.

Let's jump right in to the advantages for the projects:

Advantages for Mono

In one word - control. Miguel, your original goal with Mono was to bring a modern and Free Software development stack to GNOME and Linux. In many respects, you and the Mono community have been succeesful, helping spur the creation useful applications for the Free desktop, as well as getting Mono deployed in interesting applications like Second Life. However, you are largely not in control of your destiny. You're stuck implementing a clone of what Microsoft creates, and besides the fact that cloning something is much less fun for your engineers, you can't help but be behind.

By helping to create the OpenVM project, you will regain control. In an OpenVM effort, drawing on the common shared work of several corporations (Sun, Novell, Red Hat, Google, and IBM, to name a few), your engineers get to help design the future of Free Software. You will instantly remove all hesitation that the Free Software community has about your work, and have been the a key part of not one but two cornerstone projects for Free Software (GNOME, and Mono->OpenVM).

Advantages for OpenJDK

Jonathan, you have said you want to take the J out of JVM. By stepping up and adding Mono technology like a high quality C# compiler to this OpenVM effort, in the short term you will regain the eroding market share of the JVM on Windows by allowing interoperability between the growing C# code base and existing Java code. In the longer term, developer attraction to OpenVM will let you accelerate improvements to Java, and reverse gains in C# market share.
Moreover, the community agile languages such as Ruby and Python are nearly certain to join an OpenVM effort. Your company will again be at the core of the stack for the vast majority of the computing industry, from the Free Software community to the proprietary applications.

From the Free Software side, turning Mono from a Microsoft technology clone into a part of a truly Free project would eliminate the increasing use of .NET in the community.

Finally, leveraging the Mono team would bring a number of excellent engineers who know the Free desktop very well, having created high-quality bindings for GNOME, and Free applications that many people use.

Advantages for the Free Software community

The Free Software community has long been split between developers using Free and agile languages like Ruby and Python, the the Mono-based community, and a huge community of developers in the world who used in the formerly-proprietary Java in Free projects like Apache. A combined OpenJDK and Mono would dramatically further the merge of all three of these communities, increasing the control the Free Software community has over the stack and reducing duplication of HTTP libraries, database access libraries, etc.

Details

Obviously, there would be many details to work out in such an effort, like how the class libraries could be merged. My intuition is that initially OpenVM would have both JDK and .NET "personalities". Over time, the Mono .NET class library would be rebased on top of an evolved JDK class library, and eventually the .NET personality could be relegated to a separate "OpenVM-.NET emulation" project as most applications are ported to use the OpenVM JDK-based class library.

But the details are just that - where there's a will there is a way. So the open question is - who will register the domain name first?

Hotwire hypershell 0.721 released

Sun, 23 Mar 2008 19:55:27 GMT

Hotwire 0.721 is now available. This release features a lot of changes since 0.710. Immediately visible will be the entirely revamped UI.

New Hotwire 0.721 UI

Full screenshot with object inspector

The goal is to be closer to a shell/terminal interface than before, giving more space to the output of commands while still allowing use of the mouse for operations. Another exciting internal change is that you can now define Hotwire builtins as regular Python functions, but with a decorator. For more about this feature, see this post.

Besides the above, there are a lot of other nice changes in this release from a growing list of patch contributors, such as Zeng.Shixin's contribution of native file icons for Win32:

Native Icons on Win32

As well as Chris Mason's improvements to the command output search:

Search highlighting

I added a nicer connection status display to the included Unix SSH client:

Connection status display in HotSSH

Mark Williamson has been experimenting with a set of Hotwire extensions to make Hotwire into an interactive Mercurial shell; see his site.

For the detailed release email, see the announcement.

Four languages - one process

Sun, 23 Mar 2008 15:38:49 GMT

The previous entry was cut slightly short because I had to head out to dinner. What I was trying to finish was a bit of code to experiment with JRuby, Jython, Rhino, and the new JSR 223 scripting framework.

My first step was to build JRuby from trunk and the Jython 2.5 branch, as well as JSR223 trunk. After a small patch, I got some code working:

package org.verbum;

import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import javax.script.SimpleScriptContext;

public class LangFun {
  public static class Hello {
    public void sayHello() {
      System.out.println("Hello world!");
    }
  }
  public static void main(String[] args) throws ScriptException {
    ScriptEngineManager mgr = new ScriptEngineManager();
        
    Hello greeting = new Hello();
        
    greeting.sayHello();
      
    ScriptContext context = new SimpleScriptContext();
    context.setAttribute("greeting", greeting, ScriptContext.ENGINE_SCOPE);          
        
    ScriptEngine javaScript = mgr.getEngineByExtension("js");
    javaScript.setContext(context);
    javaScript.eval("greeting.sayHello()");
       
    ScriptEngine ruby = mgr.getEngineByExtension("rb");
    ruby.setContext(context);
    ruby.eval("$greeting.sayHello");
        
    ScriptEngine python = mgr.getEngineByExtension("py");
    python.setContext(context);
    python.eval("greeting.sayHello()");        
  }
}

To run this, you'll need to link to jruby.jar, jython.jar, rhino.jar, as well as the respective engines from JSR223: jruby-engine.jar, jython-engine.jar, js-engine.jar.
The idea behind this code is pretty simple - we first create a Java object, with a single method. Then we use the JSR223 interface to instantiate an engine for each of the languages, hook up a context object which maps the variable greeting into the global namespace for each language, then call their respective eval methods. The result is what you'd expect:

Hello world!
Hello world!
Hello world!
Hello world!

Pretty cool! Of course, I'm not really stressing the system here with these simple scripts; but given the dramatic progress (really, look at that graph!) of projects like JRuby, the future approaching very quickly.

"Ok, but...", you might say, "what's this useful for besides sharing libraries?" The general answer is that the single process, shared memory model is fundamentally more powerful than the multi-process, communicating over bytestream pipes model. You can just do more, with fewer hacks. An example is software like Reinteract - before, it could only support Python. Now, not only could you add an input language chooser to Reinteract; you could actually pass the results from a Python computation increasingly seamlessly into Groovy, Java, or whatever. Personally, I specifically want this for Hotwire. Right now we only support Python well, because the project is based on CPython.

Many new languages for the Free Software community

There was a comment on the last entry mentioning Scala. I've only looked at it very briefly. But this is just one of many languages that are now truly, finally part of the Free Software community. The Scripting project has a list of the engines written. But that list is far from complete, because some don't have the engine glue written yet, and because other languages like Scala are more designed as Java replacements that run on the JVM, rather than "scripting".

And of course through all of this, venerable Java isn't standing still - it will likely gain closures. And remember - all of this will soon be available (if it isn't already) via yum install or apt-get install, etc., all of it entirely Free Software, and increasingly integrated with the operating system and your favorite libraries.

A software tsunami

Sat, 22 Mar 2008 22:43:09 GMT

A large underwater earthquake ends up creating an effect called a tsunami. The event can be detected by sensors around the world, but the resulting tsunami isn't immediately visible; if you're in the surrounding ocean you'll notice it, but it's only when it hits land that you really notice the effect.

On May 8, 2007, there was an effect a lot like an underwater earthquake in the software world. What are we talking about? The complete release of OpenJDK, of course. Since that time, we've mostly been in the underwater propagation stage. A lot has been happening behind the scenes such as removing proprietary bits, fixing OS integration, etc. But now, I think we're close to moving into the stage where the ocean recedes, so you can see the first visible effects.

The original OpenJDK release was a snapshot of the in-development version 7, so it was not quite suitable as a drop-in replacement for software that was targeting JDK 6. But recently in February, Sun released the sources to the stable version of the JDK, version 6. Thanks to the combined work of the OpenJDK team and the IcedTea project, this is now suitable to effectively work as a drop-in replacement for the earlier proprietary JDK releases.

On Fedora 8, this command worked for me:

sudo yum --enablerepo=development install java-1.6.0-openjdk

It pulled in new versions of a few things like zlib, but no big deal. In any case I think you'll likely see OpenJDK 6 pushed as an update for Fedora 8 too.

Why is this so important?

It's pretty hard to underestimate the transformative impact Java and the JVM have had on the software industry in the last decade or so. Now, Java is a very well designed language, but what I think is equally (if not more) important is the JVM. The JVM was really pretty far ahead of its time; the optimizing JIT compiler, the class structure, the threading model, concurrent generational garbage collection, etc.

There has been quite a lot of innovation on top of that platform. Ok, that sounds a bit buzzword-y. How about this: People have written a lot of awesome software for the JVM.

The examples seem endless; but let's mention some:

Processing - a combination language and mini-IDE designed for creating visual effects

NASA's World Wind

Groovy, which is along with Boo in my opinion one of the top contenders for the next-generation of programming languages

MIT SIMILE projects like Welkin.

Hibernate - Best-in-class object-relational mapping

Eclipse

Hadoop - Free Map/Reduce implementation

Impact ahead

But before OpenJDK, most of these projects effectively did not exist to the core of Free Software community. Even though all of these projects are themselves Free Software, to run it you had to download and install proprietary JDK. I think for most of us, we might as well have been required to download a Windows or OS X VM. At least that's the way I felt. It wasn't very integrated with the operating system. But most importantly, it was a blob which we didn't ultimately control, and we were right to avoid the Java Trap.

But, those were pre-earthquake times. The integration of the formerly-separate Java/JVM world with the Free Software community is ramping up very quickly. For example, Fedora is close to landing all of the dependencies of JRuby. I don't think anyone has started on things like Processing or World Wind yet - it could be you!

A shared Free Software runtime

I want to talk specifically about efforts like JRuby, and the newly-invigorated Jython. In an earlier blog entry, we looked at the fragmentation in the Free Software community. Every free language has its own runtime and libraries; and until now, building on the JVM wasn't an option if you wanted many contributors from the Free Software world. OpenJDK is finally changing that. Now, you can write a library using Java, it can be sensibly integrated with Free operating systems like Fedora, and can be consumed by anything on the JVM, which includes Python and Ruby, as well as new languages like Groovy. We'd actually be sharing more than just the OS kernel and C library.

Crucially, this is a platform we now control, because it's Free Software. And the original upstream is not just giving some standards or source code drops, they are actively helping us. In fact, "they" are "us" now!

Part 2 of this entry

Making use of those blank new tabs

Fri, 14 Mar 2008 22:16:44 GMT

I've just uploaded a new version of the Firefox Journal to Firefox Addons. It's in the "sandbox" right now, which means you need to create an AMO account and enable the sandbox to download it from there.

In this new release, because the awesomebar does search so well, this version of the journal removes the search interface; it's now focused on helping you get back to sites you recently visited quickly, as well as providing an automatically-generated list of your 5 most-frequently visited web sites.

Here's an updated screenshot. For more links and background information on the journal, see the home page.

Drawing power from the sky, part 2

Thu, 13 Mar 2008 19:44:33 GMT

In a previous entry, we briefly looked at Amazon's Web Services from a high level. Now that I have my application running reasonably well and debugged, I wanted to write a bit more about my experience. I'm not claiming true expertise here, but I've done my best to learn what I can and hope I can pass on some of this in a comprehensible way.

So it has certainly been a learning experience trying to understand the overall picture for how one writes an application on top of these APIs. As background material in this domain, I found the Google FS, Bigtable, and Map/Reduce papers useful, as well as the Building Scalable Websites O'Reilly book by a Flickr developer. If you don't read anything else, read the Map/Reduce paper at least - every programmer should understand it. Coming from the perspective of an OS/desktop developer, I personally found the GFS and Bigtable papers the most interesting as compared to the POSIX file APIs and SQL, respectively.

"Everything you know is wrong." - Marble Madness

"scalable"

One of the most important things I've learned is that there are two kinds of "scalable". One is the kind of "scalable" that MySQL clustering, JBoss clustering, etc. offer. These systems take you from one machine to smaller values of N. They're typically based on UDP broadcasts or the like.

The other kind of scale is called "web scale" - this is where your application is a completely distributed system, running in multiple data centers. No one machine is truly critical. Your application gets just faster as you add machines.

What's the tradeoff between "scalable" and "web scale"? The answer is pretty simple. Your application has a number of facets such as reliability, consistency, and availability. Researchers have essentially come to the conclusion that you can't have all 3 at the same time as load increases, and of the 3 you almost certainly want to sacrifice (immediate) consistency. If I write a new review of a book on Amazon, someone hitting Reload on the same page a few seconds later might not see it. If I receive an email in GMail, it might fail to be in the search index. If I delete a picture from Flickr, it might still be in my "photostream" display. But one property the system can have is eventual consistency, on the order of minutes or even seconds perhaps, but not immediate. My review should eventually appear to the person pressing refresh, the email will appear in the search index, and my picture will truly be gone (from the UI, anyways).

On Relational Databases

If you look at almost any modern web development framework like JBoss Seam, TurboGears, Rails, etc etc., at the heart is a relational database for storage. Using a relational database lets you effectively push most of the hard problems like persistent storage and concurrent access onto an external system (though that could be in-process using SQLite). There are some very smart people who developed SQLite, MySQL, Postgres, etc. A relational database gives you a lot for free, and unless you know what you're doing, you should probably not attempt to store your data directly using say the POSIX file APIs (and this is true on the desktop side too, but that's another blog entry).

In other words, for a lot of applications, relational databases are exactly the right solution. You can get very far with a relational/small cluster system; Wikimedia is an example here, though their job is obviously made much easier by the fact that reads truly dominate access.

Also worth mentioning is the approach of partitioning a relational database; though at this point you're starting to move away from the normalized relational model, since you can no longer perform an operation over your entire data set.

So you've decided to write a distributed application

It should come as no surprise that there's no magic API to code to in order to create a web-scale system. What you need to do depends on your application. You need to have an understanding for how data flows into your system, what kinds of operations you need to perform on it, where and how you can sacrifice consistency, etc. There's a lot of more implementation/deployment level things like load balancing and tuning that matter a lot too.

However, there is a sort of general approach of asynchronous, cached data generation. In the relational model, every web page view creates a transaction which gets the data it needs, then returns. This way your users always see up-to-date information. But in the web scale model, you can massively denormalize by precomputing data. The advantage is a web page request goes no where near a database; you simply pull the cached data from storage close to the edge web server. When you get a mutation in your system, you trigger an asynchronous (e.g. map/reduce) process to regenerate all the cached data. This may not be instant, but with some intelligence in your system you should still be able to have eventual consistency.

Writing a bug tracker

Let's say you're writing a bug tracker. In a traditional web app, you'd have a form which would do a POST to send an updated comment. In the handler for that POST you open a transaction, and append a new row to your Comments table. But in a distributed system, you could generate cached data for each bug. A GET request for that bug is just a simple transformation of the cached data (perhaps XSLT, or maybe you save the final HTML). But the POST request to add a comment put is into a reliable work queue (like SQS). Another machine reads from this queue and triggers a regeneration of the cached bug data - which could be more than just the cached data for that particular bug. Let's say you want to be able to display a list of all open bugs. One way to handle this would be to store in S3 a key /open/ for each bug. Then a query of open bugs maps to a listing of all keys with that prefix. But probably what you really want for these kinds of metadata operations is a system like Hypertable or SimpleDB.

A new kind of operating system?

If you look at Amazon's APIs, they provide the root services you'd expect of an operating system: storage (key/value as opposed to hierarchical), IPC (reliable/persistent), and task creation (Xen image as opposed to process/thread). The challenge though is rewriting applications to use new APIs.

Storage

I mentioned in the previous entry that I thought storage was the most important of these 3. There are a lot of messaging systems out there (in particular AMQP looks pretty good), and virtualization APIs abound. But there are many fewer high-profile storage APIs. Storage is hard, and rewriting applications for a new form of storage is not at all trivial. That's why so much is invested in POSIX-compatible storage systems - if you have that, then you instantly have Apache, rsync, and probably on the order of hundreds of thousands of other useful applications. But there are limits to the POSIX API; the Google FS paper discusses those. Just as there are limits to the relational model. What is going to be interesting over the next few years is to see which of these new APIs start to win application developers, and how the traditional database-based free software development stacks adapt.

So you have some XML...

Wed, 12 Mar 2008 02:15:15 GMT

Recently I battled the Typica library for accessing Amazon Web Services. There was an obscure conflict between its JAXB dependency and our app server. JAXB is...well, at its root it compiles XML schemas using a mapping to Java classes. This is in theory cool - you can get completion in your IDE for web services. In practice though, it just feels to me like it's not worth the complexity.

So, I needed to quickly roll my own Java bindings for AWS. Basically we need to do a HTTP POST request, and then parse a small returned XML document. For the first part, Apache Commons HTTP is quite good. But then I was in a situation in which I'm sure many developers find themselves - you need to get the data out of a small XML document. No need for schema validation, etc.

There are a lot of XML libraries out there. We could use the DOM API, which is familiar because it's fairly standardized across languages like Python, Java, and JavaScript, despite how nonnative it feels in all of them. If you're really a masochist, you'll try to write a state machine with SAX. Then there are the bindings which target a specific library like JDOM for Java, ElementTree for Python, etc. These can be nice, but they're also not portable if you happen to move between different languages (and many developers today probably move between JavaScript and a sane language on the server side). So, there is another option which gives you a decent API that's also largely portable: XPath with DOM.

Here's our sample document:

<ReceiveMessageResponse>
  <ReceiveMessageResult>
    <Message>
      <MessageId>11YEJMCHE2DM483NGN40|3H4AA8J7EJKM0DQZR7E1|PT6DRTB278S4MNY77NJ0</MessageId>
      <ReceiptHandle>Z2hlcm1hbi5kZXNrdG9wLmFtYXpvbi5jb20=:AAABFoNJa/AAAAAAAAAANwAAAAAAAAAAAAAAAAAAAAQAAAEXAMPLE</ReceiptHandle>
      <MD5OfBody>acbd18db4cc2f85cedef654fccc4a4d8</MD5OfBody>
      <Body>foo</Body>
    </Message>
    <Message>
      <MessageId>0MKX1FF3JB8VWS8JAV79|3H4AA8J7EJKM0DQZR7E1|PT6DRTB278S4MNY77NJ0</MessageId>
      <ReceiptHandle>X5djmi3uoi2zZ8Vdr5TkmAQtDTwrcd9lx87=:AAABFoNJa/AAAAAAAAAANwAAAAAAAAAAAAAAAAAAAAQAAAEXAMPLE</ReceiptHandle>
      <MD5OfBody>37b51d194a7513e45b56f6524f2d51f2</MD5OfBody>
      <Body>bar</Body>
    </Message>
  </ReceiveMessageResult>
  <ResponseMetadata>
    <RequestId>b5bf2332-e983-4d3e-941a-f64c0d21f00f</RequestId>
  </ResponseMetadata>
</ReceiveMessageResponse>

And here's the code:

XmlParseData parsed = XmlUtils.parseXml(post.getResponseBodyAsStream(), 
                                        new String[] { "q", "http://queue.amazonaws.com/doc/2008-01-01/" });			
NodeList msgNodes = (NodeList) parsed.xpath.evaluate("/q:ReceiveMessageResponse/q:ReceiveMessageResult/q:Message", 
                                                     parsed.doc, XPathConstants.NODESET);
Message[] msgResult = new Message[msgNodes.getLength()];
for (int i = 0; i < msgNodes.getLength(); i++) {
  Node node = msgNodes.item(i);
  msgResult[i] = new Message(parsed.xpath.evaluate("q:Body", node),
	                     parsed.xpath.evaluate("q:ReceiptHandle", node));
}

I think this is pretty clear; the straight DOM or SAX alternatives would have been notably more lines of code. The key part being the XPath expression /q:ReceiveMessageResponse/q:ReceiveMessageResult/q:Message which gives us all the Message nodes.

For more about parsing XML in Java (though largely applicable to other systems that have XPath bindings), see this DeveloperWorks article may be useful. Also, this is the wrapper class I used in the code sample. And yes, I am using new String[] to work around Java's lack of hash literals.

Drawing (nearly) unlimited power from the sky

Thu, 21 Feb 2008 03:26:31 GMT

There was an interesting article in the New York Times recently about drawing fuel from the air. Pretty cool. However, there's another kind of power from the sky I want to talk about - Amazon's Web Services, which give you unlimited computing in the "cloud", with just a credit card.

Scaling the Mugshot polling

Recently I've been working on improving the way Mugshot does its polling for changes in various things like RSS feeds, Last.fm and Flickr APIs, etc. Previously, we parsed and stored RSS feeds as individual entries in our relational database. This worked fine for a while, but as we grew we started spending a lot of time reading the feeds out of the database, just to compare their content to the new version to decide when something changed. A much simpler way to do this is just to store a hash value (say SHA1), and compare it to the hash of the new version - no database involved. If the hash has changed, then we do a full parse and check for changes.

The other thing I wanted to do was easily allow for the feed polling load to be spread out across a number of machines. We don't yet have enough feeds where we really need multiple machines, but it seems like something that would be good to be ready to parallelize. I'm calling the new system Firehose; it's not really a very complex codebase, mostly I spent a lot of time dealing with AWS binding issues; updating the Python bindings for AWS needed to be updated for the new SQS API, and I had to toss together a lightweight Java binding for SQS too because of an obscure conflict between dependencies of typica and our version of JBoss. Anyways, that is the boring stuff.

Entering the cloud

In a very brief summary, the key Amazon services are:

Key/value pair persistent storage (S3)

Compute nodes (EC2)

A messaging system (SQS)

I'm not going to drown you in buzzwords here - but it is very important to understand just how different this system is than a LAMP stack or J2EE app server cluster. First, all of these services are web scale; the key architectural point here being that by adding nodes, the service just gets faster and more reliable. However, these APIs also require a very different way of writing your application.

For example, the architecture behind Firehose is that there is a master process which simply maintains in-memory a set of pending tasks. It then dynamically allocates a set of slave servers to process the active tasks, using the EC2 API. In other words, the application is automatically provisioning new servers. It scales itself. If there is a sudden influx of new feeds to process, no intervention by me is required to allocate new servers or storage. If we delete tasks (for example, if we start an agreement with a service provider such as Last.fm to have them efficiently push changes to us rather than requiring polling), then the service scales itself down.

EC2 is pretty neat - it gives a feeling of unlimited raw power at your fingertips. I know that at any time, on demand, I could spin up any number of servers I want, have them process a lot of data, and then shut them down when I'm finished, all for a price of cents per hour. If I stop running any servers, it doesn't cost me any money.

There are many things that fit into this dynamic batch-processing model. For example, this article by the creator of the Python bindings for EC2 shows how to batch-convert a large video collection. A use case topical for Fedora would be infinitely scalable software rebuilds; no more getting held behind kernel and openoffice rebuilds. As the large North American engineer group comes online in the morning and afternoon, the system could scale itself up, allocating more servers. At night when load is less, the system deallocates servers, saving money.

Coding for the cloud

What I find most interesting though is new APIs for writing server applications. These APIs are not all of the same complexity - EC2 is mostly just a fancy fork() that actually allocates a new OS image. Of the APIs, S3 is the most interesting. In a future blog post, I'll talk about storage, because how you store your data has a profound effect on application architecture. Developers before now have generally coded to either the POSIX filesystem API, or relational storage. But S3 is not POSIX and it's not relational. Would it make sense for the operating system (Linux) itself to provide a default API for storage that isn't POSIX or relational?

Internet Everywhere with CDMA and NetworkManager

Wed, 13 Feb 2008 00:23:46 GMT

I subscribed to Verizon Mobile Broadband recently, after having paid for Wi-Fi one too many times at coffee shops. Googling for my card model (Kyocera KPC680) before I ordered it seemed to turn up a few hits, but the instructions I'd found were all about setting up PPP manually in config files in /etc and the like. Seeing as this is 2008 and not 1998, I started checking out NetworkManager's support.

NetworkManager connected to Verizon EV-DO

It turned out my card was new enough that the kernel driver PCI lists didn't include it - but adding it to the driver PCI ID list, and writing a .fdi file so it was flagged as a CDMA modem got me going. Both of these changes should be showing up in Linux kernel and HAL releases hopefully. So far it's been very reliable, and all I have to do is click on the NetworkManager applet and pick "Auto CDMA Dialup connection". Pretty cool, nice work NM team!

Free Software and Linux should not be a choose-your-own-adventure game

Sat, 09 Feb 2008 18:22:02 GMT

Fairly recently, I ran out of disk space. If you're a developer, especially one who uses things like JHBuild, virtualization, etc. - this has almost certainly happened to you. The consequences are pretty unpleasant (search for disk space in the GNOME bugzilla).

This got me to wondering - why the heck didn't GNOME have a low disk space notification? This morning I decided to look into that. Certainly a patch would be not more than 100 lines to do a libnotify notification, along with an action to say empty your trash.

It turns out that I (and at least Fedora) had been victim of two things. 1) autoconf 2) GNOME release policy.

What happened of course is that the code exists - it was just conditional on libnotify-devel being installed. From the Fedora gnome-volume-manager build log:

checking for NOTIFY... no

So the the support wasn't being compiled in, for who knows how long. I just fixed this for Rawhide, but we need to stop thinking of Linux as a source of random bits that you can put together optionally, and instead think of it as one unified stack. That means if you're a software maintainer - your check should not be if (have_libnotify_h), but instead if (linux) { require(libnotify_h) }. GNOME release policy totally ignores the fact that the vast majority of people using GNOME have these components. I understand the desire for things to be optional - we want GNOME to work on Maemo and new devices where not all of the stack may exist. But the right approach is for these new distributions to use flags like --disable-notify. Or we can use a semi-standardized flag like --enable-maemo, since often these non-desktop devices require other source changes too.

The future of programming language runtimes

Sat, 26 Jan 2008 17:22:11 GMT

There is a saying in linguistics, "A language is a dialect with an army and a navy. I think this applies in a way to computer programming languages too. The most widespread programming languages aren't necessarily the ones that are "perfect" in an Esperanto sort of way, but the ones which have an army and a navy - i.e. large corporations backing them.

Fragmentation

The current fragmentation in the language runtime space is quite disheartening. It's still best described by this 7-year old hub paper by Havoc and Owen. I hope they won't be mind my linking to it. Examples of this fragmentation problem are still prevalent today. The difficulties in using gtkmozemebed from Python due to two garbage collectors and type systems in one process. To use the Linux inotify system from Ruby, you need a random third party binding that may or may not be up to date. If you want to use the Eclipse rich text editor in your next OCaml program, well you're just screwed. The examples are almost endless...most pairs of (API X, runtime Y). Not all of those are sensible combinations, but a lot of them are.

Now, the problem with Hub I think was that it assumed you could still sensibly mesh unmanaged languages like C/C++ as they are today with managed VM-based runtimes like Python and Java. If you've been following the Mozilla project, you know that they are embarking on a huge effort to automatically rewrite their 2+ million lines of code to go from C++ with reference counting to C++ with garbage collection, because the problems with leaks between the two systems are too hard. (By the way, this stuff is really cool and worth watching as I've mentioned before).

In the GNOME community, we largely get by with systems such as PyGTK and Gtk# that do mix runtimes with reference-counted C because we're not running untrusted code from the Internet, like Mozilla has to. If you forget to destroy that widget you removed but that still had signal handlers calling managed code connected to it, it's a bug in your application that is your fault - not the fault of some college student writing his first JavaScript application that was an intern at a website you visited.

The technical future

This prediction has been made plenty of times for years, but I think it's worth repeating. The future of language runtimes is pretty clearly going towards VM-based runtimes. C/C++ are dead ends for pretty much anything except a kernel. More modern AOT-compiled languages such as OCaml and Haskell have a lot to gain by switching to a VM. Concurrent, generational garbage collection isn't trivial, and a modern VM gives you that. Being able to load code dynamically, or programmatically construct new optimized functions at runtime for specific data is very appealing. Debugging is significantly easier.

The political future

So, here we are in 2008. Is the situation much better? In some ways, yes. The Free Software community now includes Java. Popularity of truly open, community-driven languages and runtimes such as Python and Ruby seems to be rising. And finally, there is a project which is attempting to create a .NET base language runtime, Mono.

Mono

The Mono project is the marriage of two major ideas. A runtime designed for multiple languages is a very unifying concept. It would solve almost all of the issues Hub was attempting to address for managed, VM-style languages. It leaves C/C++ out unless they're manually bound, but there's no helping that.

The second idea though of Mono is that it is defined as a .NET clone. This idea has proven very divisive. One company, Microsoft, is the upstream. This is not a good idea for the Free Software community; we don't have control of our technical destiny anymore. If we want to add a bytecode that makes Python run better, we have to wait for decisions to be made in some Microsoft boardroom.

If I was in charge of Mono, what I would do immediately is reach out to 3 communities that are much more clearly part of the Free Software world: Python, Ruby and finally - Mozilla Tamarin. Mono could say to these communities, "Hey, we'll be your JIT, garbage collector, and common library runtime" - but with the freedom to adapt and change to the needs of these communities. Especially having Mozilla on board would be a coup, because it's shipped to so many people. I certainly wouldn't have a problem making my application depend on Firefox being installed on Windows.

The problem with separate projects like Jython and IronPython, JRuby etc., is that they're essentially forks. They have connections with the upstream, but that's not the same as being the upstream. If Mono can unify the truly open Free Software languages around one common runtime, it would greatly circle the wagons around

Java

Sun has finally realized that the idea that every single application would be written in Java and only needs a least-common-denominator API across every platform doesn't work. Even more importantly, the JDK is now Free Software, and though the JCP isn't ideal, it at least exists.

It certainly seems plausible to me to imagine that Sun would be willing to buy into a shared runtime that wasn't .NET, but a true cross-corporation, community effort. They're already trying to add new bytecodes to better support other languages, but instead we could translate IKVM-style from Java 5.0 bytecodes into a new shared runtime.

5 years from now...

I really hope I'll be writing my Python app that

Gets efficiently JIT-compiled

Can access various OS components like inotify automatically as they're added to a common shared library

Can seamlessly blend in a Web-like widget that executes other languages such as JavaScript without pain

Is able to call into frameworks like Hibernate

Oh, and I want my jetpack.

Hotwire hypershell 0.700 alpha released

Fri, 25 Jan 2008 02:55:26 GMT

Hotwire 0.700 (mirror) is now available.

So, about 8 months ago you may remember I got really frustrated with bash for carelessly losing my history, and in my spare time, I set out to create a new developer shell. Not just a plain old Unix shell; I wanted a shell that took advantage of my computer (using a modern language runtime, and integrating with the desktop environment), rather than being limited by the "terminal emulator" concept. I'd recently tried Windows Powershell around that time as well, and thought they had some good ideas but also made some bad design decisions. I thought it was possible to do better.

Looking for a firefox process

In short, I wanted to create a "hypershell" that was better than the Unix concept of a shell, and that was also improved on the Windows PowerShell model. Hotwire 0.700 is the first release where I think that goal is, if not achieved in many ways, much closer at hand. Previous versions were more like demo toys, but this is the first one I'd call alpha. In other words, usable in many ways, although far from complete. Many developers and power users will hopefully find it a suitable replacement for a terminal+Unix shell running as the same uid on a desktop. For system administrators, Hotwire doesn't yet have a really strong story for improving the remoting experience (well, you will probably like the included ssh client), but I'm going to be taking a close look at things like func.

The new webpage (mirror) has a longer explanation of things, screenshots, etc. The ChangeLog is also a useful read, especially if you're upgrading from a previous version. And don't forget to try the updated tutorial to get a hands-on introduction to the system.

NYT on SimpleDB

Fri, 21 Dec 2007 23:18:15 GMT

The New York Times Open blog takes a look at Amazon's SimpleDB. I have to agree with him, web-scale computing architecture is a fascinating topic. I've been meaning to set aside time to look at SimpleDB; it feels like the big gap in web-scale is standard/open APIs for structured data storage. EC2 gives you compute (and the API is fairly obvious), and S3 is your big key-value bucket, but until Amazon announced SimpleDB there were really only hacks for clustering MySQL. And that only gets you so far.

Another interesting project in this area is CouchDB.