Ignorance Compounded

time-delayed feedback in the workplace

2010-10-22T11:37:00.000-07:00

The job of buildmaster rotates amongst managers. The buildmaster is primarily responsible for haranguing developers when the automated test failure rates are too high; and if they are too high for a while, the buildmaster can "lock the line", meaning that the only permitted checkins are those that ostensibly fix tests. We have some test suites that take several days to complete. Thus a bad checkin may cause test results to plunge days after the fact.

In Peter Senge's classic The Fifth Discipline, he talks about the effect of introducing a time delay into a negative feedback system. Whereas negative feedback usually stabilizes a system, negative feedback plus time delay tends to cause ever-more-violent oscillation.

Consider the following actual data:

Test	Current	EOD 10/21	EOD 10/20	EOD 10/19	TARGET
fast_suite	97.55%	98.77%	99.39%	99.39%	98%
slow_suite	86.43%	94.10%	83.61%	95.29%	97.5%

The fast suite returns feedback in a couple hours; the slow suite takes a few days to catch up to a changelist.

I am assured by various people that it sucks to be the buildmaster. It will continue to suck to be the buildmaster, I think, until we devise a system that is stable rather than oscillatory. A stable system is characterized by damping rather than nonlinear gain; and by feedback that is at least an order of magnitude faster than the forward phase response of the system. (It's possible to stabilize systems other ways, but this is the most general and reliable.)

To speed up the feedback loop, we could have fast suites that predict the behavior of the slow suites. Simply choosing a random subset of the tests in the slow suite, running those first, and providing interim results could achieve that.

To have damping rather than nonlinear gain, we need to remove or highly restrict the buildmaster's ability to lock the line; and instead, we need to increase the amount of pre-testing that is required in order to do a checkin. For instance, if interim results indicate a high failure rate, then new checkins should be subjected to a higher level of testing in the precheckin queue before they are allowed to actually commit.

The Compiler Is Not The Audience

2010-08-13T11:02:00.001-07:00

Is the code that you write making life easier or harder for the next person who has to work in the same area? Are you creating complexity and fragility that will slow them down, or platforms, patterns, and utilities that will speed them up?

test modularity

2010-08-12T16:09:00.000-07:00

Principle: keep your tests of functionality separate from your tests of business rules.

Example: I have some code that provisions licenses (bundles of permissions) to entities. I should have one suite of tests that verifies that it is possible to provision arbitrary combinations of permissions correctly - that is, the functionality. The clients of my code should have tests that verify that they are choosing to provision the particular combinations they expect - that is, the business rules. My tests should not fail when the business decision of what permissions to grant to which entities gets changed.

Code ownership

2010-08-10T15:04:00.000-07:00

As an aside: I've not written much lately because I'm working for a non-open-source company, which limits what I can talk about without crossing IP boundaries. I've decided to start posting again but will have to be a bit vague.

My company's roots are in web application development. This seems to contribute to a more horizontal rather than structured architectural style: all groups for themselves, each working on small user-facing features. There is a general principle that everyone is allowed to check code into anyone's area: code "ownership" is discouraged.

I find this a problem. At this point our codebase is quite large. No one, not even the principals of the company, understand it all; I routinely ask them questions and get back answers like "well, it's been a while since I worked on that so I'm not sure." But at the same time, no one can fully understand even the piece they work on, because it has been partied on by a myriad of developers, few of whom were well versed in its design, its test suite, its intentions, its history.

Dual dispatch

2010-01-12T18:53:00.000-08:00

Computer programming is about describing the behavior of entities in various scenarios. For instance, if you're writing a data entry form, you might need to describe how it behaves when you click a certain button, how it behaves when you type into a field, and so forth. In object-oriented programming, we try to organize the code that describes an object's behavior together with the data that describes its state.

This gets messy when objects interact. Suppose you've got three Animals (Dog, Cat, Monkey), and three kinds of Food (DogChow, CatChow, MonkeyChow). You want your program to ensure that Monkeys can only eat Monkey Chow, Cats can only eat Cat Chow, but Dogs can eat all three kinds. So, you add a method to each of the three Animal classes, something like "boolean canEat(Food chow)". The implementation for Cat looks like "return chow instanceof CatChow", Dog looks like "return true", and so forth.

What happens when you add an animal? No problem, you just have to implement its canEat() method. What about when you add a new type of food? Well, you have to go through all the existing animals and make sure their implementation is still right. For instance, since chocolate is poisonous to dogs, the Dog implementation of canEat() is wrong. No compiler error, but your dog might die.

Or, you could flip the problem around, and put methods on all the Foods, saying what animals can eat it. Now, when you add a food, it's just a matter of implementing its canBeEatenBy(Animal animal) method. But when you add a new animal, you have to check all the Food implementations.

This problem of how to describe the interactions between entities is called "dual dispatch." I know of no particularly good general solution.

Test Automation

2009-09-03T10:02:00.000-07:00

Writing good tests is hard: test code tends to be opaque, fragile, and hard to maintain even when written by the same developers who write decent product code. Why?

Test code is inherently introspective: it is about the code it is testing. Product code is about a user-domain problem - for example, managing the relationships between a company and its customers. Test code is about the user-domain problem and about the product code. As a concrete example: writing tests often requires exposing (through package sharing or through "test-only" APIs) methods that don't make sense otherwise, in order to give the test code the ability to inspect the internal state of objects in the product code.

Introspection, or self-reference, increases conceptual complexity. An object that does something is simpler than an object that reasons about an object that does something.

Dogfood

2009-07-13T11:28:00.000-07:00

The company where I spent my vacation has a very strong tradition of "eating one's own dogfood," which means using the products they are developing. This is supposed to improve the quality of the product, because one becomes painfully aware of the problems and is motivated to fix them.

My recollection is that when I first heard the phrase, back in the early 1990s, it was "tasting," not "eating," one's own dogfood. There is a significant difference between the two.

The problem with eating only your own dogfood is that you start getting used to the taste of dogfood, and you don't discover that the rest of the world has learned how to cook a decent meal.

Maybe it is better to eat whatever the best food around is, while occasionally being forced to taste one's own dogfood. For example, let development teams use whatever products they want, but have test days where the teams work through predefined customer use cases using their own products.

Vacation

2009-06-30T11:52:00.000-07:00

I spent the last month on "vacation", working for a different company on a different product in a different programming language. It was actually quite a lot of fun; I'd like to take more such vacations (not least because, unlike the sort where one travels to a different country, I got paid to do it.)

I was working primarily in C#, a language I've not spent much time in till now. I was also working on Windows-specific user interface code, rather than the platform-independent systems-level code I normally write.

C# is a weird language. It feels to me like something of a kitchen-sink language: it's got a bit of practically every language idea I've ever heard of. Generics, function delegates, event handlers, closures, ... there are about five times as many keywords as Java has. Java is to C# as Spanish is to English.

There were a few things that I really liked: I did get pretty friendly with closures, for instance, and the language support for iterators is cool. Java still feels more elegant to me, though. There's just too many ways to skin the same cat in C#. And Java's collection and concurrency libraries are still way better.

The one thing I most disliked about C# isn't really a fault of the language as such: the naming conventions. In Java, by convention, we name types with capital letters; methods, variables, and packages with lower-case. This helps disambiguate some syntax that would otherwise be hard for humans to read (even though it doesn't bother the compiler at all). In C#, by convention, namespaces, types, and methods are all capitalized; fields and variables aren't. Properties, which are fields that have built-in getter and setter methods (so they can be accessed by assignment syntax), are capitalized. To my eye, it's hard to read and ugly.

To a compiler, none of these distinctions really make much difference; the differences between the .NET and Java virtual machines is pretty small. Programming languages, as I've said before, are a way for programmers to express ideas: they're for people, not computers. Esthetics matters.

If I were in St. Louis, I'd be attending this talk on the Fan programming language, which claims to be compilable to both virtual machines and to get rid of a lot of the uglifications of both Java and C#.

I Don't Quite Get Dependency Injection

2009-05-06T21:18:00.000-07:00

Here's how you write a program that prints out "hello world" in ordinary Java:


public class Hello {
  public static void main(String[] args) {
    System.out.println("Hello world!");
  }
}

I'm reading Spring In Action, and this excellently written book starts out by writing "hello world" in Spring-enabled Java. It's about ten times as long; I won't reproduce it here. The main benefit is that the "Hello world" string is in an XML file, instead of in a Java file, so that it's easier to change.

Or something. Personally, I'd rather edit Java than XML. In my experience as a developer, it's actually much easier to write cryptic bugs in XML (or more broadly, in configuration code of any sort) than in Java. This is because the rules of Java (or other formal languages, such as C, C++, heck, even Perl) are more standardized, better defined, better testable, and better documented than those of proprietary configuration languages such as Spring. If I change a character string in Java, I can predict quite well what will happen, and I can watch it happen in the debugger. If I change a string in an XML configuration file, I have no way to know who is reading that file or what they are doing with it. It's magic.

In the 1970s, when I first learned to program, the field was predominantly procedural: a program was a bunch of instructions to be followed one after another. The instructions might involve reading data from files and acting on those data, but the data did not modify the program itself; in fact we believed at the time that it was very poor form to let a program be self-modifying, because it made it hard to understand how it would behave, so we were very reluctant to consider the data files to be part of the program.

Thirty years later, it feels to me like the software industry has moved very strongly toward configuration-based programming. We want our procedural (Java) code to do less and less, and instead we want to control behavior by way of increasingly complex configuration. A program I'm currently working on has got more configuration files than it does Java files, and they are spread over more directories on the hard drive. The procedural part is all in one language: Java. The configuration part is in EHCache, Hibernate, Spring, Maven, JDBC, and Log4J, each with its own cryptic syntax that is subject to change with every version and that is documented in bits and pieces on web forums and spottily-available books.

As so often, I find myself wondering if the emperor has any clothes. Is it really easier to program this way? Or is it just different?

I wonder whether instead, we should focus on figuring out what is "hard" about procedural programming, and work on making that easier, within the confines of a formally defined, easy to understand and read language. For instance, IDEs could easily help with changing dependencies - in fact, an IDE could look at all the pieces of a program, determine all the external dependencies, and present them as a single view, allowing the programmer to substitute equivalent components.

Improving Maven

2009-04-06T22:47:00.000-07:00

I'm still at loggerheads with Maven. There are some specific problems that bother me and that I think could be improved while preserving its basic Maven-ness. Many of the problems center around issues when simultaneously developing more than one component at a time, i.e., when depending on SNAPSHOT versions. In fact a SNAPSHOT dependency is a fairly good indicator of misery; but some small improvements in Maven could reduce that pain point by a lot.

1. Maven knows that source produces artifacts, but it doesn't know that artifacts come from source. When Maven checks dependencies, it looks in local and remote repositories, but it doesn't know enough to build (or re-build) an artifact from source.

If Maven artifacts (at least locally deployed ones) had a backpointer to the location of the source project that produced them, then it would be possible to check dependencies against source. For instance, if project B depends on A, and I touch project A's code and then rebuild B, project A should also get rebuilt and installed. Similarly, if artifact A came from local source, then it almost certainly should NOT get replaced by an "updated" artifact from a remote repository, even if the remote artifact is newer; rather, local source code should always get honored. Extra points for a "mvn svn:update" command or the like, that would transitively sync the version control system to the latest code in all upstream projects.

2. SNAPSHOTs need to be versioned. When you're collaboratively working on two projects, and an API between them changes, the downstream build is broken until the upstream project gets refreshed. But right now that happens in a nondeterministic, asynchronous way: to Maven, all SNAPSHOTs are identical until, around midnight or so, it decides to refresh. Basing refreshes on an update interval is like filling up your car's gas tank every Friday: it's either too soon or too late. This needs to be deterministic. What I really want from SNAPSHOT is the idea of a fine-grained version number, that I will throw away upon release. It could be as simple as letting me say 1.3.0-SNAPSHOT-002, instead of just 1.3.0-SNAPSHOT.

3. Maven assumes that the internet is fast and reliable. It is neither, as anyone who works from coffeeshops and airports knows all too well. When Maven fails to get a network connection, or the network dies midway or times out, it needs to be able to roll back to a known and working state. Among other things, this means that updates need to be atomic across projects, or at least they need to be nondestructive. It also means that basic help should not rely on a network connection. Maven should not attempt to update plug-ins or projects during a 'clean' or 'help' operation.

I've got other problems with Maven - for instance, I think XML is nasty to work with, and I think that "convention over configuration" translates to "doesn't play well with others." Those things are harder to address while still keeping it Maven. But if the above three improvements were made, I think no one who loved Maven would be harmed, and a lot of other folks would be helped.

Eclipse awards

2009-04-02T14:48:00.000-07:00

I spend most of my professional life feeling ignorant of one thing or another, so in the few areas where I do know at least a little bit I try to help out. For that reason I post pretty often to the eclipse.newcomer newsgroup. I'm proud to have been a finalist for the Eclipse Newcomer Evangelist award for the second year in a row:

Sharing Data

2009-03-11T12:58:00.000-07:00

I recently discovered Joe Armstrong's post "Why OO Sucks". I didn't agree with much of what he said, but I was struck by one of his claims, "functions and data structures belong in totally different worlds." This is of course the antithesis of the OO (object-oriented) programming philosophy, which holds that you should lump data together with the sets of rules and actions that apply to it.

I disagree. Data is only useful if it has integrity, and it only has integrity if there are rules that govern how it is read and changed. "Rules" is just another word for "code", so this argues that code should be tightly associated with data.

An "object" in software jargon is just a way to expose data while still wrapping it in a decent amount of clothing. Within an application, objects make it safer to work with data, because you can ensure that no matter what you do the data is still valid.

The problem with objects is that they're not easily shared between applications. They're very ephemeral; they live only as long as they're contained in a running program, and they're tightly coupled to all the details of that particular program. In Java every object is an instance of a particular class, and every class is associated with the classloader that produced it, and classloaders in turn are associated with a single instance of a single application. If you try to put an object into a different application, it looks around for its classloader and, not finding one it recognizes, gets scared and shy.

The usual solution to this is to convert the object into raw data (often some sort of text representation, like XML) for long enough to transport it to a different application, and then in that application a new object is created by reading in the raw data and associating it with a hopefully compatible class from a hopefully compatible classloader. This is slow, expensive, and inaccurate. For instance, there's no way to guarantee that the classes are truly identical, so if an object moves from application A to B and back again, it might come back in an illegal state. Also, it often requires the programmer to write a lot of code to spell out how to read and write the object.

Terracotta is a way to spread the work of a software application across a large number of computers. We allow objects to move freely from one computer to the next. Under the covers we do still convert the objects to raw data and back, but we do it in a way that is quite efficient and transparent to the application programmer. We're great at moving objects from computer to computer in a single application. Up until the latest release, however, we weren't much good at moving objects between different applications. Now we are.

The basic idea here is that even though the classloaders for two different applications are different, as long as they both contain a definition of the class being shared (and the other classes that it in turn needs to access), that's good enough. The computer has no way of knowing whether that's true; but the programmer does. So, we let the programmer tell Terracotta which applications are allowed to share classes with each other. The configuration feature is called "app-groups", and I'd point to the documentation in this post, but it's not up on the web site quite yet. It's quite simple to use; you just define an app-groups element in the Terracotta configuration file, give it a name, and inside it you list all the applications that you want to be able to share objects with each other.

A typical use case would be if you've got a user-facing application and also an administrative application. Imagine, for instance, a merchant site, that lets users build up a shopping cart. Using Terracotta you might avoid storing that shopping cart in your central database, to reduce database load; instead, you'd keep it as transient data, getting session scalability and server failover from the Terracotta system instead. But suppose you want to let a sales agent view a customer's shopping cart, to make recommendations or fix problems. How can you share the transient shopping cart data between the customer-facing application and the agent-facing administrative application? One idea is to keep the list of shopping carts as a shared root in both applications, and then place both applications in the same app-group with the Terracotta configuration. No database required; transient data is still transient; no custom serialization code or data format definitions required. Just transparent sharing of objects between two otherwise different applications that both happen to include the same Java class definitions.

There are still some caveats, of course. One ugly one is that the different applications have to be running in different Java virtual machines. That is, you can't have a single application server instance and deploy both applications to it. That's for internal technical reasons that we hope to eliminate in a future release. For now, you'd have to put the sales-agent application on a separate app server instance (although it could be running on the same physical computer). Another caveat is that you can't have multiple overlapping groups (like, A can share with B and B with C but not A with C), and you can't restrict sharing to only certain objects or roots, it's application-wide. Caveats notwithstanding, I think it's a powerful new feature, and it'll be interesting to see what new uses of Terracotta this enables.

Circus Contraption

2009-03-09T00:58:00.001-07:00

As long as we're talking about things I'm proud of, let me mention the new show that I'm doing sound for, Circus Contraption. I did the sound system design and installation, and the overall sound design, and I'm sharing the night-to-night mixing duties with two other sound guys. If you happen to be in Seattle on a Friday, Saturday, or Sunday in the next couple months, try to see the show! It's the real thing. The sword swallower actually swallows the sword, it's not just stage magic.

Doing live sound is very, very different than writing software. You do not get a chance to fix bugs, and you cannot take things slowly or stop to have a design discussion. Every night is different: the performers sing and play softer or louder, there are more or fewer people (read: sound-absorbing sacks of water) in the audience, equipment that worked the last night breaks this night, someone trips over a wire or forgets to turn on their microphone. You do what you can and move on. Frankly, I'd probably be a better software developer if I treated software more like live sound.

Terracotta

2009-03-04T17:16:00.000-08:00

I write software for Terracotta, which is an open-source company. I love working on open source code, in part because what I do is not a secret - I can tell my geeky friends about the cool problems that I wrestle with. (I also work on Eclipse, another open source project.)

Paradoxically, though, it seems like in the open source world it is often very hard to talk about who my customers are. Partly that's because we don't always know - anyone can download the product for free. But also it's because our paying customers don't always want their competition to know how they succeed, and we of course need to honor their confidentiality.

So I'm really pleased that Terracotta has lately been getting some great press about one of our important customers, Sabre Holdings. They're perhaps most commonly known for one facet of their business, Travelocity. Sabre is huge - according to one article, "On any given day, Sabre's servers have to be able to handle up to half a billion transactions a day and a peak volume that can go up to 32,000 transactions per second."

How do they get that kind of volume, and the reliability that has to go with it? Answer: they run their mission-critical, high-volume stuff on Terracotta, my software. Yes, I'm proud :-)

What should code comments do?

2009-02-12T13:50:00.000-08:00

Below I've posted some code I just had to look at. I've got nothing against this code; it's a nice clean class, simple, I'm not aware of any bugs in it.

It's easy to figure out what this code does, just by looking at it. It takes a slash-delimited string ending in "war", like the one in main(), and deletes the third token if it contains only decimal digits.

But WHY? What problem does this class solve? What is Geronimo, why is the string "war" important?

I can't help but think that someone discovered the need for this code the hard way, after time spent looking at Geronimo code or documentation, talking with peers, perhaps after fixing a bug report from the field. All that information has now been lost.

Perhaps the need for this applied only to a particular version of Geronimo. Perhaps it only turns up in a peculiar use case. Perhaps the original developer's understanding was flawed and this code is never actually needed. There's no way to know, and anyone who encounters this code in the future will have to try to figure out how not to break it. Very likely, it actually does do something important but it's not covered in the test suite, and any breakage will be discovered as a regression in the field, when some user tries to update to the latest product version and their application no longer runs.

It's like a post in the middle of the living room: you figure it's probably supporting some weight above, but how do you know? So you can't remodel the room, because the second floor might collapse. But maybe the builder put it there because they were planning on a hot tub on the floor above, where now you've got a walkin closet. Now you've got to hire a structural engineer to do the same calculations again, because the original rationale has been lost.

Well-written code shouldn't need to explain what it does. But it should explain why it does it. What other options were considered? In what situations is the code necessary?


public class GeronimoLoaderNaming {

  public static String adjustName(String name) {
    if (name != null && name.endsWith("war")) {
      String[] parts = name.split("/", -1);
      if (parts.length != 4) { throw new RuntimeException("unknown format: " + name + ", # parts = " + parts.length); }

      if ("war".equals(parts[3]) && parts[2].matches("^\\d+$")) {
        name = name.replaceAll(parts[2], "");
      }
    }

    return name;
  }

  public static void main(String args[]) {
    String name = "Geronimo.default/simplesession/1164587457359/war";
    System.err.println(adjustName(name));
  }

}

Maven Maven Maven

2009-01-30T11:48:00.000-08:00

My post Maven Continues to Suck drew a number of comments, including a helpful and thoughtful comment from Jason van Zyl, who concluded "I don't think it's so much that Maven continues to suck as much as we need to do more than the free book we already have to get people past these simple setup problems that cause frustration."

I responded briefly in the comments but wanted to expand a bit.

Perhaps I will eventually see the light - I do keep hoping, because there doesn't seem much alternative - but I have not yet. Respectfully, Maven folks, I think it's optimistic to hope that if you "get people past these simple setup problems that cause frustration" you'll be in the clear. I've been working with Maven now for half a year; it continues to be perhaps my primary source of frustration, and to interfere in almost everything I do. I cannot say the same for any of the other tools I use. Neither my IDE (Eclipse), my version control system (SVN), my language (Java) nor its libraries were this problematic this far along.

The difference may in part be in my expectations; but the issues go beyond simple setup problems. To pick two other examples: (1) The Eclipse/Maven integration is rough, causing what should be simple Eclipse operations (saving files, jumping to referenced types, debugging) to be slower and less accurate. (2) No one on our team has yet figured a way to pipe arbitrary system properties from a Maven command line into a forked app server process in the context of a system test. These may be blamed on third-party software (M2Eclipse, Surefire); or on lack of Maven configuration skillz; but the Maven ecosystem is part of working with Maven, as is the fact that most developers on a team should not be expected to be Maven experts.

It is worth mentioning that invoking "mvn help" at the command line first tries to download stuff from teh internets, and then spits out a cryptic build failure error; mvn --help spits out a command line usage message that says nothing about how to actually get help. Even "mvn clean" tries to first do an update (which is always the wrong thing to do before doing a clean, because you may lose information about what to clean). Have you ever tried to use Maven without an internet connection handy, like on an airplane? Epic fail. I know about mvn -o but have never been able to get it to work, perhaps because the web of dependencies is so fragile and unstable.

By contrast, in our core code base we have a homebrew build script that solves all these problems, is easy for anyone with basic programming knowledge to maintain and modify, and just never seems to get in the way. If Maven requires more domain-specific knowledge, skill, and time to maintain than a hard-coded build script (or Ant script) would, is it buying us anything?

No tool is the right answer for all problems, and most tools are the right answer for some. Almost any tool can be extended, with sufficient skill and time, to do anything. That doesn't make it the right tool.

Moreover, the more smarts that we build into our Maven configuration, the more that we will rely on needing to hire developers with serious Maven chops; I would rather hire developers with serious programming chops. Maven skills do not generalize to other problems; Java, Groovy, Ruby, Perl do.

What I'm saying here is that I think there is a problem in principle with basing a build on a tool that requires deep domain-specific knowledge to use well; that I think Maven is such a tool; that, further, I think even with solid knowledge Maven is based on premises (such as the idea of a SNAPSHOT) that don't model the world well (pre-release output of a CI process is not equivalent to locally-built output of a local change); and that, finally, there is only room in the world for at most one convention-based tool, and we already have more than one.

Put differently, I'm saying that I think even if I knew how to use it well, Maven would not be the right tool.

Javasaurus

2009-01-30T09:49:00.000-08:00

Nothing about dinosaurs; apologies to any 6-year-olds I've misled.

There's an interface in the Java libraries called "Runnable", that just packages up the idea of "some code that you might want to run." This is handy when writing algorithms like "do some preparation; then run whatever it is the client wants to run; then do some clean-up." It's a way to hand a series of program steps from one module to another without having to know in advance what those steps are. ("Closures," much debated in Java, are another way of doing this.)

Runnable defines one method, "run". But the "run" method doesn't allow for the possibility of failure. I needed something similar, that was allowed to communicate failure (by throwing an exception). I knew there was something, but what? A search of the likely spots didn't turn up what I was looking for.

It would be really cool if there was a Thesaurus of Java, a tool or a web site that would let me type in "Runnable" and would come back with all the other things that were kind of like "Runnable." In this case the answer, provided by my colleague Hung, is "Callable." Doh.

A similar task that comes up for me a lot is that I've got a This, and I know it's possible to convert it to a That, but I'm not sure how. For instance, let's say I've got an Eclipse IFile, and I want to convert it to an ordinary Java File. The mapping isn't perfect, but basically, given an IFile there is a chain of methods you can call that will either get you the corresponding File or tell you it doesn't exist. But what is that chain of methods?

There's a finite number of methods that take an IFile as an argument (or receiver). There's a finite number of methods that produce a File. So there's a finite, although very large, possible graph between them - for instance, you could imagine calling something like IFile.getURL() and then URL.getFileDescriptor() and then FileDescriptor.getFile(). (I just made those names up, that's not the real answer.)

Most of the paths through the graph will be wrong, and some will be long and some will be short. But you could use the same sort of semantic analysis tools that are used for natural language translation, feeding off existing code (such as the Eclipse code base, in this example), to inform the graph of common versus uncommon pairings. I'd enter my start and end types, and I'd see a view of the top ten paths through the graph of possible method calls connecting them, perhaps even with links to existing code examples where something like that path had been used before.

I tried Googling for "Java Thesaurus" to see if this existed already, but all that comes up is Java code for writing thesauri.

Build languages

2009-01-08T11:34:00.000-08:00

I mentioned a while ago that I thought the world still needed a good language for build tools. One sign that no such language exists is that there is still a 1:1 correspondence between build tools and their languages. There is no such thing as a portable build description, although some tools have a limited ability to import build scripts meant for other tools.

What makes a build language different than other languages? That's worth a long post. But for now just a couple short thoughts:

Builds are special in that they are almost always slow. Building software is, ironically, one of the hardest things that a computer can do. It's common for a build of even a mid-size project to take several hours, and running the acceptance tests can take most of a day.

Also, builds typically have significant side effects. They modify the world. Running a build may cause a public web site to be updated; it may cause gigabytes of new files to be copied to slave machines around the world; it may send emails to thousands of people; it may cause many other dependent projects to become inoperable.

This means that you can't just tweak and re-run if there's a problem. Debugging an intermittent problem can take weeks, instead of hours, and it's typically a very public process.

Most computer languages have no way to express the idea that some steps are slow, or that some steps have side effects, or even that some steps are dangerous. A good build language would do that intrinsically.

Error handling is a difficult part of any program, but it is critical for build programs, because error conditions are very common and can have damaging long-term side effects. So a good build language would make error handling easier. For instance, it should be easy to associate an action with a set of restrictions, like "execute the update step, unless it would result in deleting any existing files." This sort of thing is not impossible in existing languages but it is requires more code than anyone would actually write (or get right).

Whereas most computer programs are designed to be run many times without change - for instance, hundreds of thousands of people will post millions of blog entries before the Blogger.com software has an update - build programs have to change much more often per execution. A build script might be run a few times a day, and updated every few days. So the writing and debugging of the build program, as an activity, are nearly as important as the execution.

So, a good build language should be closely integrated with writing and debugging. Modern computer languages are typically compiled rather than interpreted, meaning that you have to finish writing the program (or at least a complete, self-consistent, self-contained subset of it) before you can begin executing it. The opposite extreme of a compiled language is a command shell, which is an interactive environment that lets a user perform arbitrary commands one at a time; a command shell may support scripting but it does not build up a "program" out of the executed commands.

In between these extremes are interpreted languages, in which there is a program but it runs within an "interpreter" that lets the user run one step of a program before the next step has even been written. In an interpreted language, it's possible to write and execute a program simultaneously: when you finish, you've got a program and you've also got its results. A good build language should be interpreted, not compiled. And the interpreter needs to be able to tell the user what the next step would do, before doing it: sort of like print preview. This is an attribute of the build tool, not the build language, but it places restrictions on the build language.

I Want Pictures

2009-01-01T16:59:00.000-08:00

With all our interesting weather of late, I've been following Cliff Mass' weather blog. I've been fascinated and impressed by the great data visualizations that meteorologists have to work with. They have a head start because their data naturally maps onto a two-dimensional plot, but they've managed to add many more dimensions in ways that even a non-meteorologist can quickly comprehend. For instance, look at this image, which is the first image from his blog post of 1/1/09:

In addition to the two dimensions of space, and the overlay of geopolitical boundaries, this image shows sea-level pressure, temperature, and the vector of wind speed and direction. And it's just plain pretty to look at, too. I'd love to have a job that involved looking at pictures like that all day.

What would the software equivalent be, I wonder? Could I combine profiler data with a dynamic class diagram from UML? What if I overlaid a metric of function complexity on top of that?

The visualization tools for software are pretty weak, when you consider that all the information is already in the computer (we don't need weather satellites to get our data). It might be because software is an abstraction that doesn't easily lend itself to a 2-D layout like weather data does, but I think it might also be that software engineers are by nature less visually oriented. I think I'm more of a visual thinker than most, but not all, of the developers I've worked with. I'm not really comfortable with something until I can draw a picture of it.

Snow

2008-12-18T12:26:00.001-08:00

It rarely snows in Seattle, but it is snowing today; we've gotten about 4" in the last few hours. I live at the top of a steep hill so this basically means I can't leave home for the next few days. Fortunately or unfortunately, I work from home. Thus, the view from the Seattle offices of Terracotta:

Through A Glass, Darkly

2008-12-07T22:48:00.000-08:00

Although I get paid to write software, most of my time is spent understanding other people's software. I find that difficult: the available information is usually fragmentary, inconsistent, and more than I can hold in my head at one time anyway. It's like trying to read a mural on the side of a building, through the holes in a construction fence, as I drive by. I get little snapshots of partial information and I have to try to piece them together in my head. Sometimes the big picture I develop is entirely wrong, or missing big chunks.

Example: I've been working with Hibernate for more than a month now, but I still don't really understand exactly how it actually works. I only discovered tonight that it creates proxies to camouflage and control all my objects. This is sort of like wondering why your friends are acting a bit odd and then discovering that everyone on the planet except you has been replaced by space aliens.

Java has so dang many layers of obscurity that it is really hard to figure out precisely what code is actually being executed. The application code you write is just the tip of the iceberg. What with Spring, Hibernate, Terracotta, JUnit, the Hotspot compiler, and all the other frameworks and code enhancement tools we use, the program code itself is almost just a hint to the system. Maybe we're getting closer to the holy grail of being able to tell the computer what we want done, rather than how to do it.

Maven Continues to Suck

2008-12-05T15:33:00.000-08:00

Maven is supposed to be a tool to support collaborative software development.

Maven is based on the concept of software modules, each of which is versioned, and which have interdependencies. Let's say version 1.0.2 of MyThing depends on version 2.4.1 of YourThing, for instance. Now, what if I want to make some changes to both YourThing and MyThing? Well, there's a special version called "SNAPSHOT" - as in 2.4.1-SNAPSHOT - that means "the latest and greatest up-to-the-minute version of YourThing version 2.4.1". Setting aside the discussion of how the concept of a "version of a version" is inherently flawed, this introduces a big problem: it does not distinguish between changes I make and changes somebody else makes.

Case in point. Another developer at my company, and I, both are working on modifications to a module that our other stuff depends on. My modifications won't ever be shared with the rest of the world, they're just for my temporary purposes, but nonetheless I need them. The other developer, however, made some changes and committed them to version control. When I went to do a build of MyThing, Maven checked dependencies and noticed that there was a new SNAPSHOT of YourThing.

In a sane world, a collaborative software tool would notice the conflict, and perhaps notify me: "a newer version of YourThing is available, but it conflicts with your local changes - do you want to upgrade and lose your changes, or keep the stale but familiar stuff you've got?" The default, of course, would be to keep what I've got; after all, if I made some local changes, it was presumably for a reason.

Not Maven, though. Because I said SNAPSHOT (so that I could make changes locally), Maven silently and transparently discards my local version and updates me to somebody else's changes, at a random time of its deciding (typically, in the first build I do after midnight, of any project that happens to depend on YourThing).

Fortunately, the other developer's changes contained a serious bug. I say fortunately, because otherwise I might not have noticed that my changes had been silently discarded, and I might have spent a lot of time trying to figure out why my code, that used to work, no longer did.

What kind of collaborative software development tool is it that can't gracefully handle the simple case of two people working on a shared module?

Maven is not really a collaborative software development tool, it seems. Maven is a tool for letting one person develop a single module of code, with external dependencies on an otherwise-static world. That does not describe any software project I have ever worked on.

I keep hoping to see the light, and discover that I'm wrong about Maven. But it keeps on sucking.

Code Reviews

2008-12-03T17:09:00.000-08:00

A spare set of eyes is always useful, so most software teams have some practice of code reviews. There is huge variation from team to team in how code reviews are actually done, and in my experience how any given team does it is an axiom of the corporate culture; suggesting a different approach is met with the same sort of response as if you'd suggested, say, using an abacus instead of a personal computer. (I haven't tried this at my current employer, Terracotta, yet.)

The range of practices I've been directly involved with includes:

No review at all. I haven't seen this in a long time.

Self-initiated. If you want a review, you find a peer and beg some of their time. Surprisingly, this has been my experience working on Eclipse, and at a number of other teams; it may reflect resource constraints or perhaps just the fact that I've been in the industry for a long time and people foolishly think I'm less likely to write bad code.

Recommended review for major work, especially for new developers. This too is pretty common in my experience, and is the approach taken at Terracotta.

Mandatory review of all code before it's committed to version control. This approach was taken by a team I worked on in the '90s at Microsoft, and resulted in some of the best code I've ever written or seen. (The project made it through beta but was then axed by management; it conflicted with corporate strategy.)

Pair programming, two sets of eyeballs at all times. I've only tried this a little bit; it did seem like we wrote pretty good code, but boy was it exhausting.

Along with that range is the question of how, exactly, the code gets reviewed. Approaches I've tried include:

Over my shoulder. The reviewer looks over my shoulder as I walk through the code on my monitor, explaining it.

Over the reviewer's shoulder. Same, except that the reviewer is the one driving.

Independent review, meet with feedback. Each reviewer is given the code to review, and given some time (a day, perhaps) to review and come up with comments; we then meet to collect together the feedback. The meeting is typically moderated by someone other than the developer, and there may also be a designated scribe to collect the comments.

Collective review. Like "over my shoulder", except that there is more than one reviewer. Having multiple reviewers makes it psychologically harder for any one reviewer to change the flow of the review: if someone wants to go back and look at a previous file, they have to interrupt someone else.

And then, there's the question of media and presentation. If code is being given to reviewers, it may be presented as a set of changes to the existing code, or as a finished document; it may be presented as hard copy or as files. The review may be conducted in person, over a video link (like WebEx or Elluminate), by audio conference, or by email.

The costs and benefits vary a lot depending on approach. I've gotten the best reviews out of the "independent review" approach: it creates a social dynamic where there is some shame for any reviewer who fails to find a particular bug that others do find, so there's incentive for thoroughness, but it also lets each reviewer bring their skills to bear in whatever way and at whatever pace works best for them. The collective feedback (whether in a meeting or in group email) is a good way to learn advanced techniques, subtle errors, and so forth.

But this approach is also expensive - I used to budget half a day to participate in a review like that, between the reviewing and the feedback meeting, and if three or four people have to spend half a day for each developer who wants to check something in, it's easy to spend all your time reviewing. Also, this strategy works poorly for small changes or even large refactorings of existing code, because reviewers tend to get distracted by things that really should be fixed, but that have been that way forever and is not in the scope of the current changes.

The opposite end of the curve is probably the "go borrow someone's eyeballs if you need them" sort of review. This misses a lot of errors, but sometimes you get so deep into a problem that you lose track of the obvious. A second set of eyes can see that you failed to close an open stream, or that you're doing something that a library routine already exists for. A second set of eyes will probably miss subtleties like locking on the wrong object.

Personally, I'm afraid that of all the possibilities, I get the very least out of collective review over video conference. The audio and video quality is generally very poor, so it's hard for me to follow what anyone else is saying; if my attention wanders for a moment, it's socially awkward to ask people to back up, so I tend to lose track of crucial details; and since it's the code owner who drives the review, any assumptions they have about what is and is not important become shared. Often bugs lurk precisely within those assumptions.

Finally there's the question of what sort of comments matter. In the most formal review process I was part of, we prioritized comments exactly the same way we'd have prioritized bugs:

pri 1 was a bug that would make the code fail significantly or a design flaw that would prevent the code from working with the rest of the product

pri 2 was a more minor bug or design infelicity, or possible performance improvements

pri 3 was a bug that wouldn't actually affect the code as it stood but that might cause problems during maintenance or if someone else tried to use it in a different way

pri 4 was typographical stuff, like use of whitespace, naming of internal variables, and the like.

Using a structure like that, we went through each module of code, collecting and discussing comments in priority order. Pri 1 bugs had to be fixed before the code could be checked in, and generally another review meeting would be scheduled to review the fix, depending on the developer's track record. Pri 2 bugs were usually fixed depending on the amount of effort required. Pri 3 bugs might be deferred or ignored. Generally priority-4 bugs would be relegated to email, that is, the scribe would just collect them from the individual reviewers and tack them onto the followup email without them ever being discussed in person, to save time and avoid religious wars; the developer would be expected to make a personal decision about whether to implement the sugggested change. Doing it this way also let us directly compare the relative effectiveness of reviewing versus testing.

I'm against any review process that takes a lot of people's time to do a shallow review. Either review thinly and cheaply, or deeply and expensively, but don't fool yourself into thinking that a lightweight review process is finding all the interesting bugs.

Imbalanced outputs

2008-11-30T23:27:00.000-08:00

Although I earn most of my living writing software, I also work on audio whenever I get the chance. A few days ago one of my clients, a bass player, came to me with a piece of equipment that was causing a buzz.

The equipment in question was powered by an AC wall adapter, and it had a balanced output for sending the signal to the PA system. Balanced signal transmission is used in professional audio in order to reduce induced noise problems: the idea is that the signal is sent along two wires simultaneously, but with opposite polarity. At the receiving end, one voltage is subtracted from the other, eliminating any common noise that might have crept into the cables and leaving only the intended signal. The problem my client had was that whenever he used this output, it caused a terrible buzz in the PA system - rather counterproductive.

It turns out that a lot of what the industry calls "balanced outputs" really aren't. There's a popular belief amongst equipment designers that to make a balanced output, what you need is two signals in opposition - that is, you split the intended signal, send it unaltered onto one wire, and then flip its polarity (that is, multiply the voltage by -1) and send that to the other wire.

This is malarkey, as has been pointed out by luminaries like Douglas Self and Bill Whitlock. Having two voltages in opposition is irrelevant; if that mattered, then when the signal was zero (dead quiet), noise would no longer be eliminated. What actually matters is that the impedance on the two legs is balanced, so that any induced common-mode noise is the same on the two legs.

This is pretty old news, but has been largely ignored in the industry. So I was not surprised to discover, on tracing the circuit in my client's equipment, the following "balanced" output stage (I've eliminated a few unimportant details, like DC blocking capacitors):

It does just what it was meant to - the voltage on pin 3 will always be -1 times the voltage on pin 2. But look at what happens when the AC adapter is plugged into the wall:

There's always a little bit of leakage between the windings in a power transformer, perhaps a few picofarads. 120V from the wall leaks through that capacitance, into the power supply, into the signal ground. From there, I've traced the two main current paths to the output. Notice that one path goes through about 21.8k of resistance before getting to the output, while the other sees only 1k.

This 22:1 imbalance, with the tiny leakage through the transformer, was enough to generate a couple mV of differential signal - a lot, in the terms of audio signals, which rarely exceed one volt. By replacing this output stage with one that was truly impedance-balanced (based on an SSM2142 chip), I was able to reduce the noise signal by a factor of 30 - enough to get it below the noise floor of the unit.

The cost of the extra components was about $8 retail. But frankly, if cost were an issue, it might have been just as good to have gotten rid of the entire inverting amp stage and simply connected pin 3 with a 1k resistor to ground. The impedances would be balanced. The differential voltage would only be half as big as before, but the common-mode noise voltage would be reduced by much more than half, so signal to noise ratio would be better than with the "balanced" output the designer came up with.

Many software errors come from using a common design pattern without understanding the problem it's aimed at. This was the same mistake in hardware.

Comments

2008-11-16T15:11:00.000-08:00

I often hear that comments in source code are A Bad Thing. Comments are evil because they don't accurately describe the code they apply to; because the code gets modified and the comment doesn't; because good code is self-documenting and therefore doesn't need comments; and because you need to read the code anyway to understand what it does and so comments just get in the way.

Horseshit.

This is roughly like saying that synchronization is evil, because it's often done incorrectly, because it gets broken when people update the code without fixing the synchronization, and because good code uses constructs like immutability that don't require synchronization.

If we treated comments as being as important as synchronization, they'd live up to their end of the deal just fine. There is nothing inherent in the idea of a comment that renders it impotent. Think of comments as being like error-handling code or synchronization: bulky, hard to write, even harder to test, but crucial to reliability.

I think the real reason so little code is commented is simply that most code is written in short bursts of effort by highly productive individuals, and while they're writing it, they understand their own assumptions well enough to not need the comments themselves, and they're in too much of a hurry to worry about the next fellow. And because this is what new programmers then see all around them, this is how they in turn learn to program.

If we built buildings this way, instead of having architectural drawings, the carpenters would come in after the foundations were poured, look at where the rebar was sticking out, and frame the walls where it looked like they should probably go. The resulting buildings would be ugly, short, and tend to collapse in a few years. Much like software, in fact.

If I were to design a programming language, any uncommented abstract definition (for instance, a method definition in an interface class) would be a compiler error. Yes, people would work around it by putting in useless comments, but it would be a start.

Control Theory

2008-11-11T23:47:00.000-08:00

In the last post I mentioned that we programmers don't generally get training in control theory. This is too bad, because I think learning to recognize the behavioral modes of feedback-controlled systems can have a lot of practical benefit for us.

Suppose that you have the following two Java classes, B and Main. Without running this, can you tell how it will behave?

If you do run it, you'll see that the output is spread across a range of numbers. Without changing class B, how would you make that range smaller? What properties of the code determine the range?

This program is an analog of an elementary problem in control theory, so basic that even my espresso machine implements a solution to it. But I suspect that most computer programmers, coming upon behavior like this while performance-tuning an application, wouldn't immediately recognize it.


public class B implements Runnable {
  private static final int ELEMENTS = 100;
  private double[] v = new double[ELEMENTS];
  private boolean state;
  private volatile boolean done = false;
  
  public void run() {
    while (!done) {
      synchronized(this) {
        v[0] = state ? 200.0 : 0.0; 
        // propagate gradually through the array
        int i;
        for (i = 1; i < v.length - 1; ++i) {
          v[i] = (v[i - 1] + v[i] + v[i + 1]) / 3.0;
        }
        v[i] = (v[i - 1] + v[i]) / 2.0;
      }
      try {
        Thread.sleep(1);
      } catch (InterruptedException e) {}
    }
  }

  public synchronized int get() {
    return (int)v[v.length - 1];
  }
  
  public synchronized void set(boolean state) {
    this.state = state;
  }
  
  public void finish() {
    done = true;
  }
}


public class Main {
  final Object l = new Object();
  final long end = System.currentTimeMillis() + 60000; // 1 minute
  
  public static void main(String[] args) {
    new Main().run();
  }
  
  private void run() {
    B b = new B();
    new Thread(b).start();
    
    while (System.currentTimeMillis() < end) {
      double t = b.get();
      boolean state = t < 50.0;
      System.out.println("t = " + t + 
        " - state is " + (state ? "true" : "false"));
      
      b.set(t < 50.0);
      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {}
    }
    b.finish();
  }
}

Self-reference and Statistical Density

2008-11-05T15:16:00.000-08:00

Programming languages are ways of mapping between three things: a set of problems in some problem domain; a set of envisioned solutions to those problems in a computer programmer's mind; and a set of instructions that can be executed by a computer.

Computers are very stupid, and not at all lazy. Programmers are very lazy, and hopefully quite smart. So, the language has a big gap to close.

For example, suppose I have an object that stores some numbers "a", "b", ..., "y", and "z", and I want to provide "fudge" and "muddle" operations on these numbers. I could write functions called fudgeA, fudgeB, muddleA, muddleB, and so on. Boring! Once I've written the code for fudge and muddle for A, I'd really like to just say "and do the same thing for the rest". So I want a language with that sort of expressive power, a language that can contain concepts like "the rest" and "the same thing". A language, that is, that can refer to itself, not only to the entities in a problem domain.

Taken to the extreme, what this leads to is programs of maximal statistical density: that is, programs that contain no repetition, but lots of self-reference and automatic code generation. There may in fact be many nested layers of this stuff.

As with Zappa's opus of statistical density, The Black Page, the problem in practice is that this becomes devilishly hard to play. Highly self-modifying programs save the original programmer's time, but generally make life for everyone else worse. The goal of making code more concise is seriously flawed. The goal should be to make code more understandable, not more concise.

I say "generally", because there's one case where it works. That's when the self-referential features of the language fit with the way that programmers think about solutions. For instance, "do the same thing to all these numbers" is the way that a programmer would typically describe the solution; to have to spell out each function separately isn't just repetitive, it makes the program code be less like what's in the programmer's brain in the first place.

My point is that I think that language designers should learn more about how human minds work, and should prefer native constructs and shy away from constructs that require unusual intelligence and training to master. Computers do not think like people; so programming languages need to.

There is a lot of overlap between computer science and cognitive psychology, but so far as I know this premise has not been applied; quite the opposite, it seems that new programming languages are more often used as a vehicle for showing how smart the programmer is.

Specifically, there are certain constructs that are demonstrably hard for most people to reason about. The ability to reason about recursion, for instance, is often used as an interview test to separate the "real programmers" from the mediocre. Recursion is actually one of the few things that computers innately know how to do, so it's not surprising that it's in computer languages too; but if it makes it so that only really smart people can write good programs, then I think it's something we should be trying to get rid of, not use more of. If a car is hard to steer for people of normal strength, we give it power steering, we don't just assume that Driving is a job open only to the abnormally strong.

Of course, I'm blowing into the wind here, because cognitive psychology has not held up its end of the deal. Language designers can't learn much more about how the mind works, because cognitive scientists haven't figured it out yet either. We know very little about what processing contructs are "native" to the mind.

But as a starting point: three things that seem hard for most people are feedback loops (that is, control theory, home of things like proportional-integral-derivative algorithms); recursion (especially if the rules change depending on the depth); and n-dimensional geometry. Electrical and industrial engineers get advanced training in handling feedback loops; I've never met a computer scientist who has, and I've never met a non-specialist who has. Physicists and mathematicians work with n-dimensional manifolds, but EEs and programmers rarely go beyond three. Programmers get advanced training in recursion, but no one else does. In each case, the fact that training is required and that these are all exclusive fields says something about the cognitive difficulty.

The question is not how we can make programming more efficient for a vanishingly small number of people. The question is how we can make it more efficient for a larger number of people.

Maven Sucks

2008-10-27T23:18:00.001-07:00

Tools like Maven make me want to quit the software industry before I die of frustration. I've wasted basically the last two days screwing around with problems related to Maven. In a nutshell, Maven does not provide a consistent or reliable build environment.

Modern software is not written from scratch; it's assembled from myriad components from other vendors, in much the same way that car manufacturers buy their parts (seats, brakes, wiper motors, radios) from other companies. And just like a car seat is itself made of springs and fabric and metal bits that come from yet other manufacturers, each software component may itself be assembled from other components.

Maven is a tool to help manage all the little pieces that fit together into a software product - to manage the fact that my product consists of components X, Y, and Z, which in turn require the presence of P, Q, R, and S, which in turn require A, B, and C; to fetch those components from wherever they come from if they're not already on my computer; and to make sure that the versions of these different components are all compatible.

But it sucks. Maven wants to update things without me asking, so that if I run a test twice in a row I don't necessarily get the same results. Maven wants to download things without me asking, so a task that took 15 seconds the last time might take a few minutes, or fail utterly (if I lose my network connection), leaving me in an unknown and inconsistent state where I can't build at all. And most frustratingly, Maven is itself highly modular, meaning that it's not all downloaded until it's needed, meaning that in the event of network trouble it can again fail utterly. Networks are not yet reliable enough for that to be okay.

Further, Maven seems to do a crappy job of understanding and resolving or reporting version conflicts. I'm extending some test code, that's supposed to exercise version 3.2.5 of product HappyWidget. The test code therefore tells Maven that it needs HappyWidget version 3.2.5. But the test code also needs OtherThing version 2.7.1; which needs FussyBit 2.0.1; which needs HappyWidget 3.1.2. Whoops! This is a version conflict. It would have been easy to fix, except that Maven didn't tell me there was a problem, it just silently deployed HappyWidget 3.1.2. So all along this test was actually testing HappyWidget 3.1.2. What's really scary is that when I uninstalled and then reinstalled everything, without explicitly making any changes in my dependencies, the behavior changed - now it's deploying 3.2.5, which is nice for my test but not so nice for the corresponding code that is supposed to test HappyWidget 3.1.2.

The first requirement of a build tool is that it should behave predictably. Maven fails.

It's late and I'm not going to spend more time saying all the hateful things that Maven has done to me in the last two days. I will blog in the future about what I think the solution to this should be.

Synchronization defines structure

2008-10-23T09:40:00.000-07:00

Consider the following code snippets, from some Aspectwerkz code:


  public synchronized ClassInfo[] getInterfaces() {
    if (m_interfaces == null) {
      m_interfaces = new ClassInfo[interfaces.length];
      // etc.
    }
    return m_interfaces;
  }

  /**
   * Returns the super class.
   *
   * @return the super class
   */
  public ClassInfo getSuperclass() {
    if (m_superClass == null) {
      m_superClass = m_classInfoRepository.getClassInfo(superclass.getName());
      // etc.
    }
    return m_superClass;
  }

Notice that the first method is synchronized, and the second is not. How come? Is this a bug in Aspectwerkz? Both methods seem to require synchronization, because the "check something and then conditionally change it" pattern is otherwise unsafe.

My inclination is to say that it's just a bug. But it might not be; there might be some external reason that the second method does not need to be synchronized here. For instance, it might always be called from within another synchronized block (though the fact that it's got public access scope makes this hard to enforce).

The point here is that synchronization (almost by definition) implies a particular call structure: to correctly synchronize a particular body of data, you need to know how that data will be accessed, by whom, in what possible sequences of events. You can't just put the "synchronized" keyword in front of every method, because over-synchronization leads to deadlock; you can't just synchronize every method that changes state, because you won't get the right visibility guarantees. You have to actually know what the code is doing, to correctly synchronize it.

This is a huge problem for two reasons. First, while you're coding, you're changing structure, so it's hard to keep up; thus, synchronization bugs creep in. In the above example, it's possible that the second method was originally private (and always known to be called from within some other synchronized block), and then someone changed it to be public without updating the synchronization. Second, it makes it much harder to change code locally: you have to understand the overall behavior of the code in more detail than would otherwise be needed.

Which brings me to the main point: unlike a lot of code, synchronization is not self-documenting. It is simply too fragile and opaque. I cannot look at the above code and figure out what pattern the developer had in mind, what assumptions s/he was making. When maintaining code, I want to preserve assumptions or else systematically and thoroughly change them. I can't do that if I can't even discern them.

As a side note, isn't that Javadoc something special? Really, "returns the superclass" is easy to figure out from the method name. What I need to know from this method's documentation are things like "under what circumstances might it return a null?" and "are there initialization requirements before this method can safely be called?".

I Blame My Tools

2008-10-22T15:48:00.000-07:00

Computer science was one of the things I studied a long time ago in school. We learned how well-chosen algorithms can operate on well-chosen data structures to achieve powerful results. The work is highly conceptual and rather intuitive - like calculating differential equations, it relies more on a flash of inspiration, of being able to see the problem in the right way, than on methodically "turning the crank". There are no good algorithms to generate good code.

That maybe described computer science, but it describes only a very small part of the day to day work of computer programming. The reality of computer programming, at least for me, is that most of my time is spent wrestling with tools and technologies that don't do what they're supposed to. Metaphorically speaking, I don't get to envision graceful bridges and soaring skyscrapers; instead I futz around with a load of concrete that won't set, my lumber delivery is delayed till next week, and the extension cord doesn't reach from the outlet to my power saw.

I'm trying to do a little bit of programming work today. I use the Eclipse programming editor. Somehow the shortcut I use to start multiple instances of Eclipse on the Mac got turned back into a plain text file - I have no idea how. I got that sorted after half an hour or so. Now I want to build my project but I can't because someone added a dependency on another module of code that I don't have. I downloaded that, and built it, but in so doing I triggered some sort of version check and now it's complaining that my version of Maven, the build tool, is impermissibly out of date. (That Maven versions matter at all is a sign that Maven is trying to do way too much.) So now I need to download and install a new version of Maven. This is what my day has been like, all day long.

I know, it's a poor craftsman who blames his tools. But I blame them anyway.

Follow-up: my command-line tar utility (the Mac equivalent of "unzip") won't recognize the format of the Maven download file. Finder won't let me copy the files to the directory they need to go in; I don't have permissions. On my own machine.

Follow-up #2: I used sudo to let me copy the files. I used diff to see if the settings file had changed. It shows me that every line is different in some mysterious way that is not evident from looking at the files - perhaps the line endings changed between Windows and Unix style? Anyway, ignoring that, I then futzed around trying to change my old symbolic link for Maven to point to the new copy. That took a bunch of googling because I don't know how to create and delete links on Unix. All this is just so that I can run the build tool, to build the project that the changes I'm supposed to be working on will affect. I haven't even begun to actually do the work I'm supposed to be doing. It's quarter till 5.

Synchronization and Relativity

2008-10-10T11:33:00.000-07:00

Thinking about state as a way of reasoning about synchronization seems like a good approach. But the problem is, the concerns I have about synchronization are often about execution: will it deadlock? Will there be lock contention? Reasoning purely about state leads me to write programs where the data is always correct but nothing can actually complete. Both state and execution are important, and they're kind of like matter and energy, seemingly unrelated concepts.

Einstein managed to show that energy and matter were actually two sides of the same coin: that a certain amount of matter was, in fact, equivalent to a certain amount of energy, if you chose the right units. Particle physicists took this and ran with it, coming up with the idea of symmetry breaking and explaining the circumstances it takes for the coin to flip. I need someone to do the same for multithreading. I want a theory of multithreading relativity that explains how given multithreading constructs act on execution and state, and what the symmetries are.

For instance, if you protect state with a mutex (in Java, a "synchronized()" block), then you can debug deadlocks by asking the system what threads own what locks. But you can also protect state by waiting for a notification event; this is a common way to implement a reader/writer pattern. The result is similar: the state is protected at the expense of some thread being blocked. But when a system deadlocks because every thread is waiting for a notification, there's no way to ask the system which thread was supposed to send it.

You'll never make a program hang by removing a synchronized(), but you might make a program hang by removing a notify(). On the other hand, you'll never make a program corrupt data by removing a notify(), but you might make a program corrupt data by removing a synchronized(). Is this a real symmetry?

Similarly, it's possible (but maybe not algorithmically possible) to look at code and see at least the hallmarks of deadlockability: for instance, code that takes nested locks out of order. Is there an equivalent analysis for code based on wait() and notify()?

In electrical engineering, it's possible to take any circuit based on voltage sources and impedances, and convert it (using the Thevenin and Norton theorems) to a different but equivalent circuit based on current sources and impedances, that will behave just the same to an outside observer. Doing this is often very useful for understanding how a circuit works. Is this possible for synchronization? Given some code implemented with certain synchronization tools, is it always possible to reimplement that code using different synchronization tools, such that the behavior will be the same? What do the rules of that conversion look like, and what will I learn about the underlying synchronization pattern by doing this? Exactly what "behavior" will prove to be invariant?

Synchronization is Hard

2008-10-09T14:53:00.000-07:00

I read a newspaper story about a neuropsychologist who had a stroke. She recounted trying to call 911, but not being able to figure out which digit was which on the phone, or what the steps were to make a phone call. She knew what the right tool was, but she'd lost the cognitive tools to use it. All the while, being a neuropsychologist, she was aware of what was going on, and even somewhat fascinated by it, but also aware that her life depended on doing a seemingly simple thing that she nonetheless could not quite grok.

Synchronization is like this for me. At least I know I'm not alone - some of the smartest people I know have a hard time thinking about synchronization problems, and my industry is littered with bugs due to incorrect synchronization. But I always feel like there's a right way to reason about these problems, and I know it's there but I don't know what it is and I can't even quite articulate why it is that I can't think clearly about it. My hope is that one day I'll GET IT and then I won't be able to remember why I couldn't figure it out before.

I do know some wrong ways to think about synchronization, though. Any time I am reasoning about synchronization and I find myself thinking "okay, if two threads come in here at the same time...", I am about to make a mistake or go down a rat hole. This is how books always present the topic, but intuitively I think it's wrong - I don't believe you can think correctly about synchronization by thinking about execution.

Instead, I think it's probably better to reason in terms of state. "What states could this object be in when this variable is evaluated?" "If I modify the state, how will other threads discover the modification?"

Today I spent a couple hours trying, along with some people who are pretty good at these things, to come up with a good pattern for lazy initialization when the initialization routine is not trusted (e.g., when it might try to call back into the object being initialized). The real moral of the experience is twofold: first, we each wrote routines that we thought were good and that were promptly found wrong by the others; second, although I think we did end up with two valid solutions, I'm not sure how to PROVE that they're valid.

Hibernate locking

2008-10-08T15:13:00.000-07:00

To continue the last topic: I'm working on a program that lets examinations be taken online. It uses Hibernate to get stuff from a database - for example, we get a list of possible answers to an exam question. The list is implemented by Hibernate, so that it doesn't actually read the database until the first time someone tries to access the contents of the list. But that fact is invisible to our code - to us it just looks like an ordinary Java list.

Now, like the ordinary Java collections, the Hibernate-provided collections don't have any built-in synchronization. If you tried to read the list from two threads at a time, it might try to initialize the collection twice, or not at all, or it might just crash.

This is not a problem for most uses of Hibernate. Generally any given Hibernate collection is only accessed by one execution thread, even though there might be zillions of Hibernate collections all referencing the same table in the database.

But in our application, we use the same collection from a lot of threads - perhaps tens of thousands, distributed over a cluster of machines with Terracotta. We never modify it, we just read it. Except for the very first time it's accessed. But there's no way to tell, from the outside, whether it's the first time - so we have to treat it as if it might be, every time.

This pattern does not work well. We could wrap the collections inside "synchronized" blocks, like the java.util.Collections$SynchronizedWhatever classes do; but that means that every time any thread tries to read an entry in the list, it has to wait for every other thread to get out of the way first, just because once upon a time one of those threads did the initialization.

Like I said in the last entry: replacing the implementation without changing the interface is powerful, but it means there's no way to know whether an operation is actually a read or a write. Locking is one reason why a caller cares about implementation.

The solution is to change the Hibernate code so that it does its own locking, using a read/write lock. Within a single method, it can take a read lock to figure out whether initialization is needed; if it is, then it gives up the read lock and takes a write lock to do the initialization. The first time through, a write lock will be taken, but (nearly) every time thereafter, it'll only need a read lock, which means that no thread will ever have to wait. In practice this is very effective. In one performance test we saw a roughly 200x speedup: latencies went from 4 seconds to 20 milliseconds.

The unfortunate part is that the code gets messier. This nice code:


  int size() {
    if (!initialized) {
      initialize();
      initialized = true;
    }
    return size;
  }

Becomes:


  int size() {
    readLock.lock();
    try {
      if (initialized) {
        return size;
      }
    } finally {
      readLock.unlock();
    }
    writeLock.lock();
    try {
      if (!initialized) {
        initialize();
        initialized = true;
      }
      return size;
    } finally {
      writeLock.unlock();
    }
  }

Amidst all the locking and unlocking and multiple checking, it's hard to see what's actually being done by the method. Which gets back to my first post.

The two R's

2008-10-06T22:51:00.000-07:00

Computer software consists of a lot of instructions that read and write information to memory. The basic idea hasn't changed since World War II. Imagine a gazillion toggle switches, each of which can be flipped up or down. That's the memory, and then there's a processor that runs instructions, what we call the "code". The instructions are like "go look at switch 3,452,125 and see if it's flipped up. Then go look at switch 35,289 and see if it's flipped up. If they're both flipped up, then go find switch 278,311 and flip it down." And so forth. Some of the switches do things, like turn on a pixel on the computer screen. Others are just there to remember. We have nice tools so that we don't actually have to use numbers for the individual switches when we write the instructions, but under the covers that's exactly what's going on.

I work at a company called Terracotta Technology. We make software that connects many computers together so that they can solve problems bigger than one computer could handle. We make a sort of virtual computer, that other people's programs can execute on. We fool the programs that run on Terracotta into thinking they're running on a normal computer. A program thinks it's flipping a switch on its own computer, when actually it might be on some other computer.

So, we care a lot about when the programs try to read from memory and when they try to write to it, because we have to intercept all those operations. If all they want to do is read, we don't have to do as much work. Flipping a switch, in our world, that's real work.

This idea of replacing what's under the covers without changing how it looks to the software that's using it is not original - it comes up over and over again in software, in fact it's probably the single biggest idea the industry ever had. Hibernate is another product that does something like this. Hibernate takes reads and writes to memory, and supports them with reads and writes to a database, which is more reliable and persistent and searchable. A programmer could just write instructions to talk directly to the database, but Hibernate makes it easier by hiding some of the complexity under the covers.

But the illusion breaks down. The picture up above is what happens when you ask Hibernate how many things are in a list. Asking how many things are in a list shouldn't change it, right? That's common sense. But asking Hibernate how many things are in a list might change memory, because it might have to go fetch the information from the database and then save it in memory.

So if you want to figure out whether an operation is a "read" or a "write" or both, you need to know who's responsible for performing it. And that's something that can change on the fly, because we're so good at replacing what's under the covers.

But why do we care about the distinction anyway? Are reading and writing really the only way to compute? Why should this implementational distinction matter to a programmer?

Is this the path to functional programming?

Ignorance is the mother of invention

2008-10-06T20:40:00.000-07:00

I spend too much of my time trying to figure out what pieces of software do.

In 2008, generations after we started programming, it is still almost always easier to write an okay prototype yourself, than to use an existing component that someone else wrote. This is nuts. This is why software engineers like me still get paid as much as we do. I appreciate the pay, but it's holding back the industry.

The reason for this is that with very rare exceptions, we engineers suck at saying what the code we write does, and for that matter we also suck at coming up with code that does things that can be succinctly described. The rare exceptions have names like Josh Bloch.

Good software engineers excel at looking at other engineers' implementations - at the program code that they wrote - and figuring out what it does. (I am not very good at this, and I admire it in my coworkers.) That is because it's the only way to survive in the industry; there is no other way to figure out what a component does.

Imagine that, before going to the loo, you needed to trace the plumbing to make sure that it actually exited to a sewer rather than the drinking fountain. Imagine that before starting a rental car you needed to trace the ignition wiring to make sure that turning the key clockwise wouldn't break the timing belt. This is the state of the software industry.

When one piece of software calls another piece, it needs to give it some data, and then it expects some data back. The important things that it needs to know include: what is the range of data that it can safely pass in? What is the range of data that might be returned? Will anything change as a result of the call? Is it okay to call again, before the first answer comes back? If the rules are broken, how bad are the consequences?

The tools used in the mainstream software industry do not answer ANY of these questions. Instead we have "comments," which are bits of text written by the software engineer, hopefully describing the situation. This does not work, because comments (a) are not required; (b) are written by software engineers, who are often not very good writers; (c) do not have any validation; (d) are often not updated when the code is updated.

To continue with the loo analogy, it's as if the plumber put a sticky note on the toilet. The plumber assumes that you know the things that were obvious to him and focuses on the details you might not know, so the note says something like "sewer line is made of cast iron," rather than what you really need to know, "flushing toilet will cause contents to be safely sent elsewhere." But it doesn't matter, because at some point someone else went into the basement and replaced the cast iron with PVC, without realizing that there was a sticky note on the toilet.

This is the most serious problem the software industry faces. Arguing about whether Java 7 should include closures is irrelevant. We need enforceable, validatable contracts that describe how software components can correctly be used.