It’s tempting to ascribe this to the “cult of no-pay”, programmers and users who simply won’t pay for software no matter how good it is, or how inexpensive it may be. These people used to be called pirates. Now they’re open source enthusiasts.

And there’s been the predictable outcry of blog posts reading that as saying that open source enthusiasts are pirates, and saying things like this:

Jeff Atwood’s blog Coding Horror is one of my favorites. Until yesterday, I’d been recommending it unreservedly. Jeff’s made a big stumble, and I hope he corrects it soon….

Right. First off, anyone who’s read Jeff’s blog and is halfway observant knows a couple of things about him. He’s intelligent, and he’s outspoken about his opinions. If he wanted to say that open-source enthusiasts were the same as pirates, he would come right out and say it, and give you several paragraphs of reasons why he thought that. He wouldn’t just imply it in a throwaway sentence like that. So it ought to be obvious to any reasonable reader that that’s not what he intended, regardless of how he worded it.

More than that, though, I find the reaction of “I can no longer unreservedly recommend this blog” to be quite interesting. Suppose that Jeff did intend to say that open-source enthusiasts were pirates? I don’t agree with that, obviously, but so what? I don’t think I’ve ever read a blog that I completely agree with; that’s what makes conversation worthwhile, and if he has a good argument to back it up, it’s probably worth reading even if I think it’s erroneous. To me, what makes a blog worth recommending is when it presents intelligent arguments based on reasonable facts, in a way that I find interesting and thought-provoking and on subjects that I think are worth reading about. Making a “stumble” doesn’t change any of that.

The idea that a recipe (eg, a best practice, a methodology, or a maturity model) is completely worthless is just as wrongheaded as the idea that everything should be based on a strict recipe. In other words, even an expert chef may occasionally find it helpful to refer to a recipe card.

There’s no reason these two models can’t coexist. You should always start with the common denominator recipe, of course, but you may want to provide some alternative guidelines and ideas for those cooks who have outgrown traditional recipes, too.

I strongly agree with his general point, but I think that “guidelines and ideas” are the wrong sort of thing to include.

To take the metaphor literally, for some things, I’m a level 4 cook. (Not necessarily an especially good one — it’s worth noting that these levels are not always parallel to the quality of the output!) A particular example would be some coleslaw I made a couple of weeks ago, when I found myself in an ill-stocked kitchen with a cabbage, an empty stomach, and the Joy of Cooking.

The Joy of Cooking is a pretty good cookbook for this sort of thing, as it turns out. It has a half-page discussion of what coleslaw is, without any recipes at all. It turns out that pretty much the only common factor in slaw is that it has cabbage in it. Everything else is variable. (This was an important point; we had no mayonaisse, and the person I was cooking with insisted that coleslaw has to have mayo.) It went on to describe, in loose terms, some of the common varieties. I skimmed through those, and then went on to look at the actual recipes. The standard recipe format is well-optimized for this sort of thing; there’s a list of ingredients at the top, separate from the instructions, making it very easy to skim to pull out specific bits of data such as the ratio of sugar to vinegar in a sugar-vinegar slaw.

After looking through the cookbook, I’d gleaned the following information: sugar and vinegar are a perfectly sufficient base for a slaw dressing, and it’s common to have them in X ratio, Y is a typical amount of salt, and Z, W, Q, and a dozen other things are some ideas for spices to put it.

Those, you may object, look exactly like “guidelines and ideas”. So, why am I objecting to including guidelines and ideas for experts in the recipe? The same reason that specific instructions are wrong for experts: They are processed information, and the definition of “expert” in this context is someone who does their own information processing. In my case, when I read the recipes, I was mining them for bits of information. Any processing that complicates that information-mining is a detriment, not a benefit.

And so, when writing recipes to be useful beyond basic instructions, the thing to include is facts. Not a guideline of “this squash should be cooked for 20 minutes”, but the fact of “if you cook this squash for more than 25 minutes it will be mushy and begin to lose flavor.” Not the idea of, “You could use peaches in this,” but the fact of “Peaches are also common in this.” The guidelines and ideas are certainly useful because they contain facts, but the parts that make them “guidelines” and “ideas” rather than bare facts rarely are. Often the process of distilling the fact into a guideline or idea loses useful information, too; suppose I wanted mushy squash, or wanted an uncommon alteration?

I’m going to claim some authority in talking about this, because I’ve done that — and it wasn’t just 20 or 30 hours, either. A year ago, I thought that there was some chance that I could finish my dissertation by the end of the academic year if I got a certain set of simulations done, which required taking a computer program that was currently held together by Duck-brand Fortran Tape, and turning the structurally unsound mass of the core algorithms into a well-oiled robust, solid, machine that would do what I told it. I had two weeks in which to either accomplish it or give up.

So, I sat down on Monday to start working on it, and for all practical purposes, I worked on it straight through until Wednesday of the following week.

This is the first flaw of the Nerd Bravado claim. Hyperfocus doesn’t care if you sleep and eat and take a shower. You can think while you’re eating. Your brain keeps working on things when you sleep, and if you think about a problem when you go to bed and when you get up in the morning, you’ll often find that some tricky bit of it has become a lot clearer in the interim. You don’t have to sit for 25 hours straight to get the benefits.

That’s where the benefit of this style of working is, insofar as there is one: Hyperfocus. After a dozen or so hours into this project, I was so focused on it that I couldn’t really think about anything else — even if I tried, the problems in the code would keep distracting me back to the task. And, because I’d pretty much emptied everything else from short-term memory, quite a lot of stuff about the code could fit in there — enough to contain all the complicated interweavings of how the code all fit together. I could look at the results and see where the problems were coming from and what bits of subroutines wove together to create the particular effect that was happening, and I knew, without having to look, where to prod the program to change it, and when I prodded it, it changed. It’s a heady feeling.

I think that’s what the appeal of Nerd Bravado is, really. It’s like spice, from Dune. Get a bit of it into your system, and the universe opens up before you and whirls at your beck and call, and you have tremendous power at your fingertips to poke exactly the right thing in exactly the right place. But it’s also got downsides; do it a few too many times, and your eyeballs turn funny colors and it completely eats your life.

Now, I said I had two weeks to fix this program, and I stopped a week and a half in. Do you think that means I finished early? Hardly. A week in, and I’d gotten all the easy bugs fixed and most of the complicated ones, and I’d started tangling with the algorithms at the core of it. And they had a problem: The math that I was working with involved solving an equation where significant parts of both sides are multiplied by a factor that goes to zero. Divide that out to solve it, and very small errors in one term become very big errors in the solution, and it stops being well-behaved. There are ways around this, but they have to be designed into the algorithm from the start, not bolted on later. And by about Wednesday, it became quite clear that I was dealing with something that needed to be fixed from the ground up, and there was no way I could come close to that in the three days I had left.

This is the second flaw of the Nerd Bravado claim. It doesn’t actually solve the real big problems. Hyperfocus is a state of working on instincts, of having enough information actively in your brain that you don’t have to work through logic and deductive reasoning to know what the Right Thing To Do Now is. And that’s great, if what you need to accomplish is something that’s a chain of consecutive steps, like debugging a convoluted program or working through a complicated mathematical derivation or even writing a new program that fits a general specification that you’ve already determined.

Real big problems aren’t like that. Flash-of-insight problems aren’t like that — Einstein didn’t have his flash of insight about time dilation on the trolley because he’d been hyperfocusing on the problem for the past few days; he had it because he’d been rolling the problem around in his head for months. (Or years; I don’t actually know that bit of his biography well enough to say for sure.) And hard-work problems aren’t like that, either — writing a fixed version of my program isn’t going to work like that either, because it needs to be designed right from the ground up, and designing something right is a process that requires awareness of the outside world, and of a lot more than what fits in the hyperfocus. Yes, there are the legends of the great designers who sit down and in one day lay out the basic lines of an airplane or a car engine that are more right than their less-great colleagues create with a committee and a month, and that comes from having tremendous amounts of knowledge in their heads, but that’s the sort of knowledge that gets there over years, not hours.

This brings me to the third flaw of the Nerd Bravado claim: It’s not, overall, worth doing. When I’d finished my week-and-a-half of debugging, I was mentally exhausted. I was a week behind on all of the minutiae of daily life, I was emotionally starved for interpersonal interactions, I’d barely talked to my wife about anything that mattered to her, and my brain felt thoroughly fried. (My wife’s ironic laugh when I asked her if she remembered when I did this probably tells you all you need to know about what it was like from the outside.) It took somewhere around a month before my work productivity was anywhere near normal again. All totaled, I’d probably have accomplished as much if I’d simply worked at a reasonable pace for the month and two weeks, and I’m pretty sure the resulting solution would have been worth a bit more, as I’d have had the time to actually work out at least a first cut at a ground-up solution.

And, finally, the fourth flaw of the Nerd Bravado claim: the claim that “you must do it at least once to be a successful engineer … and to gain the respect of your peers” simply doesn’t hold true. The result of this particular experience, for me, was that I had a nicely debugged program that solved the equations the wrong way and didn’t produce useful answers. And all I learned from it was this: That style of working isn’t sustainable, and doesn’t lead to an enjoyable life, and I don’t want to do it again except in very small doses. As for the respect of my peers — well, does this cause any of you to respect me any more? No? I didn’t think so.

• The “listings” package for LaTeX is invaluable for including computer source code in a document. It’s something like syntax highlighting in a source code editor, but for typeset output rather than on-screen, and very easily user-customizable. Carsten Heinz spent almost a decade writing it and perfecting it, and it’s a marvel of intricate TeX programming — 218 pages of typeset user documentation and source code. Then, about a year ago, Carsten disappeared. As far as I know, nobody in the TeX community knows what happened to him; he simply stopped replying to email, and letters to his last known physical address just disappear. As a result, the official version of the listings package has effectively been frozen for the last year, despite a couple of important bugfixes that have been informally circulating. Luckily, this isn’t a permanent situation; it’s licensed under the LaTeX Project Public License, which provides a process by which the maintainership of the “official version” of a package can be transferred to someone else in such a case.
• In a blog post I was reading this morning (I’m leaving this vague so as not to point fingers excessively at only one example of a common problem), the author mentioned a general-purpose data structure that he had included in his open-source simulation code, and provided a link to the source file for the implementation. Like many such implementations, it’s sufficiently general that it would be useful in many programs that I’m writing, and it’s a better implementation than I could come up with (and test!) in a day’s work. However, there’s a problem: The license for this software package is not quite the same as any common software license, and it requires that its particular set of conditions and disclaimer be included in any redistribution. Further, the license has the relatively common clause that any scientific publications which contain data resulting from use of the program must cite it.

I’ll start with the last item. It’s completely reasonable that the author of a piece of scientific software would want to be cited when their work is used, so what’s wrong with putting that in the license? The answer is a lesson that should have been learned years ago. It only makes sense for cases where what’s being redistributed is pretty much the same as the original work. Beyond the well-known problems described at that link (short form: someone else builds on it enough to merit citation and adds their paper, and after a few repeats of that it quickly becomes unweildy), there’s the problem of partial reuse: if I include this little data structure implementation in my program, according to the license I then have to cite their paper in all of mine, even though my work is in a quite unrelated field. And so does anyone who uses my program. While that might arguably be appropriate for an advertising clause, it’s not appropriate for scientific citations.

So, functionally, I pretty much can’t reuse their data structure in my own software, if I wish to do so under its published license. That’s completely contrary to the spirit of free software; even though it’s open source and I can read it, the license prohibits me from simply copying it and reusing it unless I’m willing to agree to inappropriate terms. This almost certainly isn’t the intent of the program’s author; he simply didn’t consider the possibility of someone reusing a small piece of his program when he decided how to license it.

In this case, there’s probably a simple solution. If I want to use this code, I can write to him and ask for permission to use it under a different license than the one it’s published under — specifically, one which doesn’t contain this citation clause. Probably he’ll agree, and that will be that.

However, this is where the first example is relevant. What if it was something in Carsten’s code that I wanted to reuse, and he had a similar sort of clause in his license? Plain and simple, I’d be stuck. Even if I were willing to ignore the license on the assumption that his heirs wouldn’t find out or wouldn’t care enough to sue me, I certainly couldn’t relicense the code with a clear conscience.

That’s why license compatibility is important, and why it’s a bad idea to add private clauses like the “cite my paper” clause. Without it, free software isn’t a commons — it’s a balkanized set of little gardens fenced off from each other and unable to cross-fertilize. We’ve had this problem with railroad tracks, with fire hoses, with all sorts of things early in the industrial revolution; as engineers, you’d think we’d know better by now. Four feet eight and a half inches probably isn’t the optimal solution for any given railroad in isolation, but it’s certainly better in practice in the real world for nearly all of them, because it’s what everyone else uses, and in the long run it’s just not worth the trouble of being different.

“But,” you say, “I still want people to cite my paper when they use my software!”

Consider this: Most researchers and users of your program are, by and large, honorable and honest people. They understand the value of citing sources, and they appreciate that you’ve written this software and made it available for them to use. For the vast majority of them, all that you need to do to get them to cite your paper about the software is to ask.

License terms aren’t “asking”. They’re a demand, one with legal teeth behind it. They’re what you put in there for the people who you don’t trust to do right, and for the corporations that are too big to have morals at all. They ought to be the final line in the sand for the things that are vitally important: you don’t claim you wrote this, you don’t claim I’ll support it, you don’t put restrictions on it that I disagree with. If it’s not that critical, and if you wouldn’t sue someone for ignoring it, then setting it in that sort of legal inflexibility does more harm than good.

Thus, I propose the idea of adding a “Requests” or “Moral Obligations” section to software licenses, for this sort of thing. When I release a program that’s large enough to merit a paper citation if someone uses it, somewhere near the license clause in the documentation will be a section that says something like this: “The requirements in this section are not legally binding; however, they represent things that the author would appreciate: If you write a paper based on work that uses this software, please include a citation to my paper, and send me a copy of your paper. If you create a derivative work based on a this software, tell me about it and let me know how I can get a copy, and include this request if it’s appropriate.”

In virtually all cases, that will work just as well as adding those as terms to the software license. (How do I know it will work? Well, it already works quite well for the papers themselves, though there the request is implicit rather than stated.) And, importantly, it will work equally well in the cases where something I asked for turns out to be terribly inconvenient for some reason that I didn’t think of when I wrote it, even if I’m not around to change the license terms.

Dave Thomas suggests in his Code Kata post that one way of practicing programming is by solving small-but-complex algorithmic problems, and he has a list of suggestions.

In the shower this morning, I was thinking about a problem that would fit nicely in that list. I’m writing a complicated bit of code that takes the triangles in a triangular mesh and loads them into a data structure. It would be nice to test it by taking a given mesh and loading the triangles in various random orders, and that immediately leads to the classic card-shuffling problem: given a list of objects, how does one efficiently sort them into a random order?

With cards, this is relatively easy at first glance: Put the cards in a stack (i.e., a linked list). Pick a random card from the current stack, and use it to start a new stack. Pick another random card, and put it on the stack. Repeat until the original stack is empty.

In practice, this doesn’t work so well: people aren’t very good at picking really random cards. Or anything else random; this is why random play doesn’t win rock-scissors-paper contests. But that’s a digression; we’ll assume that we have a “good enough” random number generator to use, just like a computer would.

Having a random number generator doesn’t solve the problem, though. It picks numbers, not cards. There’s still the problem of mapping the number to the card.

The trivial solution is to number the cards in the stack from 1 to 52 (thereby making it an indexed array), get a random number from 1 to 52, and pull out the corresponding card. That works nicely for the first card, but then we’ve got a stack of cards that are numbered from 1 to 52, but one of them’s missing. Now what?

We could renumber the cards after the one we pulled out, reducing each one’s number by one so that the deck is numbered from 1 to 51, but that’s still a lot of work – for the whole deck, it’s O(n²).

(Now that I’m writing this out, I see an obvious solution here that I didn’t see earlier. I’ll get to that later.)

Alternately, we could give up on renumbering the cards, and just leave them in a linked list. Then, when the random number generator gives us a number, we count down that many cards from the top of the deck. Again, for the whole deck, we end up counting O(n²) cards.

So, consider an alternate approach: Take the top card off the stack, and put it in a random place in the new stack. This looks like the same problem – how is locating a random place easier than locating a random card – but there’s a twist. Because we’re doing this in a computer, we can keep the cards in two orders at once. We can store them in one order that makes them easy to find, and simultaneously in another order that’s randomized.

First, consider the new deck as a linked list; we pick a card randomly, and insert the new card after it. On a pointer level, this means that the new card points to what the old card had pointed to, and the old card now points at the new card. The trick is that, when picking a random card, we don’t have any reason to care what order they’re in in the linked list. So, if we’re actually storing the cards in an indexed array, and putting each new card on the end of the array, then all we need to do is pick a random index and go straight to that card. Once all the cards are in the new deck, then we can traverse the linked list and pull them out in the shuffled order.

There’s a bit of nice optimization, too; we could pull cards off the bottom of the old deck just as easily as off the top, and since we’re putting them on the top of the new deck (insofar as actual storage location), there’s no need to actually move the cards at all; everything below the current index is the new deck, and above it is the old deck. If the cards are just numbers, there’s no need to even store the cards at all; we just store the pointers.

So, how does that actually get implemented? Start at the beginning of an indexed array of pointers (which are really just numbers). Increment the index. Pick a random bin below the current index. Take the number from the random bin and put it in the bin corresponding to the current index. Put the current index as a pointer in the random bin. Repeat until done.

…

So, back to that parenthetical note from earlier. There’s really no need to renumber all the cards, because they don’t have to stay in the same order. We could just take last card on the deck, and renumber it to fit in the gap, and be done. So, we take a random card out of the deck, put it at the first spot of the new deck, and then move the last card from the deck into the hole that got left. Or, to do this in the same space, we could just pick a card, set it aside, take the first card from the deck and set it in the hole, and put the picked card in the now-empty first spot. Now, the “new deck” is the first card, and the “old deck” is the rest, and so forth as we go along.

How does that get implemented?

Start at the beginning of an indexed array of card numbers (which are essentially pointers). Increment the index. Pick a random bin above the current index. Take the number from the random bin and put it in the bin corresponding to the current index (i.e., the end of the new deck). Take the current index (i.e., the card from the end of the old deck) and put it in the random bin.

An interesting similarity, isn’t it?

There’s an interesting contrast between Rick and Russ’s idea of a good business plan and some parts of the free software world, which in many ways ends up being about the free-as-in-free-beer even if that’s not the actual drive. I think the difference is much shallower than it appears — and I think the similarities are quite important, because I suspect that much of the Web 2.0 hype is based on seeing the success of free software but missing the point of why it works.

To start with an immediately handy example, consider the Windows port of the OpenFOAM computational fluid dynamics software that I’m distributing. On the face of it, my doing this seems to completely ignore Russ’s point – the packages for that represent a good week of my own work (above and beyond the hard work of the people who wrote the original program and created the initial version of the port), and here I am giving it away without charging a penny.

And, if you look at it like that, you’ve probably missed the key point.

My business plan, metaphorically speaking, goes like this: I have the knowledge and experience to develop simulation methods for fluid flows and improve the state of the art in that field. People pay me (in the form of a student research assistanceship, at present, and in the form of research positions after I graduate) to improve the state of the art in ways that benefit them. This is straightforward, simple, and quite clear about who pays me and why.

One of the pieces that’s not in that description is the fact that, in order to advance the state of the art, I need good tools. That’s where creating and distributing the port of OpenFOAM comes in. Creating the port is essentially a sunk cost as far as the distribution question is concerned; I did that because I needed it, and because the experience I gained was valuable. Distributing it, once I’ve created it, doesn’t cost much; thus, it doesn’t take much value to make it worth doing.

So, let’s get back to the key point: What is the value I expect from this? I expect to get feedback from people who use it; bug reports and patches. Those improve the tool, and that means that I have a better tool to use in the core value-generation equation. Moreover, I have a better tool for less than it would have cost me to find all the bugs and write the patches myself, and that’s time I can use for working on the problems I’m being paid to research. That’s worth real money, and the conversion process is pretty clear.

This business plan scales up to actual companies, too. OpenCFD, who wrote and distribute OpenFOAM, have a similar business model. People pay them to solve fluid-flow simulation problems, or to develop systems to do the simulations. OpenFOAM is a large part of their toolbox, and the benefits they get from having a large and participatory userbase (and, specifically, a userbase of potential customers) pay for the costs of distributing it.

So what’s my takeaway message from Russ’s post, then? If the value that I get from distributing this software is the feedback that I get from users, then I need to put make it very clear and obvious how they can do that, and effortless for them to do so – because that connection is how I get “paid”. Right now, all I’ve got is a non-obvious and munged email link at the bottom of the page, and that’s not good enough.