Open Source Musings 
WPI Presentation
In early April I gave a talk at the WPI Computer Science Colloquium.You can find a copy of the presentation here. In the coming weeks I'll discuss a number of topics from that presentation on this blog.
I'd like to thank everyone who attended my presentation yesterday, and for the interesting discussions that followed. As promised, here is a link to the presentation.
Images of Open Source
I'm reading Images of Projects by Mark Winter and Tony Szczepanek. The premise of this book is that projects can be viewed from a number of different perspectives, and the perspective we choose (at a given moment) will influence how we think about the project and other important factors, such as how we make decisions, what is viewed as important, and so on. This book follows the tradition of Gareth Morgan's influential Images of Organization, and certainly brings a much needed higher-level view of the complexities inherent in projects.These books made me think about images of open source. Off the top of my head I came up with the following:Ideology: The part that emphasizes "free as in freedom," copyleft, and so on.Life style: Related to the Ideology image, but more concentrating on Do It Yourself (DIY) and similar movements.Business Model: Seeing open source as a way of running a business.Marketing Gimmick: Related to the Business Model image, but seeing open source as a marketing tactic to increase visibility, etc.Social Movement: Emphasizes the community aspects of open source, especially around virtual, highly-distributed, ad hoc community dynamics.Mean of Production: Sees open source as one way of producing and managing goods (or knowledge). This is the image where most of my research centers.This isn't meant to say that these images are exclusive: clearly some images overlap, but I hope that each highlights an important aspect and enables the dynamics that are discussed in the two Images books mentioned above. Also, this list likely needs refinement, so any suggestions would be appreciated. :)
The business of open source, again
Preparing for EclipseCon, I had a chance to browse my past presentations. At EclipseCon 2007 I presented The Garden & the Market: The Value of Eclipse Open Source. This is probably my favorite presentation, and I think there still is a lot to say.When I gave that presentation in 2007, I was working at a company where open source was a minor means of software creation. Since then I joined Red Hat and now open source is the means of creating software. The whole question of building a business based on open source suddenly becomes more pressing. :)One aspect I really liked about that presentation was the audience participation. We genuinely had a conversation, one which in several cases lasted longer than the actual presentation itself. I'd like to loop back with the Eclipse community this year, and see how things look three years on. When I arrive at EclipseCon, I'll seek to secure a BoF slot for this purpose, and I look forward to the conversation. If anyone would like to meet separately, please drop me a comment here (or to the email in my profile), and we'll try to meet.
On purpose, an emergency
The frantic pounding on your door jolts you awake, and noticing that it is three in the morning doesn't calm your nerves. You peek out the window, not really knowing what to expect, but any speculation is immediately cut off by the fire you see climbing up the side of your house. Before you can react, a man appears (the one who was pounding on the door, you guess: funny how the mind can still reason so quickly under such stress), takes your garden hose and begins to spray water onto the fire. By the time you stumble out the front door, the fire is almost out. Panting, you stumble up to him and stare at the scorched wall. “I was on my way home from work,” he says, “and I saw the fire. Lucky I put it out before it spread.” You thank him for helping, and offer a reward. He turns down the offer, saying he was just glad to help. Of course you would be grateful to someone who saved your home from a fire. But how would you feel if you later learned that he had also set the fire? What kind of lunatic would do that? Certainly you would not feel like offering a reward under those circumstances. Yet we routinely praise and reward people who do the same type of thing in software business. A sales person runs into trouble with a customer, waits until the customer is really upset (due to inaction), and then “helps” to solve the problem by short-circuiting company procedures to “do the right thing for the customer.” A software team waits until late in the project to announce that certain serious problems – which they had known about for a long while – are going to cause a schedule slip. The team then makes a heroic effort to finish, and is thanked afterward for “getting it done.” We are happy when a vendor fixes a problem in their software, after much complaining and arguing from our side, even though all the vendor did was fix a problem it created. Whether through ignorance, incompetence, lack of planning or gamesmanship (it is easier to force an issue with a preferred solution if time to consider the problem is artificially cut), we are are grateful that the problem was solved and that the team “pulled together” at the crucial point. We breath a sigh of relief and move on, waiting for the next fire. But we fail badly each and every time we move on with first looking back. We fail to ask the key question: How did the emergency arise in the first place? In the heat of the emergency, such questions are usually cut off with well-intentioned statements like “this is not the time to point fingers” (many times by those who created the emergency, for obvious reasons). Although that is probably right (during the emergency you need to react to address the challenge), it does not mean that you shouldn't ask in retrospect. You have lost a chance to learn and avoid future emergencies for the same reasons, and you have set the stage to reinforce the bad behavior that relies on such emergencies to get things done.
Spoon, fork or knife?
Administrative note: thanks for all the comments on my previous post! I haven't found a solution, but have moved on to the “latest and greatest,” which has circumvented the issue. For those interested in the “question overhead” discussion take a look at BZ241335, comments 7 and 8. OK, so I'll try one of these metaphorical postings. Be kind and work with me here, I'm no Ed Merks for writing these, after all! How do you consume your open source? Spoon, fork or knife? It's nice if you can use a spoon; take it as is, slurp it up whole. Even nicer if someone spoon feeds you. What if you don't exactly like the taste? Maybe add some seasoning, or prevail on the chef to alter the recipe. Perhaps suggest variations to the chef, or even help the chef cook (assuming the chef wants you mucking around in the kitchen the insurance policy allows it). You could also consume it with a fork. This gives you more control to pick and choose the next bite, but also brings the substantial responsibility of not sticking your fork into more than you can swallow. Poor use of the fork is a primary cause of choking, and it's not pretty to watch or hear. Of course, if you are skillful with the fork and can take just the right amount, you get to eat as you wish. Then there's the knife. You're given a meal, but clearly can't consume it whole (unless you're a creature with one of those magical expanding jaws and a big enough mouth). Also, there might be parts of the meal you don't like, and want to avoid through judicious use of the knife. So you carve the meal and take it as you want, when you want. It's extra work – wouldn't it be nice if the chef carved it for you, so you could just use the spoon – but worthwhile to get the bites you want. But probably you're like most people: you use the spoon, fork and knife in some combination that seems to make sense for the meal. The spoon is fastest, and the others require more work. Sometimes you might carve with the knife and use it in a stab to consume the separated piece (which is dangerous, by the way. I hear that people long ago were more skillful with the knife and actually ate like this). Or maybe you are like me: you tend to use the fork to both carve and stab (a habit, which I am told, betrays my lack of a proper upbringing). In the past I've mostly consumed Eclipse meals using a spoon, with (rare) occasions for the knife to make bites suitable for the spoon. Over time, I find myself using the knife more and more, and the fork seems to have a renewed purpose. How about you?
Walking away from bugs
One of the most frustrating things that can happen to an open source committer is to meet someone who tried your software, ran into some problem, and just walked away. Did you check the error log? Did you ask on the newsgroup/mailing list/forum? Did you file a bug? As a committer, these questions roll off your tongue. After all, open source is a collaborative effort, and success depends in large part to the collective effort of the community. Walking away like that is just not good citizenship! Or is it? Recently I needed to evaluate some plug-in code that was written on a Ganymede-based target platform. For reasons not important here, my development environment is based on Europa at the moment. Eclipse has great support for these cases, however, and setting up various target platforms is routine for me. So, I go and set up the necessary target platform, load the evaluation code into my workspace, and set up a launch configuration. Right – so let's run these plug-ins and see how they work. The splash screen appears and shortly after – bang! It dies with an error* Strange. I've been developing open and closed source plug-ins for Eclipse since V1, and have dealt with all sorts of platform configurations and error conditions: never saw this one before... Mess around with the launch configuration, read config.ini (like the error message suggests); nothing helps. Ok – off to an internet and Bugzilla search for the issue: clearly I must be doing something stupid, because Europa should be able to launch Ganymede-based target platforms (right?). Hmmm... some discussion about this type of error, but nothing seems to fit my case. Try launching the target platform directly, and it works fine. A few more minutes of searching, a few more minutes of messing around with settings. Finally, I walked away. And then I thought about it. Why did I walk away, and what does that tell me about open source users who do the same? Well, in my case, and I suspect this is common, the reason was pretty simple: my task at hand was to evaluate the plug-in code, not to debug an Eclipse launch failure. I was trying to use Eclipse to work on another problem, not on Eclipse itself. I could get the information I needed in another fashion, and that's what I did. Now, being an open source committer, I felt a little guilty: surely I'm not walking away, but rather I'll get back to that problem. I'll do The Right Thing and check all possible avenues: I'll read everything I can about it on the net, I'll ask on the newsgroup, I'll file a bug and follow up. Yes, yes I will! I will get back to it, right after I get these other things (that I'm paid for) done... Then again, maybe not (to be realistic). I even felt guilty about writing this blog entry. Who I am to complain when I haven't done my homework? But then I realized that something larger was interesting: my attitude toward the whole situation. Maybe a kind reader will point out an answer for my problem, one which will probably seem entirely obvious, if I had looked in the right place. If so, thanks! But fixing this specific issue is really not the point. Maybe you'll flame me: You don't understand how open source works, you've got to engage the community! To this I'll answer: You don't understand your users! How many of them have walked away silently? When I was in graduate school, a friend of mine was doing his Ph.D. In the physics department at MIT. Maybe it's a measure of what it takes to operate at that elite level, but he once told me (without pride or arrogance, but simply matter of fact) that he was never shy about asking a question in class, because if he didn't understand something, it was the professor's fault for not being clear. Just to be clear (because I don't mind being flamed, but prefer to be flamed for what I actually meant): I'm not blaming the Eclipse platform or any other Eclipse project/committer for the error I encountered or my inability to fix it quickly. Rather I am just pointing out the observation that I feel compelled to do a lot of work before raising an issue, and I wonder if this “barrier to the question” is the reason why people walk away. If so, what can be done? *java.lang.IllegalStateException: Unable to acquire application service. Ensure that the org.eclipse.core.runtime bundle is resolved and started (see config.ini).
Returning to the planet
It's been a while since I last blogged. In blog years, that's like a lifetime. I guess it's because I've been reincarnated in a new job and in a new role. It's taken me a while to get settled in my new environment and to figure out how, if at all, I should continue to blog.My new position involves creating Eclipse tools for JBoss ESB, and there's no shortage of things to do. :) And, yes, it's all open source. I'll continue to blog about the same sorts of things that previously appeared in this blog, but also include more about SOA and plug-in development in Eclipse.For now, I'll just make a general observation, which has been guiding my use, selection and development of tools recently: the Linux Hater says: "The best code in the world won't change the world if people can't use it."
Toxic Requirements
Regardless of the process (or lack thereof) you use for creating software, there has to be some idea of requirements. These requirements tell us, in whatever form they are expressed, what it is that we’re trying to do. Most people who have worked on software projects for a while have faced impossible/highly unlikely requirements foisted on the project by people who do not understand technology and are prone to wishful thinking. (Simply saying that something will happen doesn’t make it occur.) Clearly unreasonable requirements based on wishful thinking are not good. But that’s not what I want to talk about here. Rather, I’d like to concentrate on requirements which are actually reasonable in some local context, but in the overall project context are just as bad, if not worse, than wishful requirements. These are toxic requirements. Fifteen years ago Gerald Weinberg discussed how considerations around software quality are ultimately political: someone has to make a decision about what “quality” means in a certain case, and doing so can result in more alignment to one person/group’s interests over another. Implicit in Weinberg’s work is that requirements are political as well: statements about software quality are expressed as requirements, and by extension all decisions about what to do (and hence what not to do) are made by designated people/groups balancing options. How does this apply to open source, and how is it an antipattern? As I discussed in an earlier post, tight binding between open source and product work can cause problems. Toxic requirements emerge in this context when product requirements are foisted on the open source community. Of course many times product requirements are also community requirements, so there is no imposition. But there are also times when the decision makers (committers) use their position to impose requirements on the community that have not been demonstrated as having broad applicability beyond specific product needs. In some situations, the case for broad applicability (crucially usually including some modifications of the requirement) can be made, but it is more expedient to use committer authority to push the process. In such cases, committers will often use the “I do, therefore I decide” principle of open source to justify the decision. But these rationalizations avoid the real issue: if your idea of open source includes principles like “openness,” “transparency,” and “community” then making decisions in the narrow scope of product requirements is poison. Instead you must be willing to either build community consensus (or at least the lack of objections), or simply not implement the requirement in open source. This is very hard to do, especially when your sponsors (who “pay the bills”) are demanding fast action in a direction that was decided by a closed process.
Open Source Antipatterns: How to hurt yourself with open source (2)
OK, so it's been a while since the first post in this series... Maybe there's an open source antipattern lurking? :-)The second antipattern I'd like to describe deals with the costs of developing in open source. Since open source software typically can be redistributed freely (within the terms of use in particular cases, YMMV), somehow otherwise savvy tech and business people jump to the conclusion that open source is free. Free to develop; a sort of perpetual motion machine for creating software! Money for nothing!But, of course, it does cost something to create open source software, even if the end result is made freely available. That's probably obvious (once the hype fog mentioned above clears). But, here's a secret: developing open source software makes you incur costs that you would have otherwise avoided if you decided to create the software some other way (say, for example, the standard closed-source model). That's not to say closed-source development doesn't have costs (potentially) avoided by open source-development, but rather the costs associated with open and closed-source development not always the same. To further complicate the picture, many open source initiatives are taken by employees on behalf of their companies, and so -- if you're not careful -- you can end up incurring both open source and closed source development costs for the same component! (Though, unless you try really hard, you probably will not incur all the possible costs associated with each.)So now the antipattern: Not understanding the cost structure discussed above before embarking on open source development. Next, you (or worse, your boss!) discovers the open source-specific costs, and things get unpleasant. One way of helping the situation (aside from understanding the cost structure in advance) is to isolate the additional, open source specific costs and seek to share these specific costs with the community.
Open Source Antipatterns: How to hurt yourself with open source (1)
One of the benefits of working on an Eclipse project is that you get to see firsthand how open source is produced and consumed. In a series of "Antipattern" posts, I will talk about some things you don't want to do with open source: sort of like a set of "worst practices."For the first installment, let's talk about how to consume open source components in a commercial product. The lack of a viable consumption process is one of the biggest problems I've seen, and is most severe when the company consuming the components is also working on that open source project. That last bit is counter-intuitive: if your company is working on open source components that you also consume, shouldn't it be easier to manage the quality, features, timing, etc.? In some circumstances, yes, but crucially it all depends on your consumption process. This process has an impact on how you participate in the open source project and how attractive it will be for others to join your efforts.The heart of the problem is binding commercial products too tightly to open source components. When you do so, you expect that the open source components will work in a way that you alone have determined necessary for your product, and that they will evolve in a controlled path aligned with product aims. Yet sharing in open source goes beyond simply making source code available. It goes beyond the other associated transparency areas (bug lists, development mailing lists, etc.). That is, success open source is about sharing control of the components as well. If you really want an engaged community to rally around your open source efforts, you have to share control, otherwise you limit how much people can actually participate, in ways meaningful them, in your project.Now, here's the rub: when you share control, sometimes the open source components will move in a direction not aligned with your product goals. If your products are very tightly bound to the open source components, then you'll either have to accept the loss of control or stop using the components to regain full control.With a better consumption plan, however, you can enable the best of both sides: control over your product while allowing the open source to move with the community. What you need is a process that is flexible and that can adjust the open/closed source mixture as necessary. You should also consider the wisdom of crowds: if the community is moving the open source components in a certain direction, perhaps that's better than what you had previously considered for your product? Perhaps you're getting hyper-early beta feedback at a time when you can actually react to it?Whatever you do, don't stifle an open source project with commercial product goals and then wonder why a strong community hasn't developed around either.
Code wants to be miscellaneous
I’ve been reading the fascinating Everything is Miscellaneous by David Weinberger, and have a few preliminary ideas about how this relates to open source. The final two sentences of his prologue: “…information doesn’t just want to be free. It wants to be miscellaneous.” I’d like draw out an implication of this statement:Code doesn’t just want to be free. It wants to be miscellaneous.So, what’s that supposed to mean? I’m only going to make a few suggestions here, as I best understand it at the moment, and leave the implications to more study and discussion of Weinberger’s book.Open source code is free, as we all know, like in “free speech” not “free beer." Well, it is free in this sense modulo the open source licenses morass, but that’s another post… But it is not "miscellaneous" in the sense that Weinberger uses it. For him, an important characteristic of information being miscellaneous is the ability to recombine it in ways desired by the consumer of that information. Information is allowed be an unstructured, unordered, teaming mass with rich associations. Consumers of this information take slices configured according to their wishes, and in the process both create additional information and associations.I wonder if open source can get to this miscellaneous state? A state in which components, source files, and even parts of source code itself can be recombined as desired by consumers, and in which rich meta-data enables this usage. An example from Eclipse: there are feature definitions which bundle plug-ins into groups based on usage patterns or functional areas. While the feature mechanism is a great tool, anyone who has provided Eclipse plug-ins to enough adopters can tell you that no set of feature definitions will ever satisfy everyone. There’s just too many ways to slice and dice plug-in sets, even in smaller projects, and by the time you get to the size of most Eclipse Foundation top-level projects, the possibilities are huge. In practice most adopters just take the best approximation of their needs based on existing features, though some apparently define their own custom feature definitions. In this system, the plug-in set is structured into features at the wrong end (following Weinberger’s theory): the providers are imposing what they feel to be the appropriate feature set on adopters, whereas it should be the adopters who are defining features as desired. And I don’t mean the adopters suggest a new feature set to the Eclipse project: I mean that the adopters dynamically create their feature sets based on their needs as the moment.Another example: what if there were an open source project where anyone could modify the code? I don’t mean “submit a patch” or “modify their local copy,” but rather an open source project where anyone could commit changes. You say that this would quickly become chaos? The code would never run? Security bugs could be injected on purpose? How can the quality/intellectual property be managed? Yes, these are real concerns today. But, to me, this points to the need for a new set of tools and a new way of working with source code. One which lets me understand the lineage, modifications, and intent of various “layers” (versions) within the miscellaneous code base. One which lets me have the stability demanded by (especially commercial) adopters, with the free-flowing experimentation of an ideal(istic?) open source community.I can see two objections here: this whole “miscellaneous” business is a sham, and arguments that we can/are doing so already. Maybe the first objection is correct, but I’d reserve judgment until the ideas and implications of Weinberger’s book are better understood. Personally, I doubt this is a sham: on the contrary, it is pointing to something very important. For the second, I’d agree in theory, if not practice. In other words, in is possible, but not at all practical, to do these sorts of things with existing technology. But that’s not the point: the reason why we’re not still writing everything in assembly language, for example, is that higher-level languages bring abstractions and enable thinking in different ways. The same is true here: we need tools working at the right “level” so miscellaneous information can be harnessed on a routine basis, without requiring large effort, knowledge or time.
Do software commodities exist?
The word “commoditization” often comes up in reference to software [one example], especially open source software. The standard notion is that a particular class of software becomes commoditized when there are a number of equally good substitutes. If the software is offered free of charge, then cost strategies based purely on that software itself are not tenable. We tend to think about commoditization as a one way, final process: once a specific type of software is commoditized, it remains so. And, for obvious reasons, those who commoditized the software and those who built on it are happy to sustain this permanent state.Yet the word “commodization” obscures important perspectives. For example, at EclipseCon 2007, Brent Williams argued that the software market is not a commodity market. His presentation is excellent reading for this and the other points it makes, and has been widely noticed in the Eclipse community. How can we account for the widespread notion of software commoditization in light of Brent’s argument? In my opinion, there are two things going on here. First, the software market as a whole does not behave like a commodity market (as Brent says), but there are specific types of software that do, for certain classes of consumers/users. Simply put, there are certain software components that I would never be able to justify to management building another version of, simply because the existing (often open source) alternatives are good enough for our needs and we do not seek to differentiate based on that particular type of functionality. Secondly, I think the word “commodity” (and its variants) is used in at least two different ways: the technical, investment world usage that Brent cites, and the more broad sense of good enough, undifferentiated functionality. Maybe the second usage would be better phrased as something like “soft-commodity,” but it far too pervasive to change.Another aspect of commodization is the perceived permanence. In their book, The Innovator’s Solution, Christensen and Raynor argue that commodization is a cyclic process, one which includes de-commodization.* De-commoditization: the word is uncommon and probably strange to most of us. But their model is detailed, backed by thorough research, and very important for the software industry to understand. The key claim is that commodization is accompanied by de-commodization at some location in the Value Chain. Thus, it is vital to predict where de-commodization will occur and to prepare for it, which allows reaping profits from the new opportunity.The notion of de-commoditization might be unfamiliar, but it has appeared before in a number of cases. An early analysis using this term is found here, which (interestingly) is discussing the suggestion of a de-commodization tactic in the famous Halloween Memo. Unlike the specific lock-in protection tactic mentioned there, however, Christensen and Raynor’s model is much deeper and has many more implication for how we think about open source, especially business models/strategies associated with open source software.* The relevant chapter is also available as a HBS Press Chapter (#1542BC)
Purpose of this blog
This blog is a spin-off from my previous one concentrating on the Eclipse Data Tools Platform (DTP) project. In that blog I've mixed in discussion of open source in general with DTP posts. Based on my recent presentation at EclipseCon 2007 (The Garden & the Market), I feel it is time to separate these discussions, and hence this new blog.So, what sorts of things will we discuss here? As an initial estimate, here's some links to posts on the Data Platform blog covering the intended topic areas:The Eclipse MineEclipse is Smarter than meThe City Hall and the ForumThe Open Source HuntWhat motivates Eclipse?Fractured Eclipse?Thinking about open source
Adminstrative Notes
The postings on this site do not represent positions, strategies, opinions, etc. of Sybase, Inc., the Eclipse Foundation, or the Eclipse Data Tools Platform (DTP) project.