At first glance, commit messages seem to be a small, relatively innocuous thing for a developer. When you commit code, you type in some description about your code changes and then, typically, push your code somewhere for review by someone.

Regardless of whether the code you pushed is going to an open source project, an internal proprietary code repository, or just some code exchanged between friends working on a joint project, that simple little commit message tells the person reading your code a whole lot about you. It speaks volumes about the way you feel about the code you submit and the quality of the review you expect for your code.

As an example, suppose I was working on some code that fixed a bug. I got my code ready for initial review and I did the following:

$> git commit -a -m "Fixes some stuff"

And then I push my commit somewhere using git push …

Inevitably, what happens is that another developer will get some email or notification that I have pushed code up to some repository. It is likely that this notification will look something like this:

Change subject: Fixes some stuff
......................................................................

Fixes some stuff

Change-Id: I79bbac32b5c99742b5cb283c6e55e6204bf92adc
---
M path/to/some/changed/file
1 file changed, 1 insertion(+), 1 deletion(-)

And in the notification will be some link to a place to go do a code review.

Now, what do you think is the first thought that goes through the reviewer’s mind? My guess would be: Really? Fixes what stuff? By not including any context about what the patch is attempting to solve, you leave the reviewer with a bad taste in their mouth. And a bad taste in the reviewer’s mouth generally means one thing: a reluctance to review your patch.

OK, so what could we do to make the commit message better, to provide the reviewer with more initial context about your patch? Well, the first thing that comes to mind is to reference a specific bug that you are fixing with this patch.

Alright, so we amend our commit message to include a bug identifier:

$> git commit --amend -m "Fixes Bug 123456"

And subsequently push our amended commit message. The reviewer now gets a new notification that you’ve amended a previous patch. Now the notification includes the bug identifier. What do you think the next thought a typical reviewer might have? My guess is this: What, does this developer think that I’ve memorized all the bug IDs for all open bugs? How should I know what Bug 123456 is about? And here comes that bad taste in the mouth again.

OK, so this time, we will forgo the use of the time-saving -m switch to git commit and actually type a proper, multi-line commit message in our editor of choice that describes the bug that our patch fixes, including a brief description of how we fixed the bug:

git commit --amend  # This will open up your editor...

Now we’d enter a good commit message … something like this would work:

Fixes Bug 123456 - ImportError raised improperly in Storage Driver

Due to a circular dependency, an ImportError was improperly
being thrown when the storage driver was set to XYZ. Rearranged
code to remove circular dependency.

The commit message now will give the reviewer everything they need in the notification to understand what the patch is for and how you solved a bug, without needing to go to their bug tracker to figure out what the bug was about.

A detailed commit message shows you care about the time that reviewers spend on your patch and that you value the code you are submitting.

Anyway, I’ve uploaded a PDF of the presentation to this website and provided a link to the Google Docs presentation along with a brief overview of the topics covered in the slides below. As always, I love to get feedback on slides. Feel free to leave a comment here, email me or find me on IRC. Enjoy!

Google Presentation (HTML)
PDF slides


Topics included in the slides:

• OpenStack Contribution Process
• Running Devstack Locally
• Running Tempest against an Environment
• Walkthrough the Tempest Source Code
• Progressively improving a test case
• Common Scenarios in Code Review and Submission

I agree that OpenStack is one of the most compelling open source projects this year. I think back to the Cactus release earlier in the year and really am amazed at the amount of change in many of the core OpenStack projects — Nova in particular. We’ve had an explosion of contribution from a ton of individuals working at a who’s who list of technology companies, and I expect to see that passion and enthusiasm for OpenStack only continue to grow through 2012. Lots to look forward to.

Here’s to an innovative 2012 and continued collaboration in the open source community on this terrific set of cloud infrastructure projects.

There are quite a few folks interested in QA coming to the OpenStack
Essex Design Summit next week. I wanted to give you all a heads-up on
the sessions that may be of interest to you.

Here they are:

Monday, Oct 3rd:

09:30-10:25 – Essex Release Cycle

Thierry Carrez, our illustrious release manager, will do a post-mortem
on the Diablo release cycle and discuss potential changes for the
Essex release cycle. I know almost all QAers have expressed desires to
have maintenance branches managed by the QA team and I’ve heard
suggestions about various QA-centric freeze points. Those interested
in advocating for these things should plan to attend this session.

14:00-14:45 – Stable Release Updates

Dave Walker from Canonical plans to outline some possibilities for how
to maintain and update stable releases of OpenStack projects.

15:00-15:45 – Separating API from Implementation of the API

Total self-promotion of a session I’ve proposed… I think anyone
interested in stabilizing the OpenStack APIs and having OpenStack APIs
become the open standards for the cloud computing industry should
attend.

16:30-17:15 – OpenStack Compute API 2.0

Glen Campbell will be leading a discussion about how to improve the
Compute (Nova) API for a 2.0 API series. I think it’s important that a
number of folks on the QA team attend this session and get an idea of
the things that we will be looking at in the future regarding the
Compute API. Personally, I’m definitely planning on attending this
one.

17:30 – 17:55 – NetStack Continuous Integration Planning

Personally, I will not be at this session as I have another session to
lead. However, I think it is important that a number of people from
the QA team attend this session, listen to the needs of the NetStack
contributors, voice our support for their projects, explain what the
goals of our team are, and enable some cross-team collaborative
efforts around CI and QA.

Tuesday, Oct 4th:

09:30 – 09:55 – Documentation Strategies for OpenStack

Anne Gentle will be leading a discussion about documentation of
OpenStack projects. One of the deliverables of the OpenStack QA team
is clearly to identify areas where specifications don’t match
behaviour, so I think it’s pretty critical that the Doc Team and the
QA team be on the same page when it comes to how to coordinate
communication of documentation discrepancies.

09:30 – 09:55 – VM Disk Management in Nova

At the same time as the documentation session, Paul Voccio is leading
a discussion about VM disk management in Nova. Those QAers focusing on
disk/volume management may want to attend this session to ensure the
QA team has a good grasp of changes coming in this arena.

10:00 – 10-25 – OpenStack Common

Brian Lamar will be leading a discussion on getting serious about the
potential of an openstack-common Python library of common code shared
amongst many OpenStack projects. Hey, it’s a heck of a lot easier to
QA code that’s in one location than the same code, written with slight
differences, spread across many projects… seems like a no-brainer
for the QA team to attend and support this idea.

11:00 – 11:25 – Monitoring in Swift

John Dickinson will be leading a session to discuss what things should
be monitored across a Swift cluster, and what tools are available for
monitoring. I think this discussion will be valuable for those of us
interested in long-running production integration tests where Swift is
one of the components of a full OpenStack test cluster.

12:00 – 12:25 – Integration Test Suites and Gating Trunk

A no brainer… in this session we will talk about the various
integration test suites for Nova/Glance/Keystone and discuss the
effort already underway to combine them. In addition, we will talk
about what policies to recommend for OpenStack projects regarding what
level of passing integration tests should hold up a gated trunk.

15:30 – 15:55 – Making VM State Handling More Robust

Phil Day is leading a discussion about ways in which the handling of
VM state transitions can be inconsistent and confusing. Since the QA
team is responsible for documenting just such inconsistencies and
building tests cases for such inconsistent behaviour, I think this
session would be good to hang around in and listen/take notes.

16:30 – 17:25 – OpenStack Faithful Implementation Test Suites (FITS)

Josh McKenty will be talking about certain proposals regarding a FITS
for OpenStack APIs. Should be an interesting session

Wednesday, Oct 5th:

09:30 – 10:25 – XenServer/KVM Feature Parity Plan

This session should be good for those QAers interested in identifying
areas where feature parity between hypervisors is lacking, and
discussing ways in which the QA team can document these disparities
and produce tests for identifying future disparity among hypervisors.

11:00 – 11:45 – Glance Throughput Improvements

This session is being led by Tim Reddins, from HP, who (along with his
team) have done some analysis on ways to improve Glance’s throughput.
QAers interested in stress, capacity, and parallelism testing should
definitely attend!

11:30 – 11:55 – Nova Upgrades

Ray Hookway will talk about ways that Nova’s update process can be
made more robust. I imagine that the talk’s recommendations will be
generally applicable to many OpenStack projects, not just Nova. I also
think that some members of the QA team should attend — we should be
able to create functional tests for upgrade processes for all
OpenStack projects…

14:30 – 14:55 – Git/Gerrit Best Practices

Monty Taylor is leading this session on Gerrit/Git best practices. I
recommend everyone go, if only to see the fireworks.

15:30 – 15:55 – Quality Assurance in OpenStack

Uhm, duh, you should all be at this one. We’ll discuss how to
divide the voluminous amount of work among our members, talk about
which projects (and components within certain projects) are
high-priority items, the ways we should communicate and track
progress, etc

17:00 – 17:25 – Internal Service Communication

Brian Waldon is leading a session on internal service communication
that should be quite interesting. The integration testing coverage of
major internal service components of Nova is currently light, and is
one of those areas I think should be carefully picked over by our QA
team.

OK, that’s the recommendations from me, but of course, feel free to
attend whatever sessions are of most interest to you. I’m very much
looking forward to meeting all of you (we’re up to 28 members as of
this writing).

Cheers, and see you tomorrow!
-jay

I tried commenting on Maureen’s article on their website, but the login system is apparently borked, at least for registered users who use OpenID, which it wants to still have a separate user ID and login. Note to sys-con.com: OpenID is designed so that users don’t have to remember yet another login for your website.

Besides having little patience for content-sparse websites that simply provide an online haven for dozens of Flash advertisements per web page, the article had some serious problems with it, not the least of which was using large chunks of my Happiness is a Warm Cloud article without citation. Very professional.

OK, to start with, let’s take this quote from the article:

Drizzle runs the risk of not being as stable as MySQL, because the Drizzle team is taking things out and putting other stuff in. Of course it may be successful in trying to create a product that’s more stable than MySQL. But creating a stable DBMS engine is something that has always taken years and years.

This is just about the most naïve explanation for whether a product will or will not be stable that I’ve ever read. If Maureen had bothered to email or call any one of the core Drizzle developers, they’d have been happy to tell her what is and is not stable about Drizzle, and why. Drizzle has not changed the underlying storage engines, so the InnoDB storage engine in Drizzle is the same plugin as available in MySQL (version 1.0.6).

The pieces of MySQL which were removed from Drizzle happen to be the parts of MySQL which have had the most stability issues — namely the additional features added to MySQL 5.0: stored procedures, views, triggers, stored functions, the INFORMATION_SCHEMA implementation, and server-side cursors and prepared statements. In addition to these removed features of MySQL, Drizzle also has no built-in Query Cache, does not support anything other than UTF-8 character sets, and has removed the MySQL replication system and binary logging — moving a rewrite of these pieces out into the plugin ecosystem.

The pieces that were added to Drizzle have mostly been done by adding plugins that provide functionality. Maureen, the reason this was done was precisely to allow for greater stability of the kernel by segregating new features and functionality into the plugin ecosystem, where they can be properly versioned and quarantined, therefore increasing kernel stability. It’s pretty much the biggest principle of Drizzle’s design…

The core developers of Drizzle (and much of the Drizzle community) would also have been happy to tell Maureen how the Drizzle team defines “stability”: when the community says Drizzle is stable — simple as that.

OK, so the next thing I took objection to is the following line:

Half of Rackspace’s customers are on MySQL so there’ll be some donkey-style nosing to get them to migrate.

I think my Rackspace colleagues might have quite a bit to say about the above. I haven’t seen any Rackers talking about mass migration from MySQL to Drizzle. As far as I have seen, the plan is to provide Drizzle as an additional service to Rackspace customers.

Rackspace evidently wants its new boys, who were not the core pillars of the MySQL engineering team, to hitch MySQL, er, Drizzle to Cassandra

MySQL != Drizzle. Implying that the two are equal do a disservice to both, as they have very different target markets and developer audiences.

The smart money is betting that even if a good number of high-volume web sites go down this route, an even higher number such as Facebook and Google will continue with relational databases, primarily MySQL.

Again, probably best to do your homework on this one, too. Facebook runs an amalgamation of a custom MySQL version and storage engines, distributed key-value stores, and Memcached servers. I would think that Facebook moving to Drizzle would be one tough migration. Thousands (tens of thousands?) of MySQL servers all running custom software and integrated into their caching layers is a huge barrier to entry, and not one I would expect a large site like Facebook to casually undertake. But, the same could be said about a move to SQL Server or Oracle, for that matter, and has little to do with Drizzle.

Google is moving away from using MySQL entirely. Mark Callaghan, previously at Google, has moved over to Facebook (possibly because of this trend at Google to get rid of MySQL), and Anthony Curtis, formerly of MySQL, then Google, left Google partially because of this reason.

OK, so the next quote got me really fired up because it demonstrates a complete lack of understanding (maybe not Maureen’s, but the unnamed source it’s from at least):

Somebody – sorry we forget who exactly – claimed that as GPL 2 code Drizzle “severely limits revenue opportunities. For Rackspace, the opportunity to have some key Drizzle developers on its payrolls basically comes down to a promotional benefit, trying to position Rackspace as particularly Drizzle-savvy in the eyes of the community and currying favor for its seemingly generous contributions. What’s unclear is whether they may develop some Drizzle-related functionality that they will then not release as open source and just rent out to Rackspace hosting customers…that would be a way for them to differentiate themselves from competitors and GPLv2 would in principle allow this.”

A few points to make about the above quote.

First, name your source. I find it difficult to believe that the most-read technology writer would not write down a source. Is it the same person you deliberately left out of a quote from my Happiness article? (why did you do that, btw?).

Second, the MySQL server source code is licensed under the GPL 2, and so is Drizzle’s kernel, because it is a derivative work of the MySQL server.

Let me be clear: Developers who contribute code to Drizzle do so under the GPLv2 if that contribution is in the Drizzle kernel. If the code contribution is a plugin, the contributor is free to pick whatever license they choose.

Third, licensing has little if anything to do with revenue at all. The license is besides the point. There are two things which dictate the company’s revenue derivation from software:

1. Copyright ownership
2. Principles of the Company

Drizzle, Rackspace, or any company a Drizzle contributor works for, does not have the copyright ownership of the MySQL source code, from which Drizzle’s kernel is derived. Oracle does. Therefore, companies do not have any right to re-sell Drizzle (under any license) without explicit permission from Oracle. Period. Has nothing to do with the GPLv2.

That said, contributors do have the right to make money on plugins built for the Drizzle server, and Rackspace, while not having expressed any interest to yours truly in doing so, has the right like any other Drizzle contributor, to make money on plugins its contributors create for Drizzle.

It is my knowledge (after actually having talked to Rackspace managers and decision makers), that Rackspace is not interested in getting into the business of selling commercial Drizzle plugins. Their core direction is to create value for their customers, and I fail to see how getting into the commercial software sales business meets that goal.

Next time, please feel free to contact myself or any other Drizzle contributor to get the low-down on Drizzle-related stuff. We’ll be nice. I promise.

One of the questions on the survey was this:

How would you describe your development process?

followed by these answer choices:

• Individuals request features
• Large/small group empowered to make decisions
• Benevolent dictator
• Other, please specify:____________

I thought a bit about the question and then answered the following in the “Other, please specify:” area:

Bit of a mix between all three above.

The more I think about it, the more I really do feel that Drizzle’s development process is indeed a mixture of individuals, groups, and a Benevolent dictator. And I think it works pretty well. Here’s some of the reasons why I believe our development process is effective in enabling contributions by being a mix of the above three styles.

Who’s the Benevolent Dictator of Drizzle?

First, let me get the BDFL question out of the way. We’ve made a big deal in the Drizzle community and mailing lists that anyone and everyone is encouraged to participate in the development process — so why would I say that Drizzle has a benevolent dictator?

Well, although he would probably disagree with the tile of BDFL, Brian Aker does have some dictator-like abilities with regards to the development process, and rightfully so. Brian came up with many of the concepts that Drizzle aspires to be, and Brian has more experience working on the code base than any other contributor.

After having worked closely with Brian now for 18 months or so, I can definitively say that Brian’s brain works in a very, well, interesting way. Those of us who work with him understand that sometimes his brain works so fast, his typing fingers struggle to keep up, resulting in something I call “Krowspeak”. It’s kinda funny sometimes trying to translate

With this wonderfully unique noodle, Brian tends to knock out large chunks of code at a time, and often he wants to push these chunks of code into our build and regression system and into trunk to see the results of his work quickly. Sometimes, this can cause other branches to get out of sync and get merge conflicts, and Brian will inform branch owners of the conflicts and work with them to resolve them.

So, regarding dictator-like development processes, I suppose we have Brian acting as the merge dictator because he’s got a lot of experience and understands best how both his code and other’s code integrates. We tried a little while back having myself and Monty Taylor be merge captains, but that distribution of merge work actually created a number of other problems and we’ve since gone back to Brian being the merge captain by himself, with Lee, Monty, and myself improving our automated build and regression system to help Brian with the repetitive work.

That said, what Brian does not do is make decisions in a dictator-like way. Decisions about the code style, reviews, features, syntax changes, etc are made on the mailing list by consensus vote. If a consensus is not reached, generally, no change is made which would depend on the decision. Brian does not influence the direction of the software or the source code style any more than anyone else on the mailing list which expresses an opinion about an issue; and for this, I greatly respect his wisdom to seek consensus in an open and community-oriented way.

Groups Empowered to Make Decisions

I’m assuming that what Selena’s “large/small group empowered to make decisions” answer meant was what is sometimes called “Cabal Leadership” of a project. In other words, there is some group which steers the project and makes decisions about the project which affect the rest of the project’s contributors.

Drizzle has at least one such group, the Sun Microsystems Drizzle Team, which is composed of Brian, Monty Taylor, Lee Bieber, Eric Day, Stewart Smith, and myself. One might call us the core committers for Drizzle.

However, while the Sun Drizzle team certainly is empowered to guide development, it is no different than any other group of developers that choose to contribute to Drizzle. There isn’t a “what the Sun Drizzle team decides” rule in effect. Our “power” in the development process is no greater or less than any other group of contributors. We act merely as a team of individuals who work on the Drizzle code and advocate for the project’s goals.

Individuals Empowered to Make Decisions

One thing I’ve been impressed with in the past 18 months is how the Drizzle community has embraced the opinions and work of individual contributors. I believe Toru Maesaka, Andrew Hutchings, Diego Medina and Padraig O’Sullivan were among the first individuals to begin actively contributing to Drizzle. Since then, dozens of others have joined the developer and advocate community, with each individual carving out a piece of the source code or community activities that they want to work on.

I have learned much from all these individuals over the last year or so, and I’ve tried my best to share knowledge and encourage others to do the same. Our IRC channel and mailing list are active places of discussion. Our code reviews are always completely open to the public for comments and discussed transparently on Launchpad, and this code review process has been a great mixing bowl of opinion, discussion, learning and debate. I love it.

More and more we have developers showing up and taking ownership of a bug, a blueprint, or just a part of the code that interests them. And nobody stands in their way and says “Oh, no, you shouldn’t work on that because <insert another contributor’s name> owns that code.” Instead, what you will more likely see on the lists or on IRC is a response like “hey, that’s awesome! be sure to chat with <insert another contributor’s name>. They are interested in that code, too, and you should share ideas!” This is incredibly refreshing to see.

In short, the Drizzle developer process is a nice mix of empowered individuals and groups, and a dash of dictatorship just to keep things moving efficiently. It’s open, transparent, and fun to work on Drizzle. Come join us

About 20 months ago, I was able to purchase a Lenovo Thinkpad T61 from the lenovo.com website without an operating system installed. Today, I went to purchase a new Lenovo Thinkpad laptop, again without having to pay the Windows Tax. Turns out Lenovo has stopped offering this option. What a complete PILE OF SHIT. Somebody in Microsoft’s “Business Development” or “Partners” team must have told Lenovo to stop offering its customers a simple choice of not having to pay the OEM license fees for Windows. And there’s nothing anyone can do about it. Microsoft is just too big and too pervasive for anybody to have a damn effect on them.

Frankly, it’s anti-choice, anti-competition, anti-innovation behaviour from Microsoft.

And its ridiculous.

Does anyone out there know how to get a decent laptop any more without having to fork over my money to a software giant that continues to bully all competition out of the market? Your suggestions are most welcome.

P.S. Mac is not an option for me. Sorry.

P.P.S The only thing this post has to do with MySQL is the general discussion on the acquisition of Sun by Oracle, and the pending investigation into possibly monopoly concerns by the EC…but of course I can’t comment on that directly…grr.

UPDATE:

Seems DELL offers laptops with Ubuntu installed instead of Windows, at least according to search results from their website. Yeah! \o/ Of course, now I have to just figure out how to get to that customization option. When I’ve gone through the customization screens, no option other than Windows is available.

UPDATE 2:

The DELL representative on their online chat program was quite helpful and offered this link to laptops they offer with no Windows Tax.

I was reviewing some code today, and a thought crossed my mind that sparked my landlord brain. I was staring at the copyright and license header in a Drizzle plugin and it struck me…

If ever there was a steady, rock-solid tenant, I would guess that it would be the occupant of this address:

51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

Now, I don’t know if the Free Software Foundation owns the address above, or if they rent. But, it occurred to me that if they do rent the fifth floor of that building, that the owner of that building must have one of the most trustworthy and reliable tenants ever.

I mean, we all know how much of a pain in the ass it is to move households. You’ve got to notify everyone about the new address, find friends who will help move furniture for the price of a six-pack, etc. But think of the giant problem the FSF would have if it ever decided to move. Think of the tens (hundreds?) of thousands of source code files which would suddenly have an erroneous address. I wonder if the owner of that Franklin Street building has thought of this, and has smiled, knowing just how much of a pain changing addresses would be for the FSF. Hmmm, food for thought.

One of the biggest “aha” moments I have had in the past three years is the following:

It is the role of a community manager to remove the barriers — both technical and ideological — between the user/developer community and the company or group of individuals which produces the open source software

Some barriers are small. Sometimes these barriers can be overcome by a simple email to an annoyed community member who has misunderstood a poorly communicated company objective.

And, sometimes barriers are large, and require a concerted effort of many people to overcome. One of these large barriers — the usage of the closed-source BitKeeper source control system — was recently removed and is a shining example of what can happen when many advocates within a company come together for the benefit of the community as a whole.

Yesterday, Kaj Arnö announced that MySQL has officially moved away from BitKeeper and has shifted development to use the free and open source Bazaar revision control system. This is an immensely important move for the MySQL community and the MySQL engineering team and its importance cannot be understated. While BitKeeper is technically a fast and feature-full revision control system, there was a significant barrier to the community that it enforced: the free BK client was essentially a read-only client, without even basic abilities to do diffs on a source code branch. This inhibited contributions from the external community by pipelining contributions into closed BK branches with no commit access for external contributors.

With the move to MySQL hosting its source code on the Launchpad.net service in Bazaar trees, we have removed this barrier to open source development. On Launchpad, there is now an overarching mysql super-project which contains all MySQL-related projects. Under this super-project, there are two projects which contain branches of the MySQL Server. One project, “mysql-server” contains the official MySQL server code branches. The other, “sakila-server“, contains branches of the server codebase that are being worked on by external community members.

Why did we set up two separate projects, one called “mysql-server” and one called “sakila-server“? There are a number of reasons.

Showcase Community Work

First, and perhaps most importantly, the “mysql-server” project contains all official branches and MySQL engineering team trees. Consequently, the code area of Launchpad for this project, is beginning to fill up with lots of different branches, and it is a little difficult to differentiate which branches are written by community members. We wanted a separate area that community members could showcase their work, and not have their work “swamped” by all the different team trees.

Use of Launchpad’s Bugs and Blueprints Services

Secondly, one of the issues we ran into was how do community-written projects track roadmap tasks or bugs in their distribution? MySQL has its own Bug Tracking system and its own Worklog system which it uses to track “roadmap” type tasks. Unfortunately, there is no way for community-developed projects to use these systems, as they are heavily customized for MySQL’s internal procedures. Furthermore, Launchpad.net already offers services for tracking bugs and assigning roadmap tasks to projects. So, with a separate sub-project (sakila-server), community developers now have a built-in bug and blueprint (roadmap) services ready to use for their code branches.

Full Community Control of the Project

Finally, within the “mysql-server” sub-project, community members wouldn’t have the rights to “brand” their projects the way they wanted to. As MySQL is a trademarked and copyrighted product, community members would be restricted to how they could administer and brand their own branches. Having their branches under a separate sakila-server sub-project, those restrictions go away and they have full administrator rights over everything, including branding…

I think this move is fantastic, and we welcome your input and feedback about ways we can continue to open up, be transparent, and work better with our community. We’re all ears. Let a hundred dolphins swim.

Please be sure to check out both server code project areas and investigate the various branches that community members are working on!

The initial interview led Amanda McPherson of the Linux Foundation to take issue, and long comments on those posts from supporters and objectors, shed light on a rift between OpenSolaris insiders, Linux community developers, and Sun’s overall marketing approach around open source.

Amid various snits about Sun’s “fluffy marketing” practices[1] towards OpenSolaris, there was a defection of a high-level OpenSolaris developer, Roy Fielding. The defection ostensibly was due to Roy’s frustration that Sun was not living up to its promises of a truly open development model and community, a frustration that Steven O’Grady finds uninspiring.

Unfortunately, these conversations are really just a sideshow of personal and anti-corporate rhetoric that miss the underlying truth about open source (and indeed closed source) products/projects: it’s not about the marketing, it’s about the product, silly.

I’m sure my colleagues of the marketing persuasion may take issue with that statement, so let me elaborate why I believe this to be true, and why I think that OpenSolaris has a chance of becoming a true competitor to Linux in the future — and that chance has absolutely nothing to do with Sun’s OpenSolaris marketing team (sorry, folks.).

It’s All About the Product (Quality)

As much as developer communities[2] like to think of themselves as the purveyors of a grand vision of truth towards the user community, all developer communities are dwarfed by their corresponding user communities. Members of a user community are unlikely to be aware of discussions within the developer community nor care about the debates within that community.

User communities use the product. It is the product which counts, and nothing more. And the user community will continue to use a product as long as other competing products are inferior in three basic ways:

• Ease of use
• Performance
• Stability/reliability

Before you say “whoah, doesn’t licensing count, too?!”. Sure, it does, but I group licensing in with “Ease of Use”. For some groups, a BSD-style license is “easier to use” because of a perceived viral effect of the GPL on their own code (especially for embedded products). For other groups, the GPL-style licenses are easier to use because they provide certain safeguards and benefits for their own products or environments.

So, my point is that it is not the marketing of a product that counts — indeed, false marketing statements or tactics can backfire quickly. What counts is the product itself. Marketing should be the function of promoting the innovative advantages of a product in the above three categories.

Likewise, developer communities should be about improving a product in these three ways. And, NO, “features” is not a differentiation that the user community (or customers) use in picking one competing product over another. Features that deliver better ease of use, faster performance, or more stability are what can drive differentiation. Features that do none of the above are a waste of space.

Why Open Development Models Improve Product Quality

An open development model and community is simply a method of attaining better product quality. Sure, happy developer communities translate to happy user communities — happy developers make better code and better code is enjoyed by users. But simply having a “community”, as the OpenSolaris marketing team would have you believe, does not magically make a good product. A community can take many shapes. It is my belief that a more open development community does make a better product. Why?

Increased Modularization of the Core Product

In my estimation, there are many reasons for open communities producing better products than closed communities. First and foremost amongst these reasons is the first evolutionary process which occurs when development of a product is opened up in any way: increased modularization and pluggability of the core product.

The vast majority of open source projects which have a truly open development community exhibit much higher architectural modularization than projects with closed development communities. Apache, Eclipse, Linux, PHP — these are all projects with open development communities and they each exhibit extremely high levels of modularity. Why? Because modularization enables development to continue/begin on a specific feature or part of the codebase without affecting or hindering the development of the core of the product. This enables all sorts of things from easier versioning and release practices to an increase in external development of needed features.

Contrast the architecture of these projects with the architecture of a closed development project such as MySQL. The architectural differences are sometimes stunning, especially in early versions of MySQL. When I wrote the System Internals chapter in Pro MySQL in 2005, I wrote:

MySQL’s architecture consists of a web of interrelated function sets, which work together to
fulfill the various needs of the database server. A number of authors have implied that these
function sets are indeed components, or entirely encapsulated packages; however, there is
little evidence in the source code that this is the case.

I wrote that chapter against the MySQL 5.0.2 code base. Admittedly, lots of improvements have made it into MySQL since those days, but much of the same interweaving of subsystems still exists, and complicates the code base substantially. I believe that if an open development model had been instituted at MySQL years ago, we would have a very different and much more modular core database kernel than we have today. Heck, the embedded library might even work correctly…

Modularization Leads Directly To Innovation

So, besides enabling cleaner release management and increased development of external features, why is a modular product better than one that isn’t? This all goes back to the “Big Three Basics”: Ease of use, performance, and stability. Let’s tackle the last one first.

Increased Stability

Why would modularity lead to an increase in stability? This can be answered quite easily. When a product’s architecture undergoes the refactoring needed to modularize, two things happen. First, a “core kernel” or runtime begins to take shape. This kernel takes shape because developers in an open development community want to be able to develop against a firm set of application programming interfaces (APIs). Programming against an API enables a developer of an extending feature to form a contract with other developers in the community: “I will write my feature to connect to other stuff in a predefined and contractual way”.

Secondly, this method &mdash programming against an API — allows the developer to develop her feature in isolation from ongoing development in other areas. Development in other areas can count on other developers’ code not interfering with their own code. In addition, work done on one feature or module can be released irrespective of the hangups or problems occurring in other parts of the code.

So why does this lead to greater product stability and reliability? Because of the need for APIs (so that other developers can code modules outside of the interference of other modules) a set of APIs begins to emerge that outline the “black box” of the core product. This standardization process of creating APIs for connecting to the core kernel acts as a stabilizing force for the product in general. The stability of the APIs, and their consistency in implementation, can be directly correlated to the stability of the underlying product as a whole. The reason for the correlation is that when changes to an API are more difficult to effect &mdash because of their corresponding effects on other code — stability increases due to fewer changes in the way that components interact with each other.

Case in point: the “pluggable” storage engine API in MySQL was introduced in early MySQL 5.0 versions. This API has changed numerous times over the past two years as different vendors (internal and external to MySQL) needed or wanted additional functionality. The API has started to stabilize now, but I see additional changes in the future. What has happened to the stability of the underlying code base of MySQL? I will leave that answer to the reader’s intuition.

Increased Performance

The reason I believe modularization leads to better performing software is because the act of modularizing identifies pieces of the software which can and should be “pulled from the core”. Pieces not central to the functioning of the core kernel are pulled out of the core product and placed, rightfully, into modules which provide additional, value-added-for-a-certain-group-which-needs-it features. By placing extra features outside of the realm of the “necessary”, the core kernel begins to take the shape of a lithe marathon runner and not the bloated monolith it once was.

You may wonder why this removal of extra features can lead to better performance of the product as a whole. By removing non-essential parts of the software into modules, the code for the core part of the software is simplified and shortened. Now, simple, shorter code isn’t necessarily faster just for being shorter (though often it can be). However, the ease of maintenance on a simplified core software kernel allows developers to focus on performance-related tasks involving that simplified kernel. Improving performance of essential runtime functionality is much easier and faster when the code isn’t littered with calls and code paths that correspond to the functionality that belongs in an external module. If performance issues in the core kernel are easier and faster to fix, then the product as a whole becomes better performing.

Secondly, the modularization of a code base means that users who do not need an array of features can use a smaller, streamlined product that only fits their needs. Often, when users can make use of a streamlined project that doesn’t contain code they don’t need or want, the resulting binary is smaller and faster.

Increased Ease of Use

How modularization increases ease of use is related to how it increases performance: by enabling developers to only focus on the module at hand, or the core itself, instead of a mix of both, the developer is freed to focus on fixing ease of use bugs and addressing usability issues. The more time a developer can spend actually doing the things that comprise her outstanding bug and feature list, and not on “how will my code affect ten other developer’s code”, the better the chances of a product’s usability becoming better.

Finally, Increased Competition Fosters Innovation

I think it’s fairly obvious that increases in stability, performance and ease of use translate into increases in innovation for the product as a whole.

My final thought on why modularization leads to increased innovation is that modularizing a product leads to competition in the developer marketplace for a specific feature. This competition spurs the developers of a competing module to do it better and faster. Case in point: the pluggable storage engine API “modularization” effort at MySQL spurred the creation of numerous competitive engines — from Paul McCullagh’s PBXT engine to SolidDB’s transactional engine to the new Falcon and Maria engines. Each engine demonstrates different characteristics, benefits and disadvantages. Without each other, the wellspring of innovation would be much drier.

Innovation Leads to Market Share

So, assuming you’ve come with me in my theory that open development communities foster modularization of a product, and that this modularization leads to more innovation for the product as a whole, then I think you will be able to make this final step easily: innovation in a product leads to greater market share.

It doesn’t matter whether the product is the result of a group of loosely-affiliated individuals or a company like Sun or MySQL, each producer of a product wants their product to have the most market share compared to competing products. It’s human nature; we just want to be popular!

Jonathan Schwartz is Right About Community, But Wrong About Why He’s Right

So, in Babcock’s interview, Jonathan Schwartz says,

Everything begins with the development of a community

Jonathan is right about everything beginning with the development of a community. However, I believe Jonathan, and a number of folks I’ve met personally at Sun, feel that simply having a community automatically makes a good and popular product. This simply isn’t true. It’s about the product, first. Then the community. A vibrant community (both developer and user) springs from the fountain of enthusiasm about a product. If that enthusiasm wanes, because of anything from the closing of the development model to the inability to get their voices heard regarding a product roadmap or architecture, then the community begins to die along with the product. It is a delicate thing, this product and community balance sheet.

This well is poisoned; the company has consumed its own future
and any pretense that the projects will ever govern themselves
(as opposed to being governed by whatever pointy-haired boss
is hiding behind the scenes) is now a joke. Sun should move on,
dissolve the charter that it currently ignores, and adopt the
governing style of MySQL. That company doesn’t pretend to let
their community participate in decisions, and yet they still
manage to satisfy most of their users. Let everyone else go
back to writing code/documentation for hire.



There’s nothing particularly wrong with that choice — it is
a perfectly valid open source model for corporations that
don’t need active community participation. IMO, the resulting
code tends to suck a lot more than community-driven projects,
but it is still open source.

The above is a quote from Roy Fielding’s email to the OpenSolaris developer’s mailing list in which, at the end, he resigns from the OpenSolaris community. I think it is telling about how delicate the balance between product and community really is. He points out that it was the broken promises about a truly open development model that was the deciding factor in his resignation. I suspect that frustration at not having a voice in the architectural decisions surrounding OpenSolaris were also a big factor, too. Pointedly, he states that MySQL has never pretended to let our community participate in decisions, but MySQL still enjoys a large and vibrant user community.

I theorize that the MySQL user community is in danger of becoming fragmented and is increasingly fragile because of the “we decide” nature of MySQL’s closed development model. I believe that MySQL achieved ubiquity and a huge user community because of this:

1. In the beginning, it “just worked”, was fast, reliable, and easy to use.
2. It became ubiquitous in both language packaging and Linux distributions very early on and only because of #1 above

In other words, it was about the product itself.

If the product loses touch with its roots and because of a closed development model moves further away from easy to use, fast, and stable, the user community and MySQL’s ubiquitous nature, will move away to competitors.

Again, an open development community is a method to achieve greater innovation in a product. I believe it is silly for either MySQL or Sun to make any steps that do not open up its development activity to the wider community. If openness spurs innovation of the product, closing off of development only means that the company is stifling innovation and decreasing its own market share. If a method is there to increase innovation, why disregard it?

Why OpenSolaris Could Be a Real Competitor to Linux

It’s about Innovation in the Performance, Stability and Ease of Use of a product. As Ian Murdock wrote on the OpenSolaris blog:

My basic observation, as someone
who came into the OpenSolaris community from the outside – even perhaps
from the competition – and who represents the target market this
community needs to reach was this: That the packaging and presentation
of OpenSolaris as it stands today represents a barrier to adoption and,
thus, an obstacle to growing the OpenSolaris community and bringing in
new users. To lower these barriers, OpenSolaris needs to be more than
just the code base. It needs to be a binary that users can easily
download and install to get easy access to OpenSolaris technology. Put
another way, as I said in a blog post in June, we need to have a
better answer to the question, “Where do I download OpenSolaris?”

Ian understands that a product gets adoption when innovations in ease of use, performance, and stability are demonstrated. In the above quote, he talks about the problems in ease of use that are barriers to adoption of OpenSolaris.

Similarly, there are a number of performance-related and stability-related innovations that are contained within OpenSolaris. But the product will not achieve true competitive status with Linux until the development community is opened up and innovation from the external community — in the form of a voice and control over architectural issues and design — is truly embraced at Sun.

If Sun embraces an open development model and embraces its developer community in discussions on roadmap and architectural decisions, I think OpenSolaris has an excellent chance of competing head-to-head with Linux on the innovation front. If the product is more innovative on a variety of levels, it will gain market share.

What This Means for MySQL

As Brian Aker rightly points out, MySQL has never been an open development model. But steps have been taken to open it up. The resignation of Fielding suggests that it’s not the lack of openness which frustrated him, but the promises of Sun to do so. This means that MySQL must be very careful not to over-promise on something it cannot or will not deliver regarding openness.

In addition, if we are to reverse the current course of a community-in-flux, we must embrace the fear we have of opening up our development process. We must get out of the cathedral and put up shop in the bazaar. We won’t be jumping out of the cathedral’s top window; more likely we’ll rappel down Rapunzel’s hair slowly and carefully. I see very good prospects about opening up in the future and I am excited. It will be happening in the nick of time.

Take everything I write with a grain of salt. After all, I’m just a lowly community relations manager.

Footnotes

[1] Amanda McPherson writes:

We may not have fancy Linux analyst days and mountains of spin, but it is all about the development community. Literally.

to which Mike Dolan responded in comments:

Yes, yes, and yes. This Sun nonsense needs to be called out; it’s great to see someone else actually looking into the critical details behind Sun’s fluffy marketing.

[2] I differentiate “developer community” as the developers which develop the product, as different from the “user community” which a) develops products on top of or for the product or b) simply use the product.