My Top 5 Wishlist for MySQL

I’ve actually been meaning to finish this blog entry for quite some time, and now finally had the chance. Recently, Brian Duff wrote an article entitled “If I Had 5 Oracle Wishes, They Would Be…“. I thought the topic would be a good blog meme to start within the MySQL blogger community, so I’ll go ahead and start it off for PlanetMySQL and see if anyone else chooses to pick up the topic and pass it on. So, here are my top 5 wishes for MySQL in the future…

The Modularization of a Core MySQL Server Kernel

Certainly, this particular idea isn’t new at all. I discussed the concept with a number of people at the first MySQL Camp last November, and found some receptive ears. Additionally, I know a number of engineers at MySQL have been bandying around the concept of a modular core kernel for quite some time. Imagine a smallish MySQL runtime kernel that would essentially focus on the communication layer between the server, and external modules that would provide added functionality.

If you think that I am describing something similar to Apache2, then you are not alone. When I recently proposed this idea in an internal email, Jim Winstead wrote:

congratulations, you’ve reinvented apache 2.0.

When I read that, I thought, “well, there are certainly worse things to be compared to…” But, sarcasm aside, Jim is right on the mark. Though for sure there are ways that MySQL’s runtime would differ from Apache’s APR, the idea is the same. Isolate core communication and threading functionality within a runtime kernel, and expose additional functionality through module APIs.

There are a number of reasons I hope to see a more modular, encapsulated core kernel.

Less risk of “my code” screwing up “your code”

Having a core server kernel would allow developers — both internal and community — to work on extensions and modules outside of the core kernel without worrying about their code negatively impacting other subsystems or modules. Module developers could focus on code specific to their piece of the puzzle, while developing to a standardized set of APIs that would allow their module to both communicate with the core kernel, as well as share state and data with other modules.

“Deregulated” versioning

By modularizing a core server runtime with a standardized set of APIs, a significant problem that exists today might also be mitigated: that of release numbers and versioning. You see, right now, if MySQL Server goes into BETA, technically that means that no new features can be added to that server version. Of course, this hasn’t always happened for various reasons, but there is (rightfully) strong backlash inside MySQL whenever someone wants to include new functionality in a beta release.

If there existed a core runtime, with its own version (which likely would correspond to API versioning), then modules could have their own versioning scheme, very similar to Apache modules. Module writers could continue to release code and even come out with new alpha releases of their modules, without having to wait for any other team or individual.

Of course, this will add some level of complexity to versioning — consider if you can name off the top of your head the actual versions of Apache modules running on your machines… However, I believe the benefits of decentralized development and the ability of MySQL engineers, community members, and partner vendors to create modules and release those modules without having to wait for an alpha release schedule, are worth that extra complexity.

The Ability to Compile Specialized Packages of the MySQL Server

One of the big problems any piece of software faces is that as customer and community demand more features, the code base often becomes more complex, and with this complexity, performance can suffer. It’s not that difficult a concept: the more you add to the software, the more lines of code are needed to accomplish a specific task. Unfortunately, as you add features to software, those added features are only used by a small subset of users. For the rest of the software users who do not need or want such a feature, they often are stuck with degraded performance as the cost of those features. Having a modular kernel would allow features to be added as extensions to the core runtime, allowing users to pick and choose the features they wanted and create customized MySQL server packages that offered only those features they needed. Performance of modular systems is less impacted by adding more features, as those features would not need to be compiled into the server for those users not needing or wanting those features.

The Addition of Community Advisory Boards

OK, so I’m a MySQL Community team member, so of course, many of the thoughts I have relate to growing the MySQL ecosystem and reducing the friction points between external MySQL community members and the MySQL internal engineering and product management folks. The idea of a Community Advisory Board is fairly simple, and stems from our existing CAB — Customer Advisory Board — meetings. These CAB meetings are critical for MySQL’s product management team in identifying the pain points that our customers have with MySQL.

I see no reason why such meetings should not occur with the external MySQL community. And by “external MySQL community”, I mean those folks directly involved in the MySQL user community, as well as groups of people such as:

  • Linux distribution build folks
  • Committers for key FLOSS projects such PHP, Perl, Python, Ruby, Apache, Eclispe, etc
  • Key members of FLOSS legal groups, such as EFF, FSF, Creative Commons, etc.

The Community Advisory Board (CoAB) would meet on a regular basis, either in person or virtually, and the purpose of the CoAB would be very similar to the regular CAB meetings, except that the focus of the CoAB meetings would be what the external community feels are significant issues that should be addressed by MySQL and/or the external MySQL community. To be sure, findings from both the CoAB and CAB meetings may overlap, but that would only serve to further highlight significant issues for both the customer and user communities.

Of course, a process for identifying individuals to participate in such CoAB meetings would need to be established, and regular meeting dates be solidified, but I think this Wish might be an easy-to-accomplish one… Thoughts?

The Realization of the MySQL Build Farm Initiative

Here is another community-focused agenda item of mine that has recently been moving forward with the help of Adam Porter and Sandro Fouché from the University of Maryland. I first contacted Adam and Sandro mid last year after Sebastian Nohn urged me to get in touch with them. They are working on something called the SKOLL Cluster for Distributed, Continuous QA and Testing. More on SKOLL in a second…

The MySQL Build Farm Initiative was something that was proposed but for various reasons never really got off the ground. The idea was to have a centralized build farm that MySQL community members could enlist their spare servers for building and testing MySQL, similar to the PostgreSQL Build Farm. Internally, MySQL has a build system called PushBuild that does something kind of like this, but only on internal machines. The Build Farm was intended to take PushBuild to the next step and allow MySQL community members to “fill in the gaps” of platforms and architectures that MySQL did not have actual machines for testing.

It was a grand idea, but sadly fell victim to the “too many projects, too little time” sickness. Fortunately, however, while nothing was going on with the MySQL Build Farm, the same could not be said for Adam and Sandro’s work over at the SKOLL Cluster. In fact, Sandro and Adam had been steadily building and compiling over 80,000 configurations of MySQL over the last 6 or so months using a prototype distributed continuous build system for MySQL, and had started winnowing down, through mathematical analysis, which configuration and build options would produce the largest number of failures.

Omer BarNir, our Senior QA Architect and Mads Jørgensen, our Release Manager, had a conference call with Adam, Sandro, and myself a couple weeks ago, and I think it’s safe to say Mads and Omer were extremely excited at what Adam and Sandro have accomplished and are eager to take the prototype to the next level. We have our second conference call this Thursday to discuss the next steps.

Eventually, my hope is that the SKOLL Cluster and the QA system will be the catalyst for a true distributed MySQL Build Farm platform. That day may not be that far off… 🙂

Working with Microsoft to Improve MySQL Performance on Windows

I know Reggie will be very pleased to see this one in my Top 5 Wishes… The idea stems from a phone conversation I had back in April with Andi Gutmans. We were talking about what needed to happen to get a PDO working group established, and I had asked Andi what Zend engineers had been up to recently. He responded that one of the big things that they had accomplished in the past year or so was improving PHP application performance under Windows.

Andi said that Zend folks had worked quite closely with Microsoft engineers in identifying the pieces of the Zend Engine where performance could be significantly improved on Windows, and that the relationship between Zend and Microsoft had truly produced impressive performance results for PHP on Windows. He said being able to work with engineers at Microsoft that really understood the low-level API issues on Windows platforms was critical in getting PHP performance on Windows to be on par with *nix systems.

I would love to see the same level of interaction between MySQL and Microsoft. There is no reason it shouldn’t and can’t happen. Software patent legislation aside, there are no good technical reasons why a solid pipeline between Microsoft engineers and MySQL engineers can’t be laid so that MySQL’s performance on Windows can be improved in the same way that PHP’s performance was. The point here is not that MySQL’s performance isn’t good on Windows, but rather that Microsoft’s expertise can only help improve it.

Identifying and Supporting Unique Sub-communities of MySQL Users

Wish #5 stems directly from thoughts generated from listening to the excellent Jono Bacon‘s talk at SCALE5X called “How to Herd Cats and Influence People“, which was about the challenges FLOSS communities face, how to enable communities to efficiently produce things, and how to remove friction points and communication barriers between related groups in the ecosystem.

One of the biggest points of the talk was the idea that user communities will flourish when the identity of a subset of the users is strongly defined, and when tools, channels, and content is directed at this targeted sub-community. In plain words, if you compare a user community that only has tools and content that addresses generic needs with a user community that has tools and content that address targeted needs of subsets of that community, the latter community will grow faster than the first.

The concept makes perfect sense. Everyone wants information and tools that address their own specific needs and allows them to “define themselves”. We want to identify with a small, niche group of people and feel like we belong to a unique, focused clique. It’s human nature… 🙂

Unfortunately, we haven’t done a great job at MySQL, especially in the community team, of following this guiding principle. Although we do have targeted content for developers and DBAs, for instance, that information is sometimes:

  • hard to find
  • outdated
  • not community-driven or controlled

My wish for the future is that we can begin to reshape and restructure the tools and content that the MySQL community provides for itself, and create ways that distinct groups of users can better communicate, find each other, share information, and in general feel like a unique clique.

PlanetMySQL already has engendered a bit of this kind of concept: there is now a unique community of MySQL bloggers, and PlanetMySQL is the glue which pulls those bloggers together. While there are certainly many improvements that can be made to PlanetMySQL, the concept of providing tools for a subset of community members is still valid. My hope is that we, as a community, can collectively work to make the MySQL Forge a real platform for providing these subgroups of our community the tools and content they want and need.