Drizzle, MySQL

Last Week in Drizzle

This is the inaugural post in the weekly series “Last Week in Drizzle” where I summarize the efforts of various folks in the Drizzle community over the past week. There is a ton of enthusiasm and excitement around what is going on in Drizzle as evidenced by an explosion in the number of people tracking the mailing list and participating in coding, refactoring, testing, documentation, and general community efforts. As of this writing, 148 people are following the mailing list; this in only the second week since Brian announced Drizzle at OSCON. Pretty awesome. What is also cool is seeing a large number of MySQL engineers, consultants, and support engineers participating in the mailing list discussions and in the refactoring work.

There are now 34 folks chatting over on the Freenode #drizzle channel, and the discussions are enlightening. It’s fantastic to see developers sharing ideas and code in the most community-oriented way: just doing it. Conversations are quick and short, usually ending in something like “OK, off to prototype that.” or “OK, cool, I’ll let you know when I get that working in a branch.” This is fantastic, and is a testament to the way the community around Drizzle is breaking down any barriers around contributing to the project.

Before I summarize what various Drizzlers have been working on this past week, I encourage anyone interested in participating to hop on to the mailing list and the IRC channel and hang out. If you’re like me, you’ll learn a lot just by being around and listening to the conversations!

Last Week’s Progress

The commits were a-flowin’ last week. No fewer than nine developers were committing to the main trunk, including Brian Aker, Eric Day, Ronald Bradford, Monty Taylor, Patrick Galbraith, Mats Kindahl, myself, Stewart Smith, and Toru Maesaka. Here is a bit of info about what we were working on. Most of the work still revolves around refactoring and removing dead or legacy code, but Brian had some interesting commits which re-tooled the libevent stuff in the scheduler and adding multi-interface support.

Tackling client refactoring and i18n

Monty Taylor had another week of tackling the refactoring of the client library and the autoconf/make system.

Monty also did a good amount of work laying the groundwork for internationalizing the error messages in both the client and server using the standard gettext library. Here wrote up a good post about it.

poll() and multi-interface support

Brian, in addition to refactoring of the scheduler to use poll() more effectively, worked on removing some more data types, including NCHAR/NVARCHAR (yep, I bet you didn’t even know they existed…). Why remove these data types? Well, it reduces the size of the parser (speeding the parser up) and reduces the complexity of the code base.

Improper or redundant typecasts removed

Mats Kindahl, resident C/C++ standards guru, tackled a refactoring effort while on vacation this past week: removal of redundant or improper typecasts in the server code. See the corresponding Blueprint and code for more information on the work he is doing.

Refactoring built-in and plugin functions

Stewart was working on refactoring the MD5() function into a plugin and removing calls to the custom crypt() functions in the mysys library. The end-goal of Drizzle is to have aall functions be loadable via the plugin interface. Why? Well, it’s a long discussion, but the summary is that it enables the HASHes of the functions used by the drizzled server to be reduced, resulting in a higher probability that function lookups will be cache-line hits, resulting in faster overall performance.

Header/Include spiderweb madness

Myself, I worked this week on refactoring the hideous beast known as mysql_priv.h. It is the large header file that is #included by a large part of the server. It also includes some client header information, which is confusing to developers and means 90% of the file is #ifdef’d around a MYSQL_SERVER preprocessing conditional. The file is a) poorly commented, b) contributes to “include spiderweb hell” and c) has served as a generic catch-all place for developers to throw struc, class, and function declarations when they didn’t want to create an appropriate header file for their code. And so, it’s a target for refactoriing.

Buildbot and MacOSX fixes

Ronald was instrumental in getting the Buildbot master and configuration setup and running on his server, hosted at 42SQL.com, and at the time of this writing, we already have 18 build slaves compiling and testing the drizzle server and clients. This is perhaps the single most thrilling accomplishment of last week, as it demonstrated how the drizzle community could quickly and effectively come together and using open source tools, get a build platform publicly visible. This is, by the way, something that the MySQL development organization internally has — although it is a custom build and test platform called pushbuild and is not publicly visible. I hope that the drizzle community’s experience and lessons in setting up Buildbot for drizzle can be a learning experience for the MySQL engineering and build team. Shared lessons and experience is, after all, what we’re all about. 🙂

Ronald wrote in an email, the following about the experience:

Using the Build Farm I was able to identify OS/X specific problems, and
following the [Contributing Code] instructions I was able to correct the
compilation errors, and push this back as a proposed merge, being my first
contribution to Drizzle Code.
Following review and correction it was accepted and incorporated into the
Drizzle main branch.

and

Working on IRC #drizzle is essential. In my case, by submitting code
for proposed merging, review by a fellow colleague showed I’d not followed
the coding standards to the letter of the law, so I was able to quickly
correct.

Great to see this level of quality being sought for any submissions.

It’s exciting when you are working with motivated people, it motivates you
to want to contribute to open source.

I agree, Ronald. 🙂

A brand new client

Eric Day was working on a new client functionality and mailed the following:

I created a branch for the development of a new client library. This involved first dissecting the old library, seeing what we wanted to keep, and then creating the first pass of a new interface for review. I’m now working on the implementation details.

This new library will offer much better memory management and non-blocking support!

I got some great feedback form Brian and Monty, along with some other people in #drizzle.

It’s great to see a real community building around this project! There are new members in the channel offering to help everyday.

Field cleanup and removal of mystrings dependencies

In the past couple weeks, Toru has been busy refactoring both server and client functionality. Inside the server, he broke up the Field classes into a separate libfield library and directory structure, which results in both a cleaner code structure and an easier-to-decipher field type system. Each field type now is housed in drizzled/field/typename.h/.cc files which makes it easier for developers to extend custom types and understand the field type system better.

In the client library (libdrizzle/), he’s been replacing calls to the custom mystrings/ library (called strings in the MySQL source tree) with calls to standard C string functions.

Testing, testing, testing

Andy Lester has taken point on the test suite refactoring and has been actively discussing moving towards a more standardized, TAP-based output for the test suite. With input from many in the drizzle community, he will be working with Ronald and Giuseppe to make the test suite more usable and extensible. I, for one, am very much looking forward to the results of their work, as I want to see a standardized method for patch and feature contributors to create test cases that validate functionality or bug fixes contributed.

DBD::Drizzle materializing

Patrick Galbraith spent last week creating a working DBD::Drizzle connector for Perl developers. He says that the module is mostly complete and now needs testing. Nice work, Patrick. 🙂

Come Join Us!

I apologize in advance for any errors above. Feel free, those mentioned, to correct me! And to everyone reading this, please come participate in Drizzle. It’s a great adventure 🙂