Comments on: Drizzle Replication – Changes in API to support Group Commit

By: David Shrewsbury: Drizzle Transaction Message Limit | Weez.com

David Shrewsbury: Drizzle Transaction Message Limit | Weez.com — Mon, 06 Sep 2010 20:09:10 +0000

[...] recent changes I made have recently been pushed to Drizzle trunk that affect the size of the Transaction protobuf message that any replication stream will see (e.g., the transaction log). This was necessary to fix bug [...]

By: Padraig O’Sullivan: Simple Drizzle Replication Plugin for Cassandra | Weez.com

Tue, 01 Jun 2010 20:08:26 +0000

[...] within the drizzled/message/transaction.proto file within the Drizzle source tree. Jay has previously gone into great detail on the GPB message definitions and I see no point in duplicating the great [...]

By: Sneak Peek – Drizzle Transaction Log and INFORMATION_SCHEMA « join-fu!

Sun, 14 Mar 2010 22:10:07 +0000

[...] also finished a new UDF for Drizzle called PRINT_TRANSACTION_MESSAGE() that prints out the Transaction message’s contents in a easy-to-read [...]

By: Recent Work on Improving Drizzle’s Storage Engine API « join-fu!

Recent Work on Improving Drizzle’s Storage Engine API « join-fu! — Sat, 13 Mar 2010 07:13:02 +0000

[...] of changes to the replication API and now the GPB messages sent through the APIs are of class message::Transaction. message::Transaction objects contain a transaction context, with information about the [...]

By: Drizzle Replication – The Transaction Log « join-fu!

Drizzle Replication – The Transaction Log « join-fu! — Sat, 13 Mar 2010 04:31:20 +0000

[...] be joinfu? « Drizzle Replication – Changes in API to support Group Commit If Ever There Was a Sure-fire Tenant… [...]

By: Drizzle Replication – The CommandReplicator and CommandApplier Plugin API « join-fu!

Sat, 13 Mar 2010 04:13:30 +0000

[...] This article is out of date and the replication API has been updated. Please see the follow-up article for the most up to date [...]

By: Drizzle Replication – The Command Message « join-fu!

Drizzle Replication – The Command Message « join-fu! — Sat, 13 Mar 2010 04:05:10 +0000

[...] This article is out of date and the replication API has been updated. Please see the follow-up article for the most up to date [...]

By: Paul McCullagh

Paul McCullagh — Thu, 29 Oct 2009 17:40:02 +0000

Hi Jay,

Great work! GPB based replication is a huge step forward for Drizzle.

I also like the way you have packed the DML statements. They can be easily converted to SQL, but are also in a convenient format for use with other, non-DBMS, destinations.

One question I have is: How do you plan to handle the following situation?

A transaction is executed and the GPB message is prepared containing all the statements of the transaction. However, before the message can be passed to the TransactionReplicator the server crashes.

Now on startup the transaction will be committed to the database, but there is no way to reconstruct the GBP message for the transaction (and the transaction will go missing from the slave).

It would seem to me that the TransactionReplicator API needs to be expanded to include XA functionality.

And, this may have to be extended to the TransactionApplier as well, to ensure that the transaction is reliably transported from end-to-end.

Is the solution part of the transaction log explanation which is coming up?

I’m looking forward to reading about it all.

Best regards,

Paul

By: Mikiya Okuno

Mikiya Okuno — Thu, 29 Oct 2009 13:46:30 +0000

Hi Jay,

Thank you for your quick response! I’m excited about your comments. Drizzle is more progressive than I thought.

“Well, Eric Lambert has written a replicator which does exactly this. A C++ TransactionApplier plugin pushes messages out to a Gearman server.”

This implementation sounds really cool and very clear to understand. As Gearman flatten some workload nicely, I can expect that some bottleneck when having many replication slaves on a single master would be eased I’m looking forward to see his entry.

Kind regards,

Mikiya

By: Jay Pipes

Jay Pipes — Thu, 29 Oct 2009 12:59:24 +0000

Hi Mikiya!

“BTW, of course you might consider doing so, I suggest adding a checksum field in each Transaction Message or Statement Message.”

Actually, there is already a checksum written after each transaction message is written to the transaction log. Please see the next article for information on that.

It is enabled using the –transaction-log-checksum-enable command line argument to Drizzle. Currently, it just uses a CRC32 checksum, but I plan on making it configurable which checksumming algorithm should be used…

“One more suggestion is to make replicator slave as an isolated, standalone process which connects to drizzled using libdrizzle. It will enable drizzle to replicate to MySQL very easily.”

Well, Eric Lambert has written a replicator which does exactly this. A C++ TransactionApplier plugin pushes messages out to a Gearman server.

Worker processes run a Java Gearman job in the worker process pool applying the replicated messages to the servers configured as replicas. Eric will be blogging shortly about this new code piece.

Cheers!

jay