join-fu!

the art of sql

Archive for category Python

Developing Nova on Linux – Getting Started

Jul 22

Posted by jaypipes in Bazaar, Launchpad, Python, openstack | View Comments

In the past few weeks, I’ve gotten involved in the newly-debuted OpenStack project. Right now, my focus is on the Compute sub-project of the stack, called Nova. The initial pieces I am focusing on are the unit tests and end-to-end systems testing of the compute stack.

I struggled over the last couple days to solve a bug that turned out to be not a bug at all, but an issue with the Python development environment I use. I figured I’d write a blog article for those Python developers who are looking to contribute to the Nova project and may also be struggling to get up and going.

If you’re contributing to an open source project like Nova, you’ll want to be able to work on multiple branches of the source code at the same time — for instance, if you’re working on fixing a few bugs simultaneously.

There are quite a few dependencies for Nova, and, because of the way Python searches for packages, it’s imperative that you use a tool such as virtualenv to isolate your multiple branches into their own development environments. Otherwise, as I learned today, the location of your site-packages and what has previously been installed on your development machine can wreak havoc on you. :)

NOTE: For this article, I assume the reader is on Debian/Ubuntu Linux, since that is what I use as my development machine. If you’re on a different flavour of Linux, feel free to adapt the instructions here to suit your particular package manager.

Installing the Tools for Installing the Tools

Before we get into our virtual development environments, you’ll first want to ensure you’ve got a few packages installed, including bzr, libssl-dev, swig and virtualenv. The following should do the trick:

sudo apt-get install -y swig libssl-dev bzr python-virtualenv

A Setup for Source Control and Virtual Environments

In order to get properly setup to contribute to the Nova project, you’ll want to setup a local repository to keep branches of source code that you work on. Although bzr is not required as your revision control system, I use bzr myself and will use it in this article. Adapt as needed if you use git-bzr or similar.

I like to have the following directory structure for working on Python projects:

~/repos/$projectname/ <-- shared repository for branches of your project
~/repos/$projectname/trunk <-- local trunk branch
~/repos/$projectname/$branch <-- a branch you work in
~/virtenvs/$projectname/ <-- Development environments for your project
~/virtenvs/$projectname/$branch <-- development environment for a branch you work in

Assuming you want to contribute to the Nova project and you want to work on fixing a bug #XXXXX, then following would get you started:

bzr init-repo ~/repos/nova
cd ~/repos/nova
bzr branch lp:nova trunk
bzr branch trunk bugXXXXX
mkdir -p ~/virtenvs/nova

At this point, we'll go ahead and create a virtual development environment for bugXXXXX:

cd ~/virtenvs/nova
virtualenv --no-site-packages bugXXXXX
cd bugXXXXX
source bin/activate

At this point, you'll notice your prompt change, indicating that you are now in a virtual development environment. The --no-site-packages ensures that your locally-installed Python packages aren't included in your Python PATH when inside your virtual environment.

Next step is to install into this virtual development environment all the packages and dependencies we'll need. This should do the trick:

easy_install twisted tornado boto M2Crypto IPy carrot mox redis
easy_install http://python-gflags.googlecode.com/files/python_gflags-1.3-py2.5.egg

Alright, next we simply link to our bzr branch location from inside the virtual environment and run the Nova test suite:

ln -s ~/repos/nova/bugXXXXX bugXXXXX
cd bugXXXXX
python run_tests.py

If all went smoothly, you'll see all passing test cases, like below :)

Having issues getting up and running? Find us on Freenode IRC #openstack.

See ya,

Jay

Macro Support in new Drizzle Client Console?

Nov 18

Posted by jaypipes in Drizzle, MySQL, Python | View Comments

Hi all!

I’ve been reading through the requested features for the new client on the wiki here:

I think all the stuff on that link is excellent so far. I’d also like to request a feature that I think will be a really cool timesaver for DBAs and developers using Drizzle.

Macro Support

Remember, “way back when” you used Microsoft Excel and were able to start recording your actions, then when you stopped recording, Excel would store a “macro” of your actions that you could subsequently replay?

I think this would be incredibly useful for folks who do repetitive work in the console.

Sure, I know, I know…the first reaction folks will say is “but HEY, you guys removed stored procedures!” Yeah, yeah… but the feature I’m proposing here is different from stored procedures in the following ways:

  1. It’s entirely client-side. There is no server-side storage/cache, processing, parsing, or anything.
  2. It’s not limited to a small subset of SQL that stored procedures (at least in MySQL) are currently limited to. Anything the new client can do would be able to go into a macro.
  3. Since the client is in Python, the macros are themselves re-writable in a scripting language. This gives the recorded macros incredible flexibility.
  4. No fussing with SQL stored procedure permissions at runtime (you know, the silly INVOKER/DEFINER crap)
  5. Ability to interact with result sets in the macro. Just try doing that easily in a SQL stored procedure. Using CURSORs is incredibly clunk and ugly. Applying a Python function or closure/lambda on each of a result set is elegant and easy.

Imagine the following rough example interface… 

drizzle> RECORD MACRO "sales_report_with_email" (to_email);
macro recording started.

drizzle> mode python;
in python mode.

python> import datetime
python> today= datetime.datetime.now().isoformat()
python> filename= "%s-%s-%s" % ("sales", to_email, today)
python> Ctrl-D

drizzle> SELECT * FROM sales
         WHERE manager = @to_email; > csv(@filename);
drizzle> mode python;
In python mode.

python> report_txt= open(filename, "r+b").read()
python> import smtplib
python> mailserver = smtplib.SMTP('localhost')
python> mailserver.sendmail('theboss@company.com', to_email, report_txt)
python> mailserver.quit()
python> print "Mail sent to %s\n" % to_email
python> Ctrl-D

drizzle> STOP MACRO;
Macro "sales_report_with_email" saved.

drizzle> macro("sales_report_with_email", "myboss@company.com");
Mail sent to myboss@company.com

Pretty powerful, eh?

If you follow the flow above, you will notice the only real trick to solve is passing the macro’s arguments into the console’s variable array, and from the console’s variable array into the Python interpreter’s variable scope. But this is a fairly simple problem to solve…

Thoughts? Suggestions? If you’ve got comments, please feel free to share here, or on the Drizzle Discussion mailing list, or even update the wiki pages posted above. Thanks! :)