Drizzle CI infrastructure getting matured

Its been quite long since I had a discussion about the new CI infrastructure for Drizzle. Well, the repository is getting better and better everyday. A lot of development has been made and its not far from completion. So I just thought I could give my readers a glimpse of what is going on while I finish off with the final few pieces of dev work in the interim.

So, in this discussion, we will be moving on to the stage where the repo had states for the following:

1. Installing Drizzle

2. Ensuring that the drizzled service is running

3. Setting up the SysBench tables for regression testing

4. Installing SysBench package

And below is the output when I run the magical  “salt ‘*’ state.highstate”..

State: – pkg
Name: drizzle
Function: installed
Result: True
Comment: The following packages were installed/updated: drizzle.
Changes: drizzle: { new : 2011.03.13-0ubuntu5 old : }
State: – service
Name: drizzle
Function: running
Result: True
Comment: The service drizzle is already running
State: – file
Name: /home/shar/sysbench_db.sql
Function: managed
Result: True
Comment: File /home/shar/sysbench_db.sql is in the correct state
State: – cmd
Name: drizzle -uroot < /home/shar/sysbench_db.sql
Function: run
Result: True
Comment: Command “drizzle -uroot < /home/shar/sysbench_db.sql” run
Changes: pid: 15947
retcode: 0
State: – pkg
Name: sysbench
Function: installed
Result: True
Comment: The following packages were installed/updated: sysbench.
Changes: sysbench: { new : 0.4.12-1build2 old : }

Soon I will give my readers the flavor of the complete configuration of test nodes with the topping of automation 🙂

Parallelization and Automation of Drizzle CI infrastructure

This post is all about my work for Google Summer of Code 2013. Drizzle Continuous Integration uses Jenkins CI tool at present. Drizzle had dedicated servers which ran Jenkins server / master with a few slaves. However, the infrastructure did not support parallel running of tests / builds to a scalable extend. So the project aims in bringing parallelism with automation.This can be envisioned as follows.

Drizzle CI does builds using Jenkins. It tests for regression using the available test suites. To bring in parallelism, the CI infrastructure executes  test jobs, which may include a build, a sysbench test, a randgen test, etc., simultaneously. A dedicated system / cloud node is configured for execution of each job in the job queue. Multiple tests + Multiple nodes = Parallelism.

Configuring these nodes is handled in a simple way. It is as simple as writing a config file of specified format, and then issuing a command which configures the nodes accordingly. Cloud nodes + Configuration files = Automated configuration.

To put it in a nut shell, the first work would be to automate the setting up of cloud nodes, and the later half would be to parallelize the test runner. In the following posts, I will be updating about the progress and write about the tools which are involved in the work, how to use them, etc. 🙂

Drizzle says hello to SaltStack

This post is about my next work, Importing Drizzle database to Salt modules. This was suggested by my mentor Patrick Crews and I thank him for giving me this golden opportunity of extending Drizzle to SaltStack. This is the first of the series of posts regarding this project.

I have done some initial work and have successfully connected to Drizzle server. I ll first tell about the changes to be made in the master / minion config file. Before using Drizzle, we have to enter the configuration parameters namely host, user, password, port and schema in the suitable config file. ( /etc/salt/master for master config or /etc/salt/minion for minion config ). If we want to use Drizzle in a specific minion, we can add these parameters in that minion’s config file. Else, if we want to access Drizzle on all the minion, then add these parameters in the master’s config file. You can just append the respective file with:

drizzle.host: ‘’

drizzle.user: ‘root’

drizzle.passwd: ”

drizzle.db: ‘drizzle’

drizzle.port: 4427

Once this is done, restart the master with /etc/init.d/salt-master restart or the minion with /etc/init.d/salt-minion restart. Now you can access the Drizzle database through remote execution 🙂

As an introductory work, I have written the method for retrieving the version of Drizzle server that is running on the minion. This can be used with the command option : salt ‘*’ drizzle.version.

In my system, I got the output as :

sharan@sharan:~$ sudo salt \* drizzle.version
VERSION(): 7.1.33-stable

So Drizzle module will be available in full swing in the fore coming weeks.. Stay tuned.. 😀

Improved Performance Regression Monitoring

As a part of Google Summer of Code 2012, I worked on Improved Performance Regression Monitoring for DrizzleDB.. At the bottom, its a Quality Assurance and Automated Testing project.. Drizzle already has *Drizzle Automation* which deals with automating QA processes, code coverage analysis and benchmarking.. However due to the following issues, its not preferred much..

1. Its not in-tree

2. Its more complex

3. Its less user friendly

In order to nullify this, the kewpie tests came into effect. Kewpie is a test-runner for MySQL based databases. Its cleaner, simpler and very much effective. Moreover, its included in-tree.. The project’s intent is to import sysbench tests from drizzle automation to kewpie. SysBench is a benchmark tool. It has benchmarks for various system parameters, which includes an oltp benchmark ( for testing database servers ). Following are the enhancements made into the Drizzle Trunk :

1. Added options for kewpie:

Options for dsn-string and mailing test reports have been included. The dsn-string specified via the –results-db-dsn option is used for connecting with a database. And if the user needs to send the report of a test to a mail id, it can be done easily be specifying the mail id with the –email-report-tgt option. You can have a deeper look over here.

2. Imported sysbench tests:

There are two variants of sysbench tests. One is a *readonly* test which shoots the database with SELECT queries and the other is *readwrite* test which includes INSERT, UPDATE, DELETE and DROP queries too.. These tests are imported to kewpie in an  ingenious way.. Instead of having the entire test code in each test case, there is a base class *sysbenchTestCase.py* in /tests/lib/util. Both the test cases, readonly and readwrite ( /tests/qp_tests/sysbench ), inherits this base class. Further, these test cases have only the configuration options in them, which is unique for both.. Peep here for more details.

3. Improved documetation:

Documentations for kewpie and sysbench were modified, and documentation for drizzletest commands was included. The latter one is not yet complete, however the api calls can be glanced at, in the documentation.. More information on each will be documented very soon..

So now I am a Drizzle Developer

The desire to contribute to opensource sprouted in me quite a long time ago. Browsing over the web and learning more about it, learning more from my friends, the eagerness reached a peak. It was at this point that I came to know about Google Summer of Code. To put it in a nutshell, Google Summer of Code ( GSoC ) is a program hosted by Google, where a lot of opensource organizations participate. Students can suggest their project ideas or develop existing project ideas, work on and develop their module(s) and receive stipend from Google.

I dived into Drizzle and chose a project on Improved Performance Regression Monitoring, which is related to Quality Assurance ( QA ). My project got selected and henceforth I need to start coding for my module. With the support from my mentor, Drizzle developers and a friend of mine, I learnt the basics. I also did some work on learning python, as I had to code my module in python and I was new to that language.

The three months time working with Google and Drizzle was really a groovy experience for me. I learnt a lot, enjoyed my work, and made friends and so..on. And at the close of the program, I had my modules completed, generated good documentations and all set. My code was merged into the Drizzle Trunk  ( the main source code ) . And that’s the most awaited moment!! I have become a Developer ( contributor ) at Drizzle!! A dream came true : )