Drizzle says hello to SaltStack

This post is about my next work, Importing Drizzle database to Salt modules. This was suggested by my mentor Patrick Crews and I thank him for giving me this golden opportunity of extending Drizzle to SaltStack. This is the first of the series of posts regarding this project.

I have done some initial work and have successfully connected to Drizzle server. I ll first tell about the changes to be made in the master / minion config file. Before using Drizzle, we have to enter the configuration parameters namely host, user, password, port and schema in the suitable config file. ( /etc/salt/master for master config or /etc/salt/minion for minion config ). If we want to use Drizzle in a specific minion, we can add these parameters in that minion’s config file. Else, if we want to access Drizzle on all the minion, then add these parameters in the master’s config file. You can just append the respective file with:

drizzle.host: ‘127.0.0.1’

drizzle.user: ‘root’

drizzle.passwd: ”

drizzle.db: ‘drizzle’

drizzle.port: 4427

Once this is done, restart the master with /etc/init.d/salt-master restart or the minion with /etc/init.d/salt-minion restart. Now you can access the Drizzle database through remote execution 🙂

As an introductory work, I have written the method for retrieving the version of Drizzle server that is running on the minion. This can be used with the command option : salt ‘*’ drizzle.version.

In my system, I got the output as :

sharan@sharan:~$ sudo salt \* drizzle.version
sharan:
VERSION(): 7.1.33-stable

So Drizzle module will be available in full swing in the fore coming weeks.. Stay tuned.. 😀

Advertisements

Automated Performance Testing of Minions

This post hovers over the new module that is imported to the list of Salt modules : SysBench. To throw light on SysBench, it is a benchmark tool, primarily conceived to test MySQL based databases, and then extending its region to other system parameters such as CPU performance, Thread and Mutex implementations, File and Memory access, etc. SysBench is best known for its simplicity and user friendliness.

The requisite for this module comes from the fact that, there are situations  in which we need to analyze the performance of Minions right from the Master node. A very crisp example for this is the case of load balancing. Knowing the performance of each Minions ( which is definitely not uniform in all minions ) prior fractionating the task, would lead to better assignment of Minions, without pressuring them.

This could be done by just executing the SysBench command line options using the famous ‘cmd.run’, then why this module? This module automates the entire process. SysBench CLI options take in a lot of parameters and are thus cumbersome ( probably the only drawback with them ). The module makes this drawback transparent to the users. Now the users can just execute an ordinary salt ‘*’ sysbench.<test> from their Master node. Moreover, the module tests not just one case. It tests different number of cases, varying from one test to another.

For example, the CPU test checks for 4 different primer limit values namely 500, 1000, 2500 and 5000, automation playing its part here too. Some test cases are derived using the Orthogonal Array testing to cover the whole set of cases. Moreover, the test report are parsed in a very neat manner, facilitating the user to get a quick overview of the performance.

The various tests and their corresponding CLI options provided by this module are :

1. CPU ( cpu )

2. Threads ( threads )

3. Mutex ( mutex )

4. Memory (memory )

5. File I/O ( fileio )

So to use this module, just straight away type the following command salt ‘*’ sysbench.<CLI option>.

Cheers.. 🙂

Improved Performance Regression Monitoring

As a part of Google Summer of Code 2012, I worked on Improved Performance Regression Monitoring for DrizzleDB.. At the bottom, its a Quality Assurance and Automated Testing project.. Drizzle already has *Drizzle Automation* which deals with automating QA processes, code coverage analysis and benchmarking.. However due to the following issues, its not preferred much..

1. Its not in-tree

2. Its more complex

3. Its less user friendly

In order to nullify this, the kewpie tests came into effect. Kewpie is a test-runner for MySQL based databases. Its cleaner, simpler and very much effective. Moreover, its included in-tree.. The project’s intent is to import sysbench tests from drizzle automation to kewpie. SysBench is a benchmark tool. It has benchmarks for various system parameters, which includes an oltp benchmark ( for testing database servers ). Following are the enhancements made into the Drizzle Trunk :

1. Added options for kewpie:

Options for dsn-string and mailing test reports have been included. The dsn-string specified via the –results-db-dsn option is used for connecting with a database. And if the user needs to send the report of a test to a mail id, it can be done easily be specifying the mail id with the –email-report-tgt option. You can have a deeper look over here.

2. Imported sysbench tests:

There are two variants of sysbench tests. One is a *readonly* test which shoots the database with SELECT queries and the other is *readwrite* test which includes INSERT, UPDATE, DELETE and DROP queries too.. These tests are imported to kewpie in an  ingenious way.. Instead of having the entire test code in each test case, there is a base class *sysbenchTestCase.py* in /tests/lib/util. Both the test cases, readonly and readwrite ( /tests/qp_tests/sysbench ), inherits this base class. Further, these test cases have only the configuration options in them, which is unique for both.. Peep here for more details.

3. Improved documetation:

Documentations for kewpie and sysbench were modified, and documentation for drizzletest commands was included. The latter one is not yet complete, however the api calls can be glanced at, in the documentation.. More information on each will be documented very soon..

So now I am a Drizzle Developer

The desire to contribute to opensource sprouted in me quite a long time ago. Browsing over the web and learning more about it, learning more from my friends, the eagerness reached a peak. It was at this point that I came to know about Google Summer of Code. To put it in a nutshell, Google Summer of Code ( GSoC ) is a program hosted by Google, where a lot of opensource organizations participate. Students can suggest their project ideas or develop existing project ideas, work on and develop their module(s) and receive stipend from Google.

I dived into Drizzle and chose a project on Improved Performance Regression Monitoring, which is related to Quality Assurance ( QA ). My project got selected and henceforth I need to start coding for my module. With the support from my mentor, Drizzle developers and a friend of mine, I learnt the basics. I also did some work on learning python, as I had to code my module in python and I was new to that language.

The three months time working with Google and Drizzle was really a groovy experience for me. I learnt a lot, enjoyed my work, and made friends and so..on. And at the close of the program, I had my modules completed, generated good documentations and all set. My code was merged into the Drizzle Trunk  ( the main source code ) . And that’s the most awaited moment!! I have become a Developer ( contributor ) at Drizzle!! A dream came true : )

Coded for an Integrated Development Environment

To say more about it, it is my first project in my college career. It was done with my self interest. The IDE was coded in GTK+. Initially I had very little knowledge on creating it. I didn’t even know how an IDE works, ie. how it compiles the code, displays the compilation message and executes the program. Then I started to browse the web, and finally ended up with GTK+. First I tried creating some small windows with some buttons and simulating a simple dialog box. Once i became familiar with the language, I started coding for the IDE.

Since I was doing my project in Linux environment (Ubuntu), it was easy for me to make the system calls. I ended up compiling and executing the code by making the respective system calls. That’s something which I learnt new.

Finally,  completed my IDE in 2 months span. Though the application has minimal features, and merely no portability, I learnt quite a lot from this project. I learnt a new language, how to work on a project, how to timeline our period of work, etc. Indeed that was a kick start for me to get involved in projects.