Drizzle CI infrastructure getting matured

Its been quite long since I had a discussion about the new CI infrastructure for Drizzle. Well, the repository is getting better and better everyday. A lot of development has been made and its not far from completion. So I just thought I could give my readers a glimpse of what is going on while I finish off with the final few pieces of dev work in the interim.

So, in this discussion, we will be moving on to the stage where the repo had states for the following:

1. Installing Drizzle

2. Ensuring that the drizzled service is running

3. Setting up the SysBench tables for regression testing

4. Installing SysBench package

And below is the output when I run the magical ย “salt ‘*’ state.highstate”..

State: – pkg
Name: drizzle
Function: installed
Result: True
Comment: The following packages were installed/updated: drizzle.
Changes: drizzle: { new : 2011.03.13-0ubuntu5 old : }
State: – service
Name: drizzle
Function: running
Result: True
Comment: The service drizzle is already running
State: – file
Name: /home/shar/sysbench_db.sql
Function: managed
Result: True
Comment: File /home/shar/sysbench_db.sql is in the correct state
State: – cmd
Name: drizzle -uroot < /home/shar/sysbench_db.sql
Function: run
Result: True
Comment: Command “drizzle -uroot < /home/shar/sysbench_db.sql” run
Changes: pid: 15947
retcode: 0
State: – pkg
Name: sysbench
Function: installed
Result: True
Comment: The following packages were installed/updated: sysbench.
Changes: sysbench: { new : 0.4.12-1build2 old : }

Soon I will give my readers the flavor of the complete configuration of test nodes with the topping of automation ๐Ÿ™‚


Get introduced with SaltStack

One tool that is gonna be used intensively in my work is the SaltStack. To introduce SaltStack to a newbie like me, it is a remote execution and configuration management tool. I know a newbie would find even that as weird. To throw some light on it, the tool is used to manage remote systems (called as minion) from the local machine (master, just another name). So one can execute commands, install packages, start a service, transfer files to and from, etc. This is just a few to mention. SaltStack provides much more. In this post, we will just look into two main portions, salt states and salt pillars, and get some introduction with salt modules and salt cloud.

First things first, we will start with salt states. Salt states is the configuration management side of SaltStack. It is used to set the remote system in the state which we want. Lets say, ย if I want a system to have MySQL installed and running, I can do that with state files. That is pretty much about what it does. Well, in reality one would have lots of such requirements, such as, having git installed, a repository cloned in, dependencies for compiling the branch to be installed, compile the code, etc. These states are written in any data serialization format with yaml being the default one, and the file has the extension .SLS.

So internally, salt treats the states mentioned in the .sls file as dictionary, list, integers or strings. So when we have a state defined for the minions, we just have to transfer these state files to the minions, which then configure themselves according to the *rules* specified in the state files. For this purpose, salt runs a simple file server, written in ZeroMQ, in the master through which the state files are served to the minions. So that explains the whole purpose of salt state and a few internals of how it works.

So now a question should arise, what if the state files deal with minion-specific data. A simple example would be, installing packages on redhat and ubuntu platforms. Both are a little different right? Atleast for the name of the package. So salt answers it with Salt pillars. Pillar is another interface provided by SaltStack, which is nothing but, data maintained about the minions by the minions. So one can use these pillar data in the state files, and the minions use the data specific to itself, when parsing the state file. Every minion has some default data, and we can also serve additional data to minion from the master by writing pillar files. These files also have the same construct as of state files, except for what we write inside.

This post focuses mainly on state and pillar interfaces, since those are the two components used intensively. Also, this is just a brief overview for newbies to get started. I hope this would make it easy for them to go through the official docs. So now touching base with salt modules and salt cloud, the modules are the remote execution part of SaltStack. There are modules already provided in the package, and we can write our own custom modules as well. Salt cloud is another tool which is a cloud provisioning tool. One can create VM nodes of specified configs and destroy them once its job is done.

Now that I have introduced this simple yet powerful tool to my readers, I hope they would have a firm hold of it when I script about my actual work. So lots more interesting stuffs coming soon. ๐Ÿ™‚

Drizzle says hello to SaltStack

This post is about my next work, Importing Drizzle database to Salt modules. This was suggested by my mentor Patrick Crewsย and I thank him for giving me this golden opportunity of extending Drizzle to SaltStack. This is the first of the series of posts regarding this project.

I have done some initial work and have successfully connected to Drizzle server. I ll first tell about the changes to be made in the master / minion config file. Before using Drizzle, we have to enter the configuration parameters namely host, user, password, port and schema in the suitable config file. ( /etc/salt/master for master config or /etc/salt/minion for minion config ). If we want to use Drizzle in a specific minion, we can add these parameters in that minion’s config file. Else, if we want to access Drizzle on all the minion, then add these parameters in the master’s config file. You can just append the respective file with:

drizzle.host: ‘’

drizzle.user: ‘root’

drizzle.passwd: ”

drizzle.db: ‘drizzle’

drizzle.port: 4427

Once this is done, restart the master with /etc/init.d/salt-master restart or the minion with /etc/init.d/salt-minion restart. Now you can access the Drizzle database through remote execution ๐Ÿ™‚

As an introductory work, I have written the method for retrieving the version of Drizzle server that is running on the minion. This can be used with the command option : salt ‘*’ drizzle.version.

In my system, I got the output as :

sharan@sharan:~$ sudo salt \* drizzle.version
VERSION(): 7.1.33-stable

So Drizzle module will be available in full swing in the fore coming weeks.. Stay tuned.. ๐Ÿ˜€

Automated Performance Testing of Minions

This post hovers over the new module that is imported to the list of Salt modules : SysBench. To throw light on SysBench, it is a benchmark tool, primarily conceived to test MySQL based databases, and then extending its region to other system parameters such as CPU performance, Thread and Mutex implementations, File and Memory access, etc. SysBench is best known for its simplicity and user friendliness.

The requisite for this module comes from the fact that, there are situations ย in which we need to analyze the performance of Minions right from the Master node. A very crisp example for this is the case of load balancing. Knowing the performance of each Minions ( which is definitely not uniform in all minions ) prior fractionating the task, would lead to better assignment of Minions, without pressuring them.

This could be done by just executing the SysBench command line options using the famous ‘cmd.run’, then why this module? This module automates the entire process. SysBench CLI options take in a lot of parameters and are thus cumbersome ( probably the only drawback with them ). The module makes this drawback transparent to the users. Now the users can just execute an ordinary salt ‘*’ sysbench.<test> from their Master node. Moreover, the module tests not just one case. It tests different number of cases, varying from one test to another.

For example, the CPU test checks for 4 different primer limit values namely 500, 1000, 2500 and 5000, automation playing its part here too. Some test cases are derived using the Orthogonal Array testing to cover the whole set of cases. Moreover, the test report are parsed in a very neat manner, facilitating the user to get a quick overview of the performance.

The various tests and their corresponding CLI options provided by this module are :

1. CPU ( cpu )

2. Threads ( threads )

3. Mutex ( mutex )

4. Memory (memory )

5. File I/O ( fileio )

So to use this module, just straight away type the following command salt ‘*’ sysbench.<CLI option>.

Cheers.. ๐Ÿ™‚