Drizzle CI infrastructure getting matured

Its been quite long since I had a discussion about the new CI infrastructure for Drizzle. Well, the repository is getting better and better everyday. A lot of development has been made and its not far from completion. So I just thought I could give my readers a glimpse of what is going on while I finish off with the final few pieces of dev work in the interim.

So, in this discussion, we will be moving on to the stage where the repo had states for the following:

1. Installing Drizzle

2. Ensuring that the drizzled service is running

3. Setting up the SysBench tables for regression testing

4. Installing SysBench package

And below is the output when I run the magical Β “salt ‘*’ state.highstate”..

State: – pkg
Name: drizzle
Function: installed
Result: True
Comment: The following packages were installed/updated: drizzle.
Changes: drizzle: { new : 2011.03.13-0ubuntu5 old : }
State: – service
Name: drizzle
Function: running
Result: True
Comment: The service drizzle is already running
State: – file
Name: /home/shar/sysbench_db.sql
Function: managed
Result: True
Comment: File /home/shar/sysbench_db.sql is in the correct state
State: – cmd
Name: drizzle -uroot < /home/shar/sysbench_db.sql
Function: run
Result: True
Comment: Command “drizzle -uroot < /home/shar/sysbench_db.sql” run
Changes: pid: 15947
retcode: 0
State: – pkg
Name: sysbench
Function: installed
Result: True
Comment: The following packages were installed/updated: sysbench.
Changes: sysbench: { new : 0.4.12-1build2 old : }

Soon I will give my readers the flavor of the complete configuration of test nodes with the topping of automation πŸ™‚


A wave of memories

Today evening, after completing a considerable portion of my work I am currently working on, I just had a break. It was a breezy evening with sun shining mildly on my face and I was sipping my mug of coffee, when a wave of memorable memories splashed upon my mind. It was about the past one month which went on with campus interviews coming towards me on a conveyor line. Yeah, that was the reason for my blog to go on a hibernate. πŸ˜‰

The last month was completely filled with campus interviews. Being an undergraduate, and having no experience in facing official interviews, I was a kind of nervous and excited. I know I have been preparing well, and the preparation was like a good old friend beside me saying, I am there to lend you a helping hand. So I just had the thought that interviews would be based on some questions which tests how far we understand the concepts and how well we are able to think, solve problems, code, etc, etc. My interview went a step more than that.

What I really enjoyed about my interview was that, I never expected such questions, it was completely related with the practical work I have been doing as a passion / hobby, and more important is that I was able to answer those. πŸ˜€ Yeah, it revolved around my open source contributions, patches I have submitted, tools I have used, and so..on. I initially had the feel the my contributions bridged the gap between practical and theory stuffs, but it was much more. And the pinnacle is even higher I believe.

So my open sourcing world comprises of three things, Drizzle, SaltStack and Google Summer of Code. Having peeped into many organizations, it was finally Drizzle which marked the beginning. That was the entry point, shown by my dear friend Vijay Samuel. And then inside Drizzle I became friends with Patrick Crews, who finally became my mentor. πŸ™‚ They were the pillars with which I was able to get a balance and stabilize myself. And of course Drizzle, which provided the platform.

I loved this work and that is when Google Summer of Code 2012 sprang in. And double joy was that Drizzle was an accepted organization. It was their third year at GSoC. I started fixing bugs, submitted patches, proved my worth and finally got my proposal selected. This 4 months of program strengthened the bond between me and Drizzle. I was able to learn a lot lot lot of new things, interesting stuffs, etc and I didn’t want to leave this organization.

With many friendly people around, I felt the warmth given by Drizzle. I started contributing to Drizzle even after Google Summer of Code 2012, then had SaltStack in parallel and today, I am again into Google Summer of Code 2013, working for Drizzle and using SaltStack tools. The whole blend of Drizzle + SaltStack + Google Summer of Code. I sincerely thank my friend Vijay and my mentor Patrick and of course Drizzle organization for helping me bring out my potential, learn new technologies, getting hands on experience and getting a job offer as well. πŸ™‚

Get introduced with SaltStack

One tool that is gonna be used intensively in my work is the SaltStack. To introduce SaltStack to a newbie like me, it is a remote execution and configuration management tool. I know a newbie would find even that as weird. To throw some light on it, the tool is used to manage remote systems (called as minion) from the local machine (master, just another name). So one can execute commands, install packages, start a service, transfer files to and from, etc. This is just a few to mention. SaltStack provides much more. In this post, we will just look into two main portions, salt states and salt pillars, and get some introduction with salt modules and salt cloud.

First things first, we will start with salt states. Salt states is the configuration management side of SaltStack. It is used to set the remote system in the state which we want. Lets say, Β if I want a system to have MySQL installed and running, I can do that with state files. That is pretty much about what it does. Well, in reality one would have lots of such requirements, such as, having git installed, a repository cloned in, dependencies for compiling the branch to be installed, compile the code, etc. These states are written in any data serialization format with yaml being the default one, and the file has the extension .SLS.

So internally, salt treats the states mentioned in the .sls file as dictionary, list, integers or strings. So when we have a state defined for the minions, we just have to transfer these state files to the minions, which then configure themselves according to the *rules* specified in the state files. For this purpose, salt runs a simple file server, written in ZeroMQ, in the master through which the state files are served to the minions. So that explains the whole purpose of salt state and a few internals of how it works.

So now a question should arise, what if the state files deal with minion-specific data. A simple example would be, installing packages on redhat and ubuntu platforms. Both are a little different right? Atleast for the name of the package. So salt answers it with Salt pillars. Pillar is another interface provided by SaltStack, which is nothing but, data maintained about the minions by the minions. So one can use these pillar data in the state files, and the minions use the data specific to itself, when parsing the state file. Every minion has some default data, and we can also serve additional data to minion from the master by writing pillar files. These files also have the same construct as of state files, except for what we write inside.

This post focuses mainly on state and pillar interfaces, since those are the two components used intensively. Also, this is just a brief overview for newbies to get started. I hope this would make it easy for them to go through the official docs. So now touching base with salt modules and salt cloud, the modules are the remote execution part of SaltStack. There are modules already provided in the package, and we can write our own custom modules as well. Salt cloud is another tool which is a cloud provisioning tool. One can create VM nodes of specified configs and destroy them once its job is done.

Now that I have introduced this simple yet powerful tool to my readers, I hope they would have a firm hold of it when I script about my actual work. So lots more interesting stuffs coming soon. πŸ™‚

Parallelization and Automation of Drizzle CI infrastructure

This post is all about my work for Google Summer of Code 2013. Drizzle Continuous Integration uses Jenkins CI tool at present. Drizzle had dedicated servers which ran Jenkins server / master with a few slaves. However, the infrastructure did not support parallel running of tests / builds to a scalable extend. So the project aims in bringing parallelism with automation.This can be envisioned as follows.

Drizzle CI does builds using Jenkins. It tests for regression using the available test suites. To bring in parallelism, the CI infrastructure executes Β test jobs, which may include a build, a sysbench test, a randgen test, etc., simultaneously. A dedicated system / cloud node is configured for execution of each job in the job queue. Multiple tests + Multiple nodes = Parallelism.

Configuring these nodes is handled in a simple way. It is as simple as writing a config file of specified format, and then issuing a command which configures the nodes accordingly. Cloud nodes + Configuration files = Automated configuration.

To put it in a nut shell, the first work would be to automate the setting up of cloud nodes, and the later half would be to parallelize the test runner. In the following posts, I will be updating about the progress and write about the tools which are involved in the work, how to use them, etc. πŸ™‚

The start of another season

It has been a long time since I posted here. Being occupied with works apart from the normal routine, kept me far from blogging. Β So now has the time come to share something with my readers.Β A lot happened over the past 2 months, out of which Google Summer of Code is the noteworthy one. And yeah, my proposal for this season has got selected!! πŸ™‚

I am very much happy to work again with Drizzle and my mentor Patrick Crews. My project for 2013 again circles around Quality Assurance. The whole idea is to create a new Continuous Integration Infrastructure and revamp the test runner for parallelism. In short, Automation + Parallelization.

Lots of design works have been made over the past few weeks and the code base is taking its shape. In my next post, I ll be writing about the idea behind the project followed by weekly updates. And once things get done, I could write up a documentation, as it would help out the users of the new CI infrastructure. πŸ™‚