Map Application Demystified

A few years ago, when I bought my first smart phone, one of the applications which I used more frequently was Google Maps. That was the time when I moved to a new city. And in the initial few days, wherever I go I had Google Maps open. After setting my destination, as I follow the route I keep wondering how this ‘Magical’ application works. But today, being a Developer at ThoughtWorks and having got opportunities to work on applications that use maps, I would say a map application is just a combination of a few moving parts which does the magic!

Though there are applications which show just a static image of map with little user interaction, a typical map application is one that allows users to interact with it. A good example is the app used for booking cabs. One can set his / her pickup point or choose the current location as the pickup point, pan the map, zoom in / out, see the cabs’ location (These are called Markers, which are nothing but images which mark points on the map), and after boarding the cab, see the route travelled (These are called Polylines, which are sequence of points).

Lets consider interactive map application. There are four primary players: User, who does the interactions; Device, which translates the user’s interactions into a language that the application can understand and shoulders some of the user’s task, say getting the current location; The Application itself; A Black Box, which provides the map images, addresses and lots more. We will get back to this Black Box in a short while. With this information, I will introduce the first two moving parts behind a map application. First one is the device itself. The device must be equipped enough to get user interactions and user location (using GPS or Network Provider). Second one is the piece of software (client-side library, used internally by the application) which does the heavy lifting work in presenting the maps, markers, polylines, etc. to the user.

In the process of looking into other moving parts, I shall demystify the black box mentioned earlier as well. There are so many data getting shown on a map application. First the map itself. There are even different images for the same location. One image show the geography, another shows transit information, another shows a dark image indicating its night time! And many many more. If you have noticed carefully, the entire map that’s being displayed is not a single image. Rather its made of many small images which together appears as a single image. These small images are called Tiles – Square shaped 256 x 256 pixel PNG images. And the component that provides these tiles is called the Tile Server. This is one of the little boxes within the black box.

Second, given an address the application correctly points out the location on the map. This process is called Geocoding – It basically means converting a human readable address to geographic coordinates. Third, the reverse process of Geocoding. A typical example is getting one’s current location. The GPS hardware in the device provides just the geographic coordinates. But the application shows the location in human readable address. This process is called Reverse Geocoding. And these are services (APIs) provided by a single or two different components. To keep it simple, lets have just tile server, Geocoding API and Reverse Geocoding API in the black box. However, there can be many other components and services involved depending on the complexity of the application.

So now that we know the various components involved, lets look at the tasks done by the client side library. As mentioned earlier, the client side library is the one that gels the user interactions and other components. Some of the work done by it are described below:

• Request the tile server to provide the tiles for the region which the user currently views.
• Handles zooming: Zoom level can be analogous to the height from which the user views a region. If the user zooms in, (s)he expects to get a closer look, and there by expects to see more details on the map. If the user zooms out, he / she probably wants a hawk eye view of the area rather than the intricate details. To support this, the tile server has tiles of different zoom levels. And the client library takes the responsibility of getting the tiles of appropriate zoom level.
• Allows panning: Whenever the user pans the map, the library calculates the missing tiles that are required to fill the view and requests those from the tile server.
• Provides the user the ability to add markers.
• Does the groundwork of converting a point from device (screen) coordinate system to geographic coordinate system and vice versa.

So the next time when you use any map application, I believe you wouldn’t be considering it to be a magic box.

Coffee Making: The Functional Way!

    Coffee making is an art! And there is a guy near my home who is good at that. He runs a coffee stall where he makes nice flavors of coffee with beautiful art works on those. And recently I just dropped in to the coffee shop while returing home from my work. As he was making a cup of coffee for me by pressing some combination of buttons on a coffee machine he had, I was just wondering the way people used to make coffee a few years back. They used to do that in a very imperative way!

cup makeCoffee (cup, coffeePowder, sweetness, canAddMilk, tempDegrees, creamFlavor) {
        Add coffeePowder amount of coffee powder in to cup
        Add sweetness amount of sugar in to cup.
        if (canAddMilk is yes)
            Fill the cup with milk
        else
            Fill the cup with water
        while(temperature of mixture is less than tempDegrees) {
            Heat the contents of cup
        }
        return cup
}

So there we go. The step-by-step procedure for making a cup of coffee. However, coffee vendors nowadays make coffee by the press of a few buttons.

cupWithCoffeePowder = addCoffeePowder(cup, quantity)
cupWithSugar = addSugar(cupWithCoffeePowder, quantity)
cupWithCoffee = fillCup(cupWithSugar, choiceOfMilkOrWater)
cupWithHotCoffee = heatMixture(cupWithCoffee, temperature)

And that’s it. Each function above is equivalent to a button on the coffee vending machine which the coffee vendor pressed.

And the above two ways of coffee making are synonymous to imperative programming vs functional programming. On careful analysis we can notice that, in the former method we specify how to make a cup of coffee. And in the latter we simply say what to do, i.e. addCoffeePowder, addSugar, fillCup, heatMixture (And the internal working of each step is handled by the machine – user has to just press the buttons and wait for his coffee). Just like how the coffee machine provides us with built in buttons and abstracts out what it does behind the scenes, a functional programming language also provides us with a basic set of functions with which we could accomplish our task or use those as building blocks for creating new functions.

And now I am gonna go and make a coffee for myself – in the functional way 😉

Drizzle CI infrastructure getting matured

Its been quite long since I had a discussion about the new CI infrastructure for Drizzle. Well, the repository is getting better and better everyday. A lot of development has been made and its not far from completion. So I just thought I could give my readers a glimpse of what is going on while I finish off with the final few pieces of dev work in the interim.

So, in this discussion, we will be moving on to the stage where the repo had states for the following:

1. Installing Drizzle

2. Ensuring that the drizzled service is running

3. Setting up the SysBench tables for regression testing

4. Installing SysBench package

And below is the output when I run the magical  “salt ‘*’ state.highstate”..

minion-ubuntu:
———-
State: – pkg
Name: drizzle
Function: installed
Result: True
Comment: The following packages were installed/updated: drizzle.
Changes: drizzle: { new : 2011.03.13-0ubuntu5 old : }
———-
State: – service
Name: drizzle
Function: running
Result: True
Comment: The service drizzle is already running
Changes:
———-
State: – file
Name: /home/shar/sysbench_db.sql
Function: managed
Result: True
Comment: File /home/shar/sysbench_db.sql is in the correct state
Changes:
———-
State: – cmd
Name: drizzle -uroot < /home/shar/sysbench_db.sql
Function: run
Result: True
Comment: Command “drizzle -uroot < /home/shar/sysbench_db.sql” run
Changes: pid: 15947
retcode: 0
stderr:
stdout:
———-
State: – pkg
Name: sysbench
Function: installed
Result: True
Comment: The following packages were installed/updated: sysbench.
Changes: sysbench: { new : 0.4.12-1build2 old : }

Soon I will give my readers the flavor of the complete configuration of test nodes with the topping of automation 🙂

A wave of memories

Today evening, after completing a considerable portion of my work I am currently working on, I just had a break. It was a breezy evening with sun shining mildly on my face and I was sipping my mug of coffee, when a wave of memorable memories splashed upon my mind. It was about the past one month which went on with campus interviews coming towards me on a conveyor line. Yeah, that was the reason for my blog to go on a hibernate. 😉

The last month was completely filled with campus interviews. Being an undergraduate, and having no experience in facing official interviews, I was a kind of nervous and excited. I know I have been preparing well, and the preparation was like a good old friend beside me saying, I am there to lend you a helping hand. So I just had the thought that interviews would be based on some questions which tests how far we understand the concepts and how well we are able to think, solve problems, code, etc, etc. My interview went a step more than that.

What I really enjoyed about my interview was that, I never expected such questions, it was completely related with the practical work I have been doing as a passion / hobby, and more important is that I was able to answer those. 😀 Yeah, it revolved around my open source contributions, patches I have submitted, tools I have used, and so..on. I initially had the feel the my contributions bridged the gap between practical and theory stuffs, but it was much more. And the pinnacle is even higher I believe.

So my open sourcing world comprises of three things, Drizzle, SaltStack and Google Summer of Code. Having peeped into many organizations, it was finally Drizzle which marked the beginning. That was the entry point, shown by my dear friend Vijay Samuel. And then inside Drizzle I became friends with Patrick Crews, who finally became my mentor. 🙂 They were the pillars with which I was able to get a balance and stabilize myself. And of course Drizzle, which provided the platform.

I loved this work and that is when Google Summer of Code 2012 sprang in. And double joy was that Drizzle was an accepted organization. It was their third year at GSoC. I started fixing bugs, submitted patches, proved my worth and finally got my proposal selected. This 4 months of program strengthened the bond between me and Drizzle. I was able to learn a lot lot lot of new things, interesting stuffs, etc and I didn’t want to leave this organization.

With many friendly people around, I felt the warmth given by Drizzle. I started contributing to Drizzle even after Google Summer of Code 2012, then had SaltStack in parallel and today, I am again into Google Summer of Code 2013, working for Drizzle and using SaltStack tools. The whole blend of Drizzle + SaltStack + Google Summer of Code. I sincerely thank my friend Vijay and my mentor Patrick and of course Drizzle organization for helping me bring out my potential, learn new technologies, getting hands on experience and getting a job offer as well. 🙂

Get introduced with SaltStack

One tool that is gonna be used intensively in my work is the SaltStack. To introduce SaltStack to a newbie like me, it is a remote execution and configuration management tool. I know a newbie would find even that as weird. To throw some light on it, the tool is used to manage remote systems (called as minion) from the local machine (master, just another name). So one can execute commands, install packages, start a service, transfer files to and from, etc. This is just a few to mention. SaltStack provides much more. In this post, we will just look into two main portions, salt states and salt pillars, and get some introduction with salt modules and salt cloud.

First things first, we will start with salt states. Salt states is the configuration management side of SaltStack. It is used to set the remote system in the state which we want. Lets say,  if I want a system to have MySQL installed and running, I can do that with state files. That is pretty much about what it does. Well, in reality one would have lots of such requirements, such as, having git installed, a repository cloned in, dependencies for compiling the branch to be installed, compile the code, etc. These states are written in any data serialization format with yaml being the default one, and the file has the extension .SLS.

So internally, salt treats the states mentioned in the .sls file as dictionary, list, integers or strings. So when we have a state defined for the minions, we just have to transfer these state files to the minions, which then configure themselves according to the *rules* specified in the state files. For this purpose, salt runs a simple file server, written in ZeroMQ, in the master through which the state files are served to the minions. So that explains the whole purpose of salt state and a few internals of how it works.

So now a question should arise, what if the state files deal with minion-specific data. A simple example would be, installing packages on redhat and ubuntu platforms. Both are a little different right? Atleast for the name of the package. So salt answers it with Salt pillars. Pillar is another interface provided by SaltStack, which is nothing but, data maintained about the minions by the minions. So one can use these pillar data in the state files, and the minions use the data specific to itself, when parsing the state file. Every minion has some default data, and we can also serve additional data to minion from the master by writing pillar files. These files also have the same construct as of state files, except for what we write inside.

This post focuses mainly on state and pillar interfaces, since those are the two components used intensively. Also, this is just a brief overview for newbies to get started. I hope this would make it easy for them to go through the official docs. So now touching base with salt modules and salt cloud, the modules are the remote execution part of SaltStack. There are modules already provided in the package, and we can write our own custom modules as well. Salt cloud is another tool which is a cloud provisioning tool. One can create VM nodes of specified configs and destroy them once its job is done.

Now that I have introduced this simple yet powerful tool to my readers, I hope they would have a firm hold of it when I script about my actual work. So lots more interesting stuffs coming soon. 🙂

Parallelization and Automation of Drizzle CI infrastructure

This post is all about my work for Google Summer of Code 2013. Drizzle Continuous Integration uses Jenkins CI tool at present. Drizzle had dedicated servers which ran Jenkins server / master with a few slaves. However, the infrastructure did not support parallel running of tests / builds to a scalable extend. So the project aims in bringing parallelism with automation.This can be envisioned as follows.

Drizzle CI does builds using Jenkins. It tests for regression using the available test suites. To bring in parallelism, the CI infrastructure executes  test jobs, which may include a build, a sysbench test, a randgen test, etc., simultaneously. A dedicated system / cloud node is configured for execution of each job in the job queue. Multiple tests + Multiple nodes = Parallelism.

Configuring these nodes is handled in a simple way. It is as simple as writing a config file of specified format, and then issuing a command which configures the nodes accordingly. Cloud nodes + Configuration files = Automated configuration.

To put it in a nut shell, the first work would be to automate the setting up of cloud nodes, and the later half would be to parallelize the test runner. In the following posts, I will be updating about the progress and write about the tools which are involved in the work, how to use them, etc. 🙂

The start of another season

It has been a long time since I posted here. Being occupied with works apart from the normal routine, kept me far from blogging.  So now has the time come to share something with my readers. A lot happened over the past 2 months, out of which Google Summer of Code is the noteworthy one. And yeah, my proposal for this season has got selected!! 🙂

I am very much happy to work again with Drizzle and my mentor Patrick Crews. My project for 2013 again circles around Quality Assurance. The whole idea is to create a new Continuous Integration Infrastructure and revamp the test runner for parallelism. In short, Automation + Parallelization.

Lots of design works have been made over the past few weeks and the code base is taking its shape. In my next post, I ll be writing about the idea behind the project followed by weekly updates. And once things get done, I could write up a documentation, as it would help out the users of the new CI infrastructure. 🙂

Drizzle says hello to SaltStack

This post is about my next work, Importing Drizzle database to Salt modules. This was suggested by my mentor Patrick Crews and I thank him for giving me this golden opportunity of extending Drizzle to SaltStack. This is the first of the series of posts regarding this project.

I have done some initial work and have successfully connected to Drizzle server. I ll first tell about the changes to be made in the master / minion config file. Before using Drizzle, we have to enter the configuration parameters namely host, user, password, port and schema in the suitable config file. ( /etc/salt/master for master config or /etc/salt/minion for minion config ). If we want to use Drizzle in a specific minion, we can add these parameters in that minion’s config file. Else, if we want to access Drizzle on all the minion, then add these parameters in the master’s config file. You can just append the respective file with:

drizzle.host: ‘127.0.0.1’

drizzle.user: ‘root’

drizzle.passwd: ”

drizzle.db: ‘drizzle’

drizzle.port: 4427

Once this is done, restart the master with /etc/init.d/salt-master restart or the minion with /etc/init.d/salt-minion restart. Now you can access the Drizzle database through remote execution 🙂

As an introductory work, I have written the method for retrieving the version of Drizzle server that is running on the minion. This can be used with the command option : salt ‘*’ drizzle.version.

In my system, I got the output as :

sharan@sharan:~$ sudo salt \* drizzle.version
sharan:
VERSION(): 7.1.33-stable

So Drizzle module will be available in full swing in the fore coming weeks.. Stay tuned.. 😀

Automated Performance Testing of Minions

This post hovers over the new module that is imported to the list of Salt modules : SysBench. To throw light on SysBench, it is a benchmark tool, primarily conceived to test MySQL based databases, and then extending its region to other system parameters such as CPU performance, Thread and Mutex implementations, File and Memory access, etc. SysBench is best known for its simplicity and user friendliness.

The requisite for this module comes from the fact that, there are situations  in which we need to analyze the performance of Minions right from the Master node. A very crisp example for this is the case of load balancing. Knowing the performance of each Minions ( which is definitely not uniform in all minions ) prior fractionating the task, would lead to better assignment of Minions, without pressuring them.

This could be done by just executing the SysBench command line options using the famous ‘cmd.run’, then why this module? This module automates the entire process. SysBench CLI options take in a lot of parameters and are thus cumbersome ( probably the only drawback with them ). The module makes this drawback transparent to the users. Now the users can just execute an ordinary salt ‘*’ sysbench.<test> from their Master node. Moreover, the module tests not just one case. It tests different number of cases, varying from one test to another.

For example, the CPU test checks for 4 different primer limit values namely 500, 1000, 2500 and 5000, automation playing its part here too. Some test cases are derived using the Orthogonal Array testing to cover the whole set of cases. Moreover, the test report are parsed in a very neat manner, facilitating the user to get a quick overview of the performance.

The various tests and their corresponding CLI options provided by this module are :

1. CPU ( cpu )

2. Threads ( threads )

3. Mutex ( mutex )

4. Memory (memory )

5. File I/O ( fileio )

So to use this module, just straight away type the following command salt ‘*’ sysbench.<CLI option>.

Cheers.. 🙂

Improved Performance Regression Monitoring

As a part of Google Summer of Code 2012, I worked on Improved Performance Regression Monitoring for DrizzleDB.. At the bottom, its a Quality Assurance and Automated Testing project.. Drizzle already has *Drizzle Automation* which deals with automating QA processes, code coverage analysis and benchmarking.. However due to the following issues, its not preferred much..

1. Its not in-tree

2. Its more complex

3. Its less user friendly

In order to nullify this, the kewpie tests came into effect. Kewpie is a test-runner for MySQL based databases. Its cleaner, simpler and very much effective. Moreover, its included in-tree.. The project’s intent is to import sysbench tests from drizzle automation to kewpie. SysBench is a benchmark tool. It has benchmarks for various system parameters, which includes an oltp benchmark ( for testing database servers ). Following are the enhancements made into the Drizzle Trunk :

1. Added options for kewpie:

Options for dsn-string and mailing test reports have been included. The dsn-string specified via the –results-db-dsn option is used for connecting with a database. And if the user needs to send the report of a test to a mail id, it can be done easily be specifying the mail id with the –email-report-tgt option. You can have a deeper look over here.

2. Imported sysbench tests:

There are two variants of sysbench tests. One is a *readonly* test which shoots the database with SELECT queries and the other is *readwrite* test which includes INSERT, UPDATE, DELETE and DROP queries too.. These tests are imported to kewpie in an  ingenious way.. Instead of having the entire test code in each test case, there is a base class *sysbenchTestCase.py* in /tests/lib/util. Both the test cases, readonly and readwrite ( /tests/qp_tests/sysbench ), inherits this base class. Further, these test cases have only the configuration options in them, which is unique for both.. Peep here for more details.

3. Improved documetation:

Documentations for kewpie and sysbench were modified, and documentation for drizzletest commands was included. The latter one is not yet complete, however the api calls can be glanced at, in the documentation.. More information on each will be documented very soon..