Thursday, August 17, 2017

How to get plain text inside any XML tag using python

While coding in python, if you need to get plaintext inside a particular tag in XML, you can make use of text iterator provided by ElementTree. You can call itertext() on any XML element, and it returns an iterator which will walk you through text of each tag inside your current element.

The code is simple and self explanatory. You iterate over the text iterator, concatinating text of each element inside given element to some string variable. At the end of the loop, you return the combined text. 

Sunday, August 13, 2017

Developing Jenkins plugin

This blog post is to help you with resources to get you started with Jenkins plugin development. This is NOT a plugin development tutorial. I do not wish to repeat any of the contents which are already available online. This blog post should be seen as a pointer towards those various contents which could help you develop your first Jenkins plugin. 

General idea

As part of my integrations development duties, I needed to work on Jenkins plugin development. The idea was simple: the plugin should work as post-build step, and pull some report from a cloud based system. This plugin is supposed to be used in CD/CI pipeline.

What you need to know to develop Jenkins plugin?

  1. Maven.
  2. Setting up your development environment.
  3. Plugin project structure.
  4. Java.
  5. Groovy script.
  6. Extension points to use in your plugin.
Since I had no prior experience with developing any plugin with Jenkins, I had to start right from the scratch. First obvious step was to start reading and trying out some code. This Jekins wiki page is really helpful getting started. It shows how to setup your development environment, how to test run your code, how to debug, and how to package your plugin.

Its better if you already know how to use Maven, but that is not any hard and fast requirement. Even if you haven't used it earlier, you can still work with that. With help of Maven archetype, you would be generating your starter code. Instructions for that are also given in that Jenkins wiki page. It generates a simple Builder step. 

Point # 6 above is Extension points. There are several classes exposed by Jenkins - called as extension points - which you will be extending or implementing to provide your plugin functionality. For example, to do something after build is completed, you need to extend Notifier class. Whichever class you annotate with @Extension annotation, Jenkins will load at appropriate time - depending on which class you are extending or implementing. 

If you are going to support only freestyle project, you would mostly need to implement SimpleBuildStep class.

Supporting pipelines

As mentioned on Jenkins pipeline page, its a suite you use for continuous integration/delivery. If you need to support pipeline project, SimpleBuildStep may not be the correct choice for you. AFAIK, you need to implement AbstractStepImpl to provide same functionality as a pipeline step. However, this class is marked deprecated and anytime down the line it could be discontinued.

Having said that, I did not find any other alternative to use for pipeline which is as simple as SimpleBuildStep. But, this blog post was very helpful in refactoring the SimpleBuildStep-based code to support pipeline.

Tuesday, June 27, 2017

Somebody found webmon useful!

It gives me great pleasure to let you all know that, somebody have found my Webmon useful! Yes, and they have forked it on github and are using it in their project - of course, with due attribution.

I used it to monitor job portals and career pages of companies of my interest. I maintained it until Feb 2014. After that, it became a dormant project. 

One fine day, I was revisiting my github profile - which I do every week, as such - and found that somebody had forked out my webmon. I quickly opened their repository in other tab and started going through it. As a developer, it was so joyful for me, that my pet work is actually of help to somebody. It just made my entire day! 

So, what is Webmon?

Webmon is a free software, written in PHP. I created it way back in 2013, with following objectives: 
  1. Detect whether listed webpage have any change in contents.
  2. If change is detected, calculate the difference, and output what have changed.
  3. Detect both - positive and negative changes.
I distributed it in the hope that it will be useful, but without any warrenty; without even the implied warranty of merchantability or fitness for a particular purpose. Its still available to everyone. You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

I do not have any plan as of now to actively contribute to it and take it further, but I would really like if some other developer starts contributing to it, and make it a truly open source project.

Thursday, October 20, 2016

How to setup Splunk Search Head Cluster?

If you already know what is Splunk, and are interested in setting up your own Search Head Cluster, continue reading.

For this, the environment will be:

  • 1 Deployer – sends apps/configurations to the search heads
  • 3 Search Heads – for the SHC
  • 1 Indexer – the “search peer” that the SHC will dispatch jobs to
  • 1 Forwarder – for testing data input from the TA/App into the indexer


Sizing wise you could make them all VMs. Something reasonably small could be as follows for each system – with the Deployer and Forwarder being much smaller.

  • 4 cores
  • 8GB RAM
  • 60GB disk


Once you have all your machines ready, follow steps given below. My steps consider linux-based setups, but you can do it on any other Splunk-supported OS. Make sure to change paths accordingly.

0) If you haven't done already, change the default admin password 'changeme' to something else. Any of the SHC setup commands will not work properly if your admin password is the default one.

1) on Deployer:
in /opt/splunk/etc/system/local/server.conf add following line under [general] stanza, write following line.
pass4SymmKey = yourKey

Replace yourKey with your plaintext key. Do not worry, Splunk will definitely encrypt it later.

2) Initialize all search head clusters:
On each SH, run these commands -
/opt/splunk/bin/splunk init shcluster-config -auth admin:splunk -mgmt_uri <mgmt uri of this setup> -replication_port <any unusual port like 20000> -conf_deploy_fetch_url <mgmt uri of deployer> -secret yourKey
/opt/splunk/bin/splunk restart

Now, at this point, each SH where you ran above commands knows who is deployer for them and the key to authenticate with.

3) Bring up cluster captain:
This step is required only for SH cluster. You can omit this step if you are not setting up SHC.
/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "<comma-separated list of mgmt uri of all search heads, including designated captain>" -auth <this setup's username:password>

4) Check search head cluster status:
To check the overall status of your search head cluster, run this command from any of the members:
/opt/splunk/bin/splunk show shcluster-status -auth <this setup's username:password>

5) Deploy the bundle (app):
/opt/splunk/bin/splunk apply shcluster-bundle -target <mgmt uri of SH where you want to deploy app> -auth <SH's username:password>

Your Search Head Cluster setup should be ready and operational now.

What is Splunk?

Its like Google for logs!

When you need to debug some application or system, what do you do? You go through log files. They tell you (almost) everything about what it was trying to do, and what happened. But what do you do when you need to debug a distributed or cloud based or microservices based system? Do you go to each and every machine/app and try to correlate that information with logs of another machine/app? Do you always have a design where all log lines from all those machines or apps are written to single log? Usually not.

That's where Splunk is really useful. Its a log processing and analysis product, which stores all your logs in indexed manner, and provides very fast searching ability.

Events and Indexes

Each entry that gets stored in Splunk is called an 'event'. And, the logical place where a particular event is stored in is called as 'index'. So, when searching, you basically query some index(es) to find some events.
Each indexed event has 4 fields associated with it: time, sourcetype, source and host. The time field indicates when that event happened. Sourcetype identifies data structure of that event, where as source identifies where that event happened. The host is the machine where this event generated. You can search your data using these fields.

Apart from that, Splunk extracts most of the fields from your data, which can also be used while searching the data.

The Splunk setup which indexes the data is called as indexer.

They call it SPL

SPL stands for Search Processing Language. Its like a query you enter to grab some data out of Splunk index(es).

Your search can be a simple term (e.g. a username) to see how frequently it appears in log, or it could be a complex one (e.g. a particular source, particular event, containing this or that, happened between 1am to 4:40am).
There are endless possibilities which you can search using Splunk. We use Splunk in my company to analyze application logs and find out which exceptions occur more often and on which days it reaches the peak point.
There's a lot to talk about this, and the best way to know about SPL is to go through Splunk's own documentation on SPL.

Apps, Add-ons and Data Sources

OK. We have got at least some idea about what it does. But tell me how it gets my log data?!
Well, there's no big magic in that. There apps, add-ons and other data import sources supported in Splunk using which you bring in the data. You can import those logs files - be it syslog, csv or json. You can also develop splunk apps or add-ons to make API calls to outer world, and produce data understandable to Splunk! Splunk is capable of consuming data outputed on stdout/stderr!

There's no database

That brings us to the next point: Where does that data go eventually? Let me tell you that there is no database to manage. Splunk stores data directly in the file system. Because of that, in fact the Splunk setup is quite fast.

Scalability is easy

If a single Splunk server is not enough you can simply add another one. The data can be distributed among multiple Splunk setups. You can have the same forwarder forwarding your data to multiple indexers, or you can have multiple forwarders feeding data to single indexer or a combination of both these cases. Not only that, you can also distribute your search operations using multiple 'search heads'. There are these two interesting deployment scenarios known as Indexer Cluster and Search Head Cluster, and worth exploring when scalability is important.

Wednesday, October 5, 2016

Expand and shrink IPv4 range

Few months ago, I had written this blog post about chunking IPv4 range into multiple sub-ranges. Soon thereafter arouse requirement to have something to expand or shrink given IPv4 range. And, I came up with this code.

There are two functions written - one expands given IPv4 range and other shrinks it. Let me explain each of them one by one.

The expand_range() function

The input IPv4 range could be a comma-separated list of IPv4 addresses or a proper range (e.g. 10.10.10.10-10.10.10.155) or a mix of both. Objective of this function is to provide a list of ALL IPv4 addresses that are part of given range.

This function starts with exploding the input based on comma. For each element in resulting array, it checks whether its a single IPv4 address or a range having a dash (-) character.

If it contains a dash, it further explodes it to get start and end IPv4 addresses, and converts them to long using ip2long(). Then, it simply runs a for loop to generate all the long values in between them and adds them into output array.

If its a single IPv4 address, that is added as it is to output array.

Finally, it sorts the output array, removes duplicates using array_unique(), and converts each element mach to IPv4 address using long2ip().

Based on optional second argument, it either returns array or a string representation of expanded IPv4 range.


The shrink_range() function

Here, the input IPv4 range could be either a string representation or an array, similar to one output by expand_range() function. Objective of this function is to shorten the given IPv4 range. So, if input is 10.10.10.10,10.10.10.11,10.10.10.12,10.10.10.13 then it should shorten it to 10.10.10.10-10.10.10.13.

The function starts with creating an array out of given IPv4 addresses. If first argument itself is array, it just copies it. And it then takes count() of array elements so that it can loop over them.

In the loop, it keeps on checking IPv4 address and current index as well as next index. It converts both of them into log and calculates difference between them by subtracting current index's long value from next index's long value.

If that difference is 1, it means the next IPv4 address is in sequence with current IPv4 address. And, that also means that we are getting into a range which can be shortened. The function then sets a flag to remember that, and starts building string for this short range.

Else, its either a standalone IPv4 address OR we have possibly reached end of short range. So it checks if the flag is still ON. If yes, it ends the short range, and copies the short range string into output array. If we aren't preparing any short range, it simply adds current IPv4 into output array.

Finally, based on optional second argument, it either returns array or string representation of shrunken IPv4 range.

Monday, May 23, 2016

From where I cloned my local repo?

If you want to determine the URL from where you cloned your local repository, use this command.

git config --get remote.origin.url

Friday, January 15, 2016

How to read attachment in servicenow scoped application

While working on a servicenow application, I came across a scenario where I need to read the attachment contents (XML in my case) and parse them. Quick google search leads to code snippets using GlideSysAttachment.getBytes() which is no longer available to scoped applications, since Fuji patch 9. After trying for a while, I could read the attachment. Here's the code I used for this.

The sys_attachment table contains the actual attachment, along with table name and table sys id to which this attachment belongs to. Query sys_attachment table, providing table name (optional, because table_sys_id itself is supposed to be unique) and table_sys_id. Then, while iterating over result set, call GlideSysAttachment.getContent(), passing the sys_attachment glide record as argument.

Monday, December 14, 2015

Chunk IPv4 range into multiple sub-chunks

One of my non-programmer-but-in-IT friends came to me last week, asking help to chunk a range of IPv4 addresses into N small chunks. And, here's the PHP code I came up with within 10 mins.

What it simply does is, explode the input range, and convert start and end IP address into long, using ip2long(). Then, using range() expand it into a whole list. Once that is done, then decide number of elements that should go into each chunk - remember, the optional argument to this function tells how many chunks are to be prepared. the formula is there on code line number 11.

Once you have this number, call array_chunk() to create chunks. Finally, iterate over them, get start and end IP of each of them (which are still long numbers), convert them back to IPv4 address using long2ip() and create the range string for them.

Do you find it useful? Copy it! Have any suggestion, let me know in the comment!!

Wednesday, May 13, 2015

Ordinal Number

Here is a PHP function to get the ordinal number. Pass it a number, and it will return a textual number suffixed with either of "st" (like 1st), "nd" (like 2nd), "rd" (like 3rd), "th" (like 4th) etc.

Grep - before and after

The grep is a command line utility for searching plain-text data sets for lines matching the given regular expression. And those using Linux (or some or other flavor of this OS), must have used this command. Here are two useful options of this command: -A for After and -B for Before. Both take a number as their value.

As their names suggest, -A gives you matching line and N lines after the matching line, similarly, -B gives you matching line and N lines before the matching line.

You can also combine both these options into single command. And, if your -A and -B values are same, then you can simply use -C option. C stands for Count.

Wednesday, May 6, 2015

PHP Blunder-2

Here is another 'oops!' moment for PHP. Do you remember the rule of transitivity? A related to B, and A related to C, implies that B is related to C. Right? Now apply this very understanding to the code below.
Now, foo equals true, and foo equals zero. So, according to this rule of transitivity, true equals zero?!

Wednesday, April 29, 2015

PHP Blunder - 1

PHP is one of those most popular web programming languages, but it do have some serious mistakes. Here's one of them.

PHP have a predefined value 'null'. And, it can be compared with numeric values. As anybody else would, you would expect null to have a constant value, right? But that's where PHP have a problem. Look at the code below, and execute it on your system if you want to check.

The null == 0 evaluates to true. So does null < -1. How can some predefined value be equal to 0 and lesser than -1 at the same time?!

Tuesday, April 28, 2015

Graceful Degradation and Progressive Enhancement

These two terms are of pretty much importance when it comes to developing web apps or sites. Both, graceful degradation and progressive enhancement consider how the web site or app works when the browsers and devices vary.But the main difference is between where they focus.

Graceful degradation is the practice of building your web functionality such that it provides a certain level of rich user experience on modern browsers and devices, but it will also degrade gracefully to a working lower level on older browsers and devices. This working lower level is of course not a nice-to-have for your app/site users, but it still provides basic functionality so that things won't break for them.

Progressive enhancement is quite similar, but it goes the other way around. You start by establishing a basic level of user experience that all browsers/devices will be able to provide when rendering your web page, but you also build more advanced functionality that will be automatically be available to browsers/devices that can use it.

In simple and short words, graceful degradation is starting with higher experience quality and trying to fix it for lesser experience; whereas, progressive enhancement is starting with basic, working functionality and allowing for extension for future environments.

Degrading gracefully means looking back where as enhancing progressively means looking forward while standing on firm ground.

Example of Graceful Degradation v/s Progressive Enhancement
Let's take a simple example of "Print This" link. As such, links that allow users to print the page are useless because the user can do the same by using browser's 'Print' button. Perhaps, letting users trigger this from within the page gives a feel of having completed the process by themselves. Think of ticket booking system to visualize this, where you yourself hit the print link or icon when you are done with all the formalities to book the ticket.

You need Javascript for that. The window object of the browser have a print() method available, which you would call and trigger the print dialog. But, what would happen when Javascript is disabled on the browser? Your code will simply not work! And, that would create very bad impression about your page because you had promised that functionality!

With graceful degradation, you would take these steps to handle this:
1) Add a link to print the page.
2) Add <noscript> tag, which shows its contents only when Javascript is not available, and use it to suggest some alternative to print this page. Something like "Printing this page requires Javascript enabled. Either turn it ON, or use File > Print menu."

But this also requires your user to know what Javascript is, and how to enable it. Also, we are assuming that Javascript is available.

On the other hand, with progressive enhancement, you would take following steps to handle the same situation;
1) Show a simple message like "Please take a print of this page to keep on your records". Use the id attribute to identify this message container.
2) Using Javascript, check if type of window.print is 'function'. And, if it is so, create a button element having 'Print Now' value and append it to the above message.

See how defensive this code will be. Here, we aren't assuming anything.This will work for every user regardless of how the environment is.

It can be said that both - graceful degradation and progressive enhancement - try to do the same thing: keep our web site/app useful to every user. Progressive enhancement is more sophisticated and at the same time a stable way of assuring that, but it also takes more time and efforts. Graceful degradation can be used more easily as a patch for already existing page, but it also means harder maintenance later on, but requires lesser initial work.

Monday, April 27, 2015

Saturday, April 25, 2015

Week of the Month

In my project work, I needed to find out whether the given date falls in 1st, 2nd, 3rd, 4th or 5th week of the given month. As usual, I googled for this solution and found many functions. http://www.hotscripts.com/forums/php/37900-week-number-month.html, http://stackoverflow.com/questions/5853380/php-get-number-of-week-for-month and http://mybroadband.co.za/vb/showthread.php/66311-Calculating-the-week-in-the-month-(PHP) are some of them. But, being lazy, I thought those functions were too complex, and there could be a simpler solution to this. So I spent some time and came up with this function. Here is the code.

Hello World!

Hello World! 

Here with this blog post, I get started with my programming blog!! It's a well known fact that learning each programming language starts with a "hello world" program. And, so I wanted to start my programming blog with a "hello world" post. :) 

Oh, what I would be blogging about? Well... I haven't decided any fix theme for this blog. And, neither I want to tie it to any particular blogging genre. So, I would definitely like to make it a mixture of how-to's, code-sharing, tips, tricks, information sharing and what not?! Most of the times I work in PHP, so, yes, most of the blog posts would be related to PHP and web-world. Apart from that, I would be blogging about web UI frameworks, web-based apps, APIs etc.

Most of the code sharing will be through gists. I am a strong believer and follower of social coding, and that's why I keep on storing (and maintaining) my code snippets on gist/github. Feel free to follow me there as well. ;)

I hope I get plenty of time and information to share here!