Random Developments: 2012

A Simple Release Calendar App

Agile Releases

Agile releases by nature are short in duration and comprised of 1 or more iterations. The length of the iteration can be and arbitrary number of days. The organization that I work for uses a 10 business day iteration. Each releasable deployment is comprised of 2 iterations. Our releases are denoted by the last two digits of the year, a dot, and followed by the month: YY.MM.

So, for example, a release destined for the March 2013 would be stated as 13.03.

Our iterations are numbered as a sequence starting from some arbitrary number. Currently at my work we are in iteration numbers in the two hundreds: 213, 214, 215...etc.

Release 12.11 12.12 13.01 13.02 13.03

Iterations 220 222 224 226 228

221 223 225 227 229

Need for Automating Release Calendar Creation

To insure all team members are aligned, a single source for release calendar is key to smooth running delivery pipeline. You could assign an individual or group to manage the creation of a release calendar and publish it to a common document location within the organization. In fact this is what my current organization does. But saw that a little automation could ease the creation of the release calendar and provide a more flexible means of consumption thru a publicly available API.

Simple App Creation with App Frameworks

Since most of these kinds of projects are not directly funded within organizations because they generate no perceived business value, it makes sense to be able to quickly build and deploy your little app on a shoestring or no budget at all.

Frameworks like Ruby on Rails, Sinatra, Cake PHP and Grails offer developers a pathway to quickly develop useful applications without a lot of friction. By friction I mean the things that get in a developers way. These can be anything from overly complex computer languages or configurations, other teams that control infrastructure like deployment engineers or even DBAs. The more things you need to configure or hoops you need jump thru, the less likely you be able to deliver a useful tool within your organization.

Ideally, a developer should be able to build an application with a few days that accomplishes something useful. Because the application is somewhat under the radar and not customer facing, you shouldn't have to spend more on additional resources like a coding pair, a build and deployment engineer or a database administrator.

I chose to use Grails to create my application after trying several frameworks. I am primarily a Java developer so Grails felt much easier to start using than some other frameworks. Groovy as a language is very easy to pick up as a Java developer. I think it is cleaner syntacticly and I am a fan of convention over configuration which the Grails framework employs.

The Virtues of Flying Solo

Working in an agile development workplace should mean I myself follow agile development practices to develop all of my applications. Well, that's true for the official projects, the ones that have real ROI and are funded by the finance department and have budget and a full project team. When real money is being spent and actual paying customers will use the application, then it makes sense to go the full distance in supporting agile practices like TDD, pairing and retrospectives and such.

For my little release calendar application that is meant to be used by my organization's project and iteration managers, I can do away with a lot of overhead in terms of pairing and a full project team. I can pretty much do it all solo. Of course I will use TDD and unit testing, but since this app won't come into contact with paying customers and generate revenue, a full functional test suite won't be required. I won't have time or budget anyway! The idea is to generate something useful, fast.

Setting Up A Grails Project Application

I won't go into extreme detail with regards to Groovy and Grails. There are plenty of great sources online and books.

A useful text to that helped me is Manning's Grails in Action. Also check out the official Grails site.

MongoDB Datasource

I decided to use MongoDB as my datasource. I could have used a relational datastore like MySQL but I liked the fact that I could store the data for my release calendar as JSON objects since I was envisioning providing a RESTful API which would return JSON. This will simplify the code by removing unnecessary mapping layers.

Cloud Ready

Another form of friction when developing an application is where to host it. The usual drill is you need to beg, borrow or steal some hardware to host and deploy your app. This usually involves justifying the need with an infrastructure team and then asking the group that holds the purse strings to fund buying the hardware. Even if you have an internal cloud to deploy to, you still need permission to carve out some VMs to use.

Luckily, a lot of companies offer free cloud resources for small VM instances. For example, I am using Cloudfoundry but I have also used AppFog, OpenShift etc. I don't know how long this situation will last, but as long as these providers offer their services, I plan to use them to speed development and deployment.

At the very least, using one of these cloud providers can allow you to showcase your great idea to the powers that be and just maybe you'll get the funding and leadership approval necessary to host in your organizations private cloud.

App Design

Seeding the Release Calendar

Not wanting to invest a lot of time on a fancy UI, I chose to make my application primarily a REST application. Meaning, to initially seed my calendar, I provide a REST get request with a POST to define a release calendar:

http://hostname/release/calendar

Request POST payload:

{

"releaseName" : "MyRelease",

"releaseDesc" : "My releases are composed of 2 iterations. Each iteration contains 10 business days.",

"startDate" : '2012-06-29T00:00:00-05:00',

"duration" : "28",

"iterations" : "2",

"iterationNumber" : "213",

"releaseFormat" : "YY.MM"

}

Domain Models

As I mentioned earlier, I chose to use MongoDB as my primary datastore. This in turn means that under the Grails model of object relational model (GORM), each datastore plugin follows uses the same conventions of object instantiation and data access methods. When you create your domain objects, Grails behind the scenes creates the underlying object creation mapping and access methods.

There are several options for MongoDB plugins in Grails. I started off using Morphia but ended up using the 'official' MongoDB Grails plugin MongoDB 1.0.0.RC5. The main reason for this had to do with the fact that the cloudfoundry deployment tools natively recognized the domain objects and mapped them to the underlying services on cloudfoundry whereas the Morphia domain objects were not.

As with all Grails apps, you generate your domain objects from the grails command line:

create-domain-class com.gap.release.calendar.Day

create-domain-class com.gap.release.calendar.Release

generate-all com.gap.release.calendar.Day

generate-all com.gap.release.calendar.Release

Notice the create-domain-class command is being used. Since I installed the mongoDB Grails plugin (and uninstalled hibernate), Grails knows to use the MongoDB plugin to generate the mapping from Grails objects to the underlying MongoDB objects and providing the access methods leveraging the 'convention over configuration' paradigm. I'll talk more about that later when I present the REST API implementation code.

Taking a look at the generated domain models, I just filled in the attributes I needed. Only really different thing I added beyond standard data types is the BSON object id type to support the MongoDB notion object ID for each document. In this example, I am referencing the owning release calendar object in each calendar day. This is done to support multiple release calendars within a single document store.

package com.gap.release.calendar

import java.util.Date;

import org.bson.types.ObjectId

class Day {
 Date relCalDay
 String release
 Integer iterationNumber
 Integer iterationDay
 ObjectId releaseId

    static constraints = {
    }
}

Here is the release calendar domain object. Again, nothing too spectacular here, just basic datatypes beyond the BSON objectId:

package com.gap.release.calendar

import java.util.Date;

import org.bson.types.ObjectId

class Release {

 ObjectId id
 String releaseName
 String releaseDesc
 Date startDate
 Integer relDurationDays
 Integer numIterations
 Integer iterationNumber
 String releaseFormat
 
    static constraints = {
    }
}

URL Mappings in Grails

One of the major reasons I like using Grails for developing simple apps is because implementing a REST API is pretty damn easy as compared to some other established frameworks like Spring. Sure there are other factors to consider if you are deploying to a heavily traffic e-commerce or social site, but for little in-house tools like my release calendar application, the trade offs are worth it.

OK, I've got a confession about convention over configuration and Grails. It turns out that not all configuration can be abstracted away. But at least Grails puts it all under one roof, more or less. Under the 'conf' location in the project navigator lies all the application configurations that may require tweaking, things like Spring application context, datasources, and in this case URL mappings.

The URL mappings file URLMappings.groovy contains the REST API URL mappings to controllers:

class UrlMappings {

 static mappings = {
  
  "/release/calendar"(controller: "relCalRest") {
   action = [GET: "list", POST: "save" ]
  }
  
  "/release/calendar/$releaseID"(controller: "relCalRest") {
   action = [GET: "listRel" ]
  }
  
  "/release/calendar/$releaseID/$calDate?"(controller: "relCalRest") {
   action = [GET: "listDay" ]
  }

  "/$controller/$action?/$id?"{
   constraints {
    // apply constraints here
   }
  }

  "/"(view:"/index")
  "500"(view:'/error')
 }
}

The URLMappings class is where you define which REST action methods get implemented in your Grails controller classes. Next we'll look at the calendar seeding controller called "restCalRest".

Calendars, Dates and Date Math

From URLMappings class we can see that when we encounter the URL http://hostname/release/calendar with a POST, we will use the "restCalRest" class to implement the backend logic to create, or seed a release calendar. Using GORM terminology we are mapping the POST to the save method of the "restCalRest" controller.

Controllers: Where The Magic Happens

It's not really that magical actually. We are passed a block of JSON, we map the JSON to our domain objects and we save them. By saving them, we are instructing the domain objects to persist to our chosen datastore. In this case our datastore is MongoDB.

 def save = {
  def json = request.JSON
  
  def release = new Release()
  
  release.releaseName = json.releaseName
  release.releaseDesc = json.releaseDesc
  DateTimeFormatter parser = ISODateTimeFormat.dateTimeParser()
  DateTime startDate = parser.parseDateTime(json.startDate)
  release.startDate = new SimpleDateFormat("yyyy-MM-dd").parse(json.startDate)
  release.relDurationDays = json.duration.toInteger()
  release.numIterations = json.iterations.toInteger()
  release.iterationNumber = json.iterationNumber.toInteger()
  release.releaseFormat = json.releaseFormat
  
  if (release.save()) {
   calDayGeneratorService.generateDays(release)
   render contentType: "application/json", {
    // Return the ID of the new release
    ['id' : release.id.toString()]
   }
  }
  else {
   render contentType: "application/json", {
    ['status' : 'FAILED']
   }
  }
 }

It's pretty straight forward actually but notice that I have utilized a service "calDayGeneratorService" to do the heavy lifting of generating the release calendar days. Let's look at the details of the "generateDays" method of "calDayGeneratorService"

The basic logic is take the initial date passed in and figure out based on that the next release startdate and fill in the iterations per release and iteration days within each iteration. It uses the Jodatime plugin because it provides convenience routines for figuring out periods between dates and date formatting for months, days and years.

class CalDayGeneratorService {
 
    def generateDays(Release release) {  
  Integer totalDuration = 0
  DateTime nextReleaseDate = new DateTime(release.getStartDate())
  nextReleaseDate = nextReleaseDate.plus(Period.days(release.getRelDurationDays()*2))
  Date relCalDate = release.getStartDate()
  Integer iterationNumber = release.getIterationNumber()
  while (totalDuration < 365) {
   Integer relCnt = 0
      while (relCnt < release.getRelDurationDays()) {
    String relName = nextReleaseDate.toString()
    relName = relName[2..3]
    def relMonth = nextReleaseDate.monthOfYear.toString().padLeft(2,'0')
    relName = relName + '.' + relMonth
  
    for (int iterationCnt=0; iterationCnt < release.getNumIterations(); iterationCnt++) {
     int itDay=1
     for (int iterationDays=0; iterationDays < (release.getRelDurationDays())/2; iterationDays++ ) {
      def day = new Day()
      day.iterationNumber = iterationNumber
      day.relCalDay = relCalDate
      day.release = relName
      day.releaseId = release.id
      if ((relCalDate[Calendar.DAY_OF_WEEK] != Calendar.SATURDAY) &&
       (relCalDate[Calendar.DAY_OF_WEEK] != Calendar.SUNDAY)) {
       day.iterationDay = itDay++
      } else {
       day.iterationDay = itDay
      }
      day.save()
      relCalDate = relCalDate + 1
      relCnt++
     }
     iterationNumber++
    }
      }
      totalDuration = totalDuration + release.getRelDurationDays()
   nextReleaseDate = nextReleaseDate.plus(Period.days(release.getRelDurationDays()))
  }
    }
}

The only thing really special in generating the iteration days happens around week ends. Since we don't (usually!) work in weekends, I put special logic to recognize when Saturdays and Sundays are encountered, just jump ahead to the next iteration day. Even this is pretty easy using Jodatime DAY_OF_WEEK and Calendar.Saturday and Calendar.Sunday shortcuts.

MongoDB

When the save method is called on a domain object, it calls MongoDB to persist the objects in what is called a collection. After calling the release calendar app to seed the release calendar days, we can take a look at what gets stored in our MongoDB collections.

barry-alexanders-MacBook-Pro:bin barryalexander$ ./mongo

MongoDB shell version: 2.0.2

connecting to: test

> show dbs

ReleaseCalendar 0.203125GB

local (empty)

test 0.203125GB

> use ReleaseCalendar

switched to db ReleaseCalendar

> show collections

day

day.next_id

release

release.next_id

system.indexes

user

user.next_id

From this sequence of commands we can see that Grails thru the MongoDB plugin created our day and release collections and sequence objects to generate unique IDs for each.

We can sample the collections to look at what documents got created in each collection:

> db.release.find()

{ "_id" : ObjectId("504c1f8b036461c1a2f31829"), "iterationNumber" : 213, "numIterations" : 2, "relDurationDays" : 28, "releaseDesc" : "GID releases are composed of 2 iterations. Each iteration contains 10 business days (14 calendar days) week days (M-F) only.", "releaseFormat" : "YY.MM", "releaseName" : "GIDRelease", "startDate" : ISODate("2012-06-29T07:00:00Z"), "version" : NumberLong(1)

}

Here is a small sampling from the day collection:

> db.day.find()
{ "_id" : NumberLong(673), "iterationDay" : 1, "iterationNumber" : 213, "relCalDay" : ISODate("2012-06-29T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }
{ "_id" : NumberLong(674), "iterationDay" : 2, "iterationNumber" : 213, "relCalDay" : ISODate("2012-06-30T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }
{ "_id" : NumberLong(675), "iterationDay" : 2, "iterationNumber" : 213, "relCalDay" : ISODate("2012-07-01T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }
{ "_id" : NumberLong(676), "iterationDay" : 2, "iterationNumber" : 213, "relCalDay" : ISODate("2012-07-02T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }
{ "_id" : NumberLong(677), "iterationDay" : 3, "iterationNumber" : 213, "relCalDay" : ISODate("2012-07-03T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }
{ "_id" : NumberLong(678), "iterationDay" : 4, "iterationNumber" : 213, "relCalDay" : ISODate("2012-07-04T07:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("504c1f8b036461c1a2f31829"), "version" : 0 }

Using the Release Calendar API

Once the calendar is seeded with days, the GET method APIs can be used to either return all releases stored in the database or return details about a given date.

 def list = {
  def List <Release> releaseList = Release.findAll()
  render releaseList as JSON
 }
 
 def listRel = {
  def release = Release.findByReleaseName(params.releaseID)
  render release.encodeAsJSON()
 }

As seen in the list method above, getting a list is as simple as calling the findAll method on the domain object and rendering as JSON. The listRel method uses the convention of the findBy method followed by the key(s) to search by, in this case ReleaseId. This will find the release with the passed parameter on the URL: /release/calendar/$releaseID

The listDay method provides the response to the GET day request of the release calendar API. The URL contains the release name (id) and a date. The method looks up the given release calendar day and returns the JSON rendering of the day domain object. The only thing of note is that I added support for JSONP to allow cross site calling of the API.

 def listDay = {
  def release = Release.findByReleaseName(params.releaseID)
  
  Date calendarDate = new SimpleDateFormat("yyyy-MM-dd").parse(params.calDate)

  def day = Day.findByRelCalDayAndReleaseId(calendarDate,release.id)
  // gotta support jsonp too!
  if (params.callback) {
   render params.callback + '(' + day.encodeAsJSON() + ')'
  }
  else {
   render day.encodeAsJSON()
  }
 }

Deploying to the Cloud

Cloudfoundry has provided a Grails plugin that allows deploying directly from Springsource Tool Suite (STS). To deploy to cloudfoundry within STS, it's just a matter of issuing this command:

grails prod cf-push

Consult the cloundfoundry site on how to open an account and setup. Since I am using the MongoDB Grails plugin from VMware, when the application is deployed, it automatically provisions the MongoDB service.

After execute the push command, the console will show the deployment progress:

| Loading Grails 2.0.0

| Configuring classpath.

| Environment set to production.....

Building war file

| Packaging Grails application.....

| Compiling 14 GSP files for package [relCal2]..

| Compiling 8 GSP files for package [jodaTime]..

| Building WAR file.....

| Done creating WAR target/cf-temp-1352438416722.war

Application Deployed URL: 'RelCal2.cloudfoundry.com'? y

Would you like to bind the 'mongodb-relcal' service?[y,n] y

| Creating application RelCal2 at RelCal2.cloudfoundry.com with 512MB and services [mongodb-relcal]: OK

| Uploading Application:

| Checking for available resources: OK

| Processing resources: OK

| Packing application: OK

| Uploading (2K): OK

| Trying to start Application: 'RelCal2'.....

| Application 'RelCal2' started at http://relcal2.cloudfoundry.com

Seeding the Release Calendar with Starting Data

Using a HTTP posting tool like XHR Poster, I sent a request POST with the following JSON payload:

{

"releaseName" : "GID Release",

"releaseDesc" : "GID releases are composed of 2 iterations. Each iteration contains 10 business days (14 calendar days) week days (M-F) only.",

"startDate" : '2012-06-29T00:00:00-05:00',

"duration" : "28",

"iterations" : "2",

"iterationNumber" : "213",

"releaseFormat" : "YY.MM"

}

This will cause the app to generate the release calendar days starting from the start date for one year worth of days.

Verifying The Data

Verifying the MongoDB datastore on cloudfoundry requires you to use something called caldecott. It's a tunneling tool to access services:

barry-alexanders-MacBook-Pro: barryalexander$ vmc tunnel mongodb-relcal none

Getting tunnel connection info: OK

Service connection info:

username : ************************************

password : ************************************

name : db

url : mongodb://c71364b2-96c4-4bfa-abd5-614f07ab4435:efbdb576-5269-4719-a301-4a9ce5c9b7d9@172.30.48.63:25138/db

Starting tunnel to mongodb-relcal on port 10000.

Open another shell to run command-line clients or

use a UI tool to connect using the displayed information.

Press Ctrl-C to exit...

In another terminal, connect to MongoDB with the mongo command using the generated username and password:

barry-alexanders-MacBook-Pro:bin barryalexander$ ./mongo -port 10000 -u *********************************** -p *********************************** db

MongoDB shell version: 2.0.2

connecting to: 127.0.0.1:10000/db

> show collections

day

day.next_id

release

system.indexes

system.users

user

user.next_id

> db.release.find()

{ "_id" : ObjectId("509c9b25b844c4506558c0d7"), "iterationNumber" : 213, "numIterations" : 2, "relDurationDays" : 28, "releaseDesc" : "GID releases are composed of 2 iterations. Each iteration contains 10 business days (14 calendar days) week days (M-F) only.", "releaseFormat" : "YY.MM", "releaseName" : "GID Release", "startDate" : ISODate("2012-06-29T00:00:00Z"), "version" : 0 }

> db.day.find()

{ "_id" : NumberLong(393), "iterationDay" : 1, "iterationNumber" : 213, "relCalDay" : ISODate("2012-06-29T00:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("509c9b25b844c4506558c0d7"), "version" : 0 }

{ "_id" : NumberLong(394), "iterationDay" : 2, "iterationNumber" : 213, "relCalDay" : ISODate("2012-06-30T00:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("509c9b25b844c4506558c0d7"), "version" : 0 }

{ "_id" : NumberLong(395), "iterationDay" : 2, "iterationNumber" : 213, "relCalDay" : ISODate("2012-07-01T00:00:00Z"), "release" : "12.08", "releaseId" : ObjectId("509c9b25b844c4506558c0d7"), "version" : 0 }

...

Source Code

Source code for this blog post can be found here: https://github.com/balexander123/RelCal2.git

Random Developments

Thursday, November 8, 2012

A Simple Release Calendar App

A Simple Release Calendar App

Agile Releases

Need for Automating Release Calendar Creation

Simple App Creation with App Frameworks

The Virtues of Flying Solo

Setting Up A Grails Project Application

MongoDB Datasource

Cloud Ready

App Design

Seeding the Release Calendar

Domain Models

URL Mappings in Grails

Calendars, Dates and Date Math

Controllers: Where The Magic Happens

MongoDB

Using the Release Calendar API

Deploying to the Cloud

Seeding the Release Calendar with Starting Data

Verifying The Data

Source Code

Followers

Blog Archive

About Me