10 min to read
De-coupling releases from deployment
IT industry is all about change. Functionality needs to constantly evolve. You need more content, more products, more promotions and what not. In fact, a static web site is not very interesting in the eyes of its users. It gives the impression that the founders have gone fishing and have left the users to their own devices. Hence software releases are a part and parcel of the lives of all folks involved with IT. In most enterprises, releases are mega affairs. They need planning and a non trivial amount of co-ordination between multiple groups. All teams need to deploy their latest code. The QA folks need to line up their test cases. The operations folks need to be ready. Business users need to get ready to change their ways of working. The product owners must polish their trumpets to blow them for the great show. Once the ducks get lined up, the grand spectacle unfolds. Is this going to be a show of triumph or a debacle? Lots of bated breaths. Finally, the whole spectacle ends. The fever passes and lives return to a semblance of normalcy. People get to relax. Until the next release! This whole charade recurs at periodic intervals much to the dread of all the participants. For anyone who has been in any kind of enterprise, all this will sound very familiar. The release planning process is itself quite complicated. Typically, multiple features get planned to be released at the same time. All software systems that need to change to implement the features then get identified. The teams are asked to plan their deployment to coincide on the grand day. Testing needs to be planned accordingly. People accept this as part of their lives. But does it have to be this hard? Why are software releases such a nightmare? The answer is simple. Releases are hard because they have been used as instruments for not only enterprise change but also IT change. This approach combines too many responsibilities and hence increases the number of potential sources of failure. A release can fail if any of the software systems that it touches fails to change gracefully.Or, if the business is not ready. Or, if the QA is not ready to test the changes. etc. etc. All for the want of a horseshoe nail! 
The solution to the problem must be to simplify. This can only be done by rethinking the way we do releases. Releases must not be viewed as conduits of IT change. Which begs the question - What is IT change? IT Change is inextricably linked to deploying new versions of the software. So to simplify releases, we need to de-link the two.
Model for a Software Release
The model for a typical software release is shown below: [caption id=”attachment_980” align=”alignleft” width=”300”]
Enterprise Release Model[/caption] Look at 15.1,15.2 etc. which represent Enterprise Releases.The features F2,F1,F4 and F3 respectively are being released in these timelines. All systems that need to be changed (system 1, system 2 etc.) need to ensure that these features are made ready during the exact day of the release. Quite clearly, this is a formidable proposition. Instead, consider an alternative model which is shown below. [caption id=”attachment_979” align=”alignright” width=”300”]
Autonomous Deployment Model[/caption] In this model, the release timeline is completely skewed. Each product has its own timeline to implement features. Some features can even be combined into one deployment. (e.g. System 3 implemented features F2 and F3 into one deployment). Some other features can be delivered out of sequence. For example system 1 delivered the orange feature before it delivered the dark blue feature while the other systems delivered the blue feature first. Quite clearly, in this model each system is able to deliver features at its own pace. But this will only work if the systems are able to track dependencies between each other and can be deployed and tested independent of each other. In this way, there are no ducks to line up. Each system is delivering its own features in its own timeline. There is not much of a tension for each team since no one is waiting for them to deliver anything. Each product is in its own timeline. Some of these features can even be hidden with a feature flag and “go dark” in the deployment.
Software Release vs. Business Release
In this context, what is a release? When do we say that Feature F1 has been delivered for instance? The astute reader would look at the Autonomous deployment model above and state that F1 (in light blue) would be ready between 5 ‘15 and 8 ‘15 - somewhere in the neighborhood of 7 ‘15. (after System 3 which is the last system to implement feature F1 deploys it in production) So the Product owner now can “advertise” Feature F1 during that time to end users. Feature F1 may possibly be “turned on” with some feature toggle. During that time, the business teams would start getting ready to make relevant business changes that might be required by feature F1. In short, we would have de-coupled deployment from release. Release would become a conduit for business change not a software IT construct. This change would be so subtle that hardly anyone notices it. Teams would not even think of it as a big deal anymore. We have effectively de-coupled business timelines from IT timelines. All of a sudden, the release fever would become a thing of the past. No more sleepless nights! Seriously! A’int that cool?
Comments
Ramanan: Hello Raja, well written thoughts! just to add if the features are delivered as services that can be independently deployed and whenever the consumer chooses to consume it, then the so called hard dependency of all systems to be lined up for the change goes away, Also, there is an inherent desire in humans to complicate things, and that is also one of the reasons for big bang releases. keep it simple can well be the new mantra that will avoid all these hassles
raja shankar kolluru: Thanks Ramanan. Absolutely agreed!
Hariharan: Sir, Recently we worked for a big client with the usual “time-to-market” proposition, but multiple releases toeing line one behind the other. These multiple releases were also managed my multiple teams, but all relying on a single code base. There was so much fuss and lamenting about not having multiple branches. Considering multiple teams, working in their own pace, and timelines, one of the ways we manifested the principle written by you - is to have a SINGLE code branch for the whole program, and allow deployments to happen per their deadlines. So, while T1 (Team 1) released their site, along with their deployment T2’s “almost complete code” would be there part of that deployment, similarly T3’s initial code, considering the deadlines are in the order: T1 < T2 < T3 However, since T2 and T3 teams’ releases were unmaterialized, there were no issues - Also we could periodically check if all the code could reside together! :) Thanks, Hari
raja shankar kolluru: This is perfect Hari. I think it is a big mindset change to move away from multiple branches and commit all code in various states of maturity to trunk with the features turned off. The interesting observation here though is that even if we do choose to exercise the “feature branches” concept wherein you have one branch per feature, this will continue to make sense to de-couple business releases from software releases. You definitely want to cut out unnecessary noise when making a business release and the best way to do so is to use the business release as an “marketing exercise” to advertise published features in the software rather than as a conduit for software development. I admire you folks though to force the “one branch” concept aka Trunk Based Development.
Hariharan: Yes, sir. In an ideal world, we would have created an additional branch for “future enhancements” after the first cut of the release, when there was enough “interval” between multiple site launches. Generally, R1, post its release, takes a separate branch, and R2 (by Team T2) begins on a new branch. Before release of R2, fixes in R1 are merged into this new branch. The TRUNK is periodically kept in sync with latest. The usual process. But in our case, we were dealing with a CMS solution, where we had multiple site releases in succession of weeks, and not months. Also, these multiple code changes was going to the same deployment cloud. So, it was a case of tradeoff between risk of merging, versus risk of deployment an unfinished code. Also we found that switching “on” of a release/ feature is not too complicated in a CMS - only if I push the “content” related to that releaseI essentially switch on that site - that is the simple idea we piggy backed on. Like what you said in your paper, between R1 and R2, during the release of R2 (which is later), we could include some features missed/ fixed during the first cut release of R1. So, since it was one branch, no one had to do merging - If at all cumulative issues came out, those all came out during the staging of R2. The testing overhead increased, but we had to have a strong management of tasks. As I have learnt in this little experience, sometimes “processes” and “standards” are given untoward importance without considering the merit of a situation - In IT, we sometimes get to work like “bapus” - People cursed me for not going for multiple branches - But then, software development is all about seeing a problem for its merit and dealing with it - not ignoring standards, but evolving something out of the existing ones for the final goal. :)
raja shankar kolluru: Great Hari. Thanks for sharing. One observation by the way. Even if you choose the multiple branch route, it is best to create the branches by “features” rather than releases. Creating a branch per software release combines the branching strategy too strongly with the release strategy.
Hariharan: Yes, sir - valid point. I will try to remember that. As such, the “release” vs a “feature” concept may be a thin line to differentiate in many projects. But, good to stop, think and proceed in the manner you mentioned.
raja shankar kolluru: Absolutely. Thin line indeed. One way I tend to distinguish between a feature and a release is by remembering that a release is typically entwined with a timeline (this will go live in August etc.) whereas a feature is a piece of functionality that the user requires.
Comments