Working towards a dream - Building a CI/CD Pipeline

It’s been almost more than a year since we started working on building a CI/CD pipeline at my company. We’re a team of around 50 software developers, QA testers, and designers.

With a team this size, there are a high number of stakeholders and collaborators involved with every module. This basically means that the final code would have gone through the hands of multiple developers before it gets deployed to production.

How do you ensure that in such high velocity environment, where code gets changed at least 50 times before getting deployed, that the bugs you squashed in development or staging environment don’t show themselves up in production?

Other than that, how do you make sure that your build only goes live after it has been completely approved by the QA team?

Until a while ago, all of our deployment system is 100% manual - from compiling code on developer’s machine to taking the production system on remote and copy-pasting the compiled code.

Release days were a major pain in the ass - one team would have to be present to ensure code compiled well, one team was ensuring the compiled code was deployed properly, and QA team was simultaneously running all the test cases manually to make sure build was deployed properly.

You can imagine what kind of a chaos soup you would have with all these people internally screaming their lungs out at 12 AM when a small coding error would make the entire build fail.

This is when we decided ENOUGH IS ENOUGH! We will start working towards a dream CI/CD pipeline. This would require minimal manual intervention from each team and help us define clear roles during the entire software development lifecycle.

With the bare minimum software development and architecture experience, I started off studying more about the theories behind CI/CD and how exactly to implement the same. But it wasn’t as easy as it seems - everyone talks about CI/CD a lot but the truth is there’s no ‘one size fit all’ approach to CI/CD.

Every CI/CD pipeline is different. Of course it has the same approach but designing it entirely depends on how flexible your team and code-base is.

But this is how my rough CI/CD workflow is mapped out -

1. Developer commits their code to the ‘staging’ branch. This is the code that needs to go for QA testing.

2. A simple Slack command triggers a Jenkins job that pulls the code from a remote git repo and runs a build.

3. The same Jenkins job deploys the complete build to the Staging environment as configured. This includes deploying docker containers through ECS, updating the Elastic server, deploying the monolith application on an EC2, and running the database migration scripts on MSSQL Server. (Imagine how all of this was done manually before…)

4. After the complete deployment is successful, another Jenkins job is triggered which starts to run all the automated tests developed using Selenium - these includes UI Tests, Functional Tests, and API Tests. The final test report is generated and emailed to the deployment team. Failed bugs are reported as bugs on JIRA. Simultaneously the QA Testers run their manual test cases and report any bugs on JIRA.

5. The same cycle gets repeated until all the automated test cases are passed and QA team gives the final approval to release the build to production.

6. Once the production release approval is processed, the code is pulled out from ‘staging’ branch and pushed to the ‘master’ branch.

7. Now ‘master’ branch triggers another Jenkins build which is responsible for building the code for production and deploying the same on the production environment.

8. After the build is successfully deployed on production, the same set of integration tests are run on the production build and any bugs are immediately reported and squashed.

This dream workflow is really difficult to achieve especially when our entire workflow was manual. However, we’re on track to achieve 80% of it over next one week.

Building this pipeline has come with a lot of learning lessons because it involved A LOT of conflict resolutions, drastic changes in coding practice, and most importantly breaking through the traditional mindsets & practices people thought that were best.

You can never bring any positive change unless you forget about what you already know and start fresh. Of course, implementing this philosophy isn’t a piece of cake on a personal level. You can imagine how difficult would it be to implement this at an organizational level.