I am in the airport at San Francisco writing this blogpost. Over the last 2 weeks, I have constantly been in meetings with some of the most important companies in the regions, the hot product companies, the “unicorn” startups and more. If anyone has been around the valley since 2014-2015, the buzzword is “DevOps”. DevOps is rapidly gaining traction as the most essential attribute of application development and deployment, since….well, application development and deployment. However, I while companies (large and small) are taking to DevOps implementations and practices extensively, they are going through the same set of issues that plagued them in the old paradigm, specifically in terms of Security. Let me get to that in a bit. But first, a little bit of context….
What is DevOps?
Agile Development changed everything in the world of applications. Using simple steps and a result-oriented approach, application developers were able to move away from the waterfall model (a practice that frankly sucks due to its high potential for project failure) to how things really work (Agile, in rapid application development lifecycles). Everyone today (including some of the stodgy, luddite corporations) are embracing the Agile model of Application Project Management, which is built on the premise of rapid application development and deployment. As agile was gaining traction, it was seen that the people and processes involved in Agile development, were still structured around the old waterfall methodology, with business stakeholders, developers, architects, QA, testers, operations, all functioning as separate units with little interaction and collaboration with each other. This of course, was anything but agile, and soon, a new way of working was identified. This is DevOps. DevOps is a series of practices, technologies and processes that involve a high degree of collaboration with the different teams involved in agile development to facilitate rapid application development. This essentially meant that business, devs, architects, testers, ops, etc work together using amalgamated practices to leverage high performance in the application development lifecycle. This leveraged practices like Continuous Development, Integration and Deployment, which means a high degree of orchestration in planning, development, testing and deployment. Especially in the cloud, these practices can be highly beneficial for organizations, resulting in nimble and truly Agile application development. This has resulted in the creation of a separate position in the organization which is focused on DevOps.
This sounds great… What’s the problem?
It does sound great, and it is. DevOps has resulted in incredible benefits for its adopters. Technologies and practices around Continuous Integration (CI), Continuous Deployment (Machine Orchestration) has been hugely beneficial for organizations that have large/complex application environments on and off the cloud. However, as is always seen with a disruptive IT industry, security is usually forgotten. Recently, a hot healthcare product startup came to us for some consulting and penetration testing. They had deployed their infrastructure on Amazon, with an analytics and search layer using ElasticSearch. They had developed an app that was slated to be delivered to some large hospitals in the US. They had a DevOps team that was handling all the orchestration between their various components, that were many and quite complex. Mid-way into the penetration testing, we had already found a rash of Object reference vulnerabilities, crypto flaws and worse, we were able to exploit their vulnerable ElasticSearch instance and completely compromise their Amazon VPC. So, the natural questions that emerged from our consulting engagement with them subsequently, were on the lines of:
When the answer to all these questions was met with a pregnant pause, we realized that they clearly had missed all/most of them. Over the last 2 weeks, I have been observing similar trends with most appdev projects or product companies that we have spoken to.
That sounds bad….What do we do?
The good thing is that there is something you can do about it. One, realize that DevOps without security built-in is just an accident waiting to happen. You may have rapid development and deployment cycles, but without Security in DevOps or “SecDevOps”, you are rapidly developing and deploying non-secure apps, which is worse than the non-DevOps state of affairs. Here are some practical steps that you can take. In our engagements with clients involved in DevOps, this is what I have seen working pretty well.
Dirty Threat Modeling
There was a time when I would have advocated a well-laid out Application Risk Assessment plan, but the reality is that in a rapid development lifecycle, Application Security Risk Assessment just doesn’t work. Its slower and people just stop following it. During your product Backlog, you need to do a solid, quick and dirty threat modeling that becomes a template for your entire lifecycle. We have done a ton of assisted threat models for clients, and this works, especially if you do it with some hardcore metrics.
Fix DevOps Awareness
Most Companies try and fix Developer Awareness. This is not nearly enough in a DevOps world. DevOps uses a ton of technology and practices like CI (Jenkins, Travis, Bamboo, etc), Continuous Delivery (Chef, Puppet, Ansible, etc), and a ton of other home-brew stuff on the cloud or on premise. Training and security awareness for the entire DevOps process is essential. I have seen more than a 70% reduction of security incidents, when we have done DevOps oriented security training for our clients.
Continuous Security Integration
Continuous Integration is an amazing practice. It has changed the way apps are developed and tested. However, it is seldom leveraged for security. At we45, we have developed some significant way to do “Continuous Security Integration” which involves creating custom security automation and powerful security validation practices that integrate with CI. This ensures that security vulnerabilities and bugs are caught early and frequently, ensuring that you don’t have the tail-end surprises emerging from a pentest. This itself can be an area of great depth. More on this in a later article though….
Security in Deployment Orchestration
Many companies that we work with use some sort of Continuous Deployment framework like Chef, Puppet, Ansible, SaltStack, etc. However, security is a critical underlying practice that needs to be integrated into cookbooks or automation scripts used in each of these frameworks. This is critical, especially when you have a large number of moving parts in your application environment.
Conclusion
DevOps is here to stay. However, without security, it will be rudderless and worse, dangerous. Security in DevOps or SecDevOps really is similar to the age-old concept of developing security right from inception. The cycles, practices and technologies need to be tweaked to ensure that security is smoothly integrated with the cycle. This as always, takes some discipline, some automation and some management.
