I’m a big fan of building security into existing processes, a term coined as “secure paved roads” by Jason Chan, Ex-Netflix CISO. The idea behind this is that security should mostly be invisible. The average employee should simply not have to think about high-consequence security domains. They’ll use tools that make their life easier, and that tooling has security built in by default. Yes, they can divert from that paved path, but they’ll generally have a worse experience. The idea is that 95% of people will stick to the paved paths, and the only people who divert off it have exceptional reasons to do so.

In the past five years or so, we’ve seen more security engineers develop these kinds of tools as well as dedicated teams solving this problem popping up. My goal in this blog is to explore:

  1. What are security platform engineering teams?
  2. What do they work on?
  3. How do you develop this function?
  4. When should you staff a team?

Why Security Platform Engineering Teams?

Your organization will likely face two sets of security problems. There will be problems unique to your company and problems where the solution is sold as a commodity.

The prime example is that staff single sign-on is a commodity; few companies are building their own single sign-on platforms. They are using vendors like Okta, Microsoft and Ping Identity for this. It’s a waste of resources for everyone to reinvent the wheel here and build their own SSO platform, especially for something so critical to both usability and security. There isn’t a single company out there that should be dedicating resources to building an SSO platform rather than product-facing features that will earn revenue. There would need to be some highly unique circumstances to break away from this model, which we’ll dive into below.

On the other hand, you might have a custom internal problem that can’t be easily commoditized. A great example of this in many software companies are customer support tools that help engineers interact with customer instances. These tools need to be secure and usable, but it isn’t a problem a vendor can easily fix because it is truly unique to your product. Generally speaking, you should focus your efforts on problems that don’t have an off-the-shelf solution. Now, of course, there are exceptions, such as:

  • Cases where you can build it cheaper than what a vendor is offering
  • When you have issues with the vendor’s security posture and want to keep it internal
  • The only vendors in that space are competitors you don’t want to give money to
  • Avoiding vendor lock-in either because you think they’ll increase prices or not build out future features you require
  • Where the solutions provided can’t scale to your needs

What Problems Do They Solve?

Security platform engineering teams are focused on fixing the above problems. They have two tasks, which are to:

  • Build new tools and services to scale security.
  • Collaborate with other engineering teams to build secure by-default systems.

I led one of many platform engineering teams at Shopify and have spoken with a few of these teams in the industry. Some common themes tend to emerge across industries and verticals:

  • Internal Access: IAM, PAM, zero trust, and internal access are areas where vendors are improving rapidly, but this has been a common task for these teams in previous years.
  • Developer Tooling: Ensuring developers build secure services. This is often in the form of encryption services, secrets management, authentication, identity and more.
  • Customer Support: Customer support tooling beyond the standard helpdesk is often unique for every software company. It’s also among the highest risk areas, given that customer support directly interact with unknown entities daily.
  • Scaling Security: Vulnerability management, security reviews, risk assessments and more. This team can build tools that help the security team scale without growing headcount.

Every security platform engineering team should know you’ll occasionally have to throw away your work. I can give you a clear example of this from my past. At Shopify, we built a zero trust solution that checked device posture and prevented access to applications unless you came from a secured, managed device. This worked great for several years, but vendors eventually caught up and overtook where we were. The reality is we only had a handful of engineers, whereas the security vendors have hundreds, so we always knew we would eventually replace components of it.

The idea behind security platform engineering teams is to fix unique problems, but if those problems no longer become unique, be ruthlessly willing to switch to a vendor. It’s not a waste; you protected the company for that time, and now you can focus on something new and exciting.

This is one of the things I love about this area; you are constantly working on the cutting edge of security research and development.

Developing This Function

There are two questions you need to ask yourself when evaluating a security development function:

  1. Do you have unique needs in your organization which can’t be fixed by off-the-shelf tools?
  2. Do you have a team big enough where you can dedicate significant amounts of time to strategic bets which don’t give immediate value?

If the answer is yes to both, you can simply request headcount and try to staff this function from day one, and that is precisely how the team did it at Atlassian.

Suppose the answer is yes to the first but no to the second. In that case, the alternative is to hire a single software engineer or have security engineers work on this as a part of their responsibilities. Still, I recommend something other than this approach for the long term. It can be an excellent way to see if this works for your organization, but there are more effective ways of operating.

If you hire security engineers, you want them to solve security problems. You don’t want them to be torn between roles, trying to be a good developer and a good security engineer. This can work for a while, but it comes with downsides, including:

  • Burnout for the engineer. Good luck taking time off if you are the sole owner of a critical Tier-0 service.
  • Lack of growth since there is nobody to learn from or bounce ideas off of.
  • Context switching which leads to slow progress in both realms.
  • If that person leaves, there is no redundancy and finding another unicorn to fill the gap is not a scalable approach.
  • Handovers to dedicated engineering teams can be difficult. Engineering teams are usually not pleased having to take over services created by a single person, where they may have had to cut corners to get projects over the line.

I’ve had cases where I’ve had to use this model in the past due to a lack of resources, and my experience is that it works well, right up until it doesn’t. Be wary of tackling giant enterprise projects, only for the engineer to burn out, leave and have nobody want to pick up where they left off. This work can often be thrown away because of this, and it would have been better for the engineer to tackle projects with more longevity.

This org should also run with the company’s engineering processes. Security engineers will often know Python or Go; if you are a Java or C# based company, you want security platform engineering teams to use the same language, frameworks and methods to build software. Don’t be that team that builds everything in Rust at a Java company.

When to Create A Team

Most security teams are small and only contain a handful of people, with the actual size being heavily industry-dependent. Security, like any team, can be a single person up to several thousand at some of the largest companies. Regarding team makeup, you will almost certainly hire AppSec, Detection & Response, GRC and IT security functions before you build out anything else.

The organizations that grow beyond this must think heavily about how they scale operations. It’s unlikely and unreasonable to think that security teams will continue to balloon in size, manually dealing with tasks one by one. Instead, we need some way to build security for the masses. Security development teams can reduce the overall burden on the security team by creating platforms that reduce security issues across the whole organization. This means the ROI for this team increases as the company grows, unlike other security teams. My rough guide is that if your security team is approaching 30 individuals, you have unique problems which need custom code to solve, and you have at least three available headcount (one eng manager, two engineers), then you should think about starting up a dedicated team. This will get you the bare minimum required to get started.

A truly effective security development function comprises of an engineering manager, multiple software engineers (Frontend and Backend) and, ideally product managers. You might also have specialists who focus on different elements, like an AI engineer, to work on AI problems across the stack. These specialists can, of course, be dedicated security engineers as well, and you’ll find most of these teams have a few ex-security engineers turned builders on them. This team should be able to run Tier-0 services, have on-call functionality and act primarily like any other software engineering team more than security. Most importantly, it should spend most of its time building. This team’s core value is focus, not context-switching between security, development and other tasks.

Summary

Recently, I did a poll on LinkedIn, and the results didn’t surprise me. Many teams have their security engineers also do development work solving these problems, and at least a few respondents had the resources to dedicate entire teams to this but chose not to. I encourage security engineers to tinker and build, but I recommend they focus on becoming non-practicing experts. They can build interesting tools that help solve problems, but they should stay away from anything that might become enterprise-scale or find its way in the path of the product. An example would be that DFIR analysts should focus on building tools which help their team rather than the entire company.

linkedin poll

Security platform engineering teams are a great option to help scale security across your entire organization, but they will continue to be a niche element only used by larger enterprises. I do encourage them at smaller organizations though; if your security team is greater than 30 people in size and you have those unique problems we’ve talked about, it’s something where you can start to think about staffing it with future headcount.

The fact is, that it’s more effective to build security guard rails into tools that make people’s lives easier. The alternatives are doing manual security reviews or having people follow set process flows. These all have their place; it depends on the problems you are trying to solve and the effort required to fix them, but these sorts of enterprise-scale issues are almost always better solved with a dedicated solution that users love. Use the carrot, not the stick.