Should a software house still invest in technology?

I was reading an article on how LinkedIn uses Kafka to do its magic (at scale). Not a big surprise since they built Kafka.

The content of the article itself is not important. The point is that, as an architect, I found it very reasonable. I appreciated how clever and well thought out the solution is. I could almost see the amount of code that these guys wrote to make it real. The story reminds us that there is no such thing as a free lunch, especially when scale becomes a matter of quality over quantity.

We live in a world where every business problem has an OSS solution that magically takes all the pain away. With my manager’s hat, I think: where do we spend so much development time, then? Kafka is already scalable, reliable, and super-fancy, so why all the effort?

My architect side answers that I cannot deliver real software with no pain just by picking the latest fancy tool.

The reality is that OSS does not take the pain away. It just makes the complexity implied by modern requirements addressable in the timespan of a normal development cycle. That is, releasing it before it’s no longer useful.

A medium/large software (especially SaaS and multi-tenant) has very demanding requirements in terms of scalability, reliability, resilience, maintainability, security, user experience, and so on. The development community identified classes of problems and proposed solutions. Some are good, while others fade away leaving dated websites and a lot of sorry users.

OSS products like Kafka are the response to single classes of problems, not to my larger problem. To achieve my goal, I must design my architecture, fill the spaces with the right OSS blocks, combine and configure them in the right way, and write a large amount of code for the plumbing. Of course, there is also the code that implements the features I need for my product (which is perhaps the only part that truly belongs to the product value chain).

This LinkedIn blog was just a glimpse at one aspect of the system. Then they (like all other architects) must deal with security, UX, automation, and many other facets of development. It’s true that each can be augmented by some open-source product, but it’s not like Lego. Putting everything together comes at a cost.

All of this leads to the reason why a large software company should continue investing in technology. It’s the same reason why a car is not the sum of its parts, which are built by several different manufacturers.

In the special case of software, a large company needs infrastructure. This is still based on a mix of OSS and proprietary code, but it is designed to solve more articulated problems (like platform management, trade management, pricing, risk calculation, and so on) in a uniform way across the organization.

Software houses gain a competitive advantage because this reduces the variability with software design, improves productivity and fungibility of resources and helps guarantee a more homogeneous approach for different lines of products.

The organization that creates the best abstraction and has developed the connective tissue between the different parts (OSS or proprietary) in the most efficient way has effectively “taken the pain away” and can concentrate on writing software that serves a purpose.

The result of this activity is what enables the majority of developers to focus on business features, while a minority of them keep the “framework” relevant.

This also explains why companies contribute to the open-source community. The valuable IP is not in the infrastructure, but in the way it’s used and assembled. Meanwhile, the cost of solving individual technical problems—like streaming messages as in the case of Kafka—can be offset by leveraging a larger community.

Of course, my view comes from the perspective of over a decade of looking after ION’s core technology.

Like in other industries, the expansion of fintech also creates a push for consolidation that has the potential of making customers’ lives easier, but increases the complexity for the organization. ION is an example of this. The variability in the approach (processes, technologies, branding, and many other aspects) together with growth may prevent a company from expressing its full potential, regardless of whether the growth is organic or via acquisitions. It’s the size that requires a divide and conquer strategy.

The software houses leading the market consolidation, and that inevitably embrace various SaaS models, will face the necessity of creating, maintaining, and evolving a ubiquitous infrastructure layer. Success depends on finding the delicate balance between leveraging diversity to boost evolution and letting complexity get out of hand.