Whoops I shipped myself

I’ve come to realize the struggle as a Software Developer is not typically with building software itself, but with mediating the conflict between building reliable software and handling the pressing needs of the organization you work within.

I’ve worked on many projects that have aspirational designs of achieving “infinite” scale. Usually there were design documents specifying key CRDTs, many layers of caching/queuing/redundancy, differing levels of CQRS – all the fixins your DDIA heart desires.

None of these plans actually made it to Production as they were formulated. One project eventually made it most of the way to the design proposal. Enough that I feel good about it to this day.

I’ve been junior and senior on these projects. I’ve watched software architecture drift. I’ve been the source of the drift too.

Architectural drift usually originates in a fast approaching deadline. Something in the project takes longer than expected, causing you to cut corners in other areas to deliver without compromising the road map.

A really convenitent short cut is to build knowledge silos. Rather than distributing tasks evenly across the team, developers are preferentially given tasks based on if they are “familiar” with that area already. Or maybe they volunteer. Regardless, silos of expertise and knowledge form in the name of velocity. Conway’s law sets in. Quoth Wikipedia:

Conway’s law is an adage stating that organizations design systems that mirror their own communication structure. It is named after the computer programmer Melvin Conway, who introduced the idea in 1967.[1] His original wording was:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. — Melvin E. Conway

When the plan to deal with an unforseen issue or undesigned component is hastily created, the solution will naturally reflect the thoughts and feelings of those working on it. When silos form in the execution of a project, solutions will reflect those silos. And thus architecture will drift from design to resemble the silo.

I find the last “variation” of Conway’s Law in Wikipedia to be the most useful in my life. This probably isn’t not a surprise since it comes from a book called Organizational Patterns of Agile Software Development:

If the parts of an organization (e.g., teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationships between organizations do not reflect the relationships between product parts, then the project will be in trouble … Therefore: Make sure the organization is compatible with the product architecture.

In short, make sure there is organizational appetite for the architecture you propose. Either scale your expectations to meet the funding or scale the funding to meet your expectations (lol). But to have a mismatch is to delude yourself. You will ship the lesser of the two.

I think its kind of easier to just accept this as a universal law and look for what you can do as an individual developer to leverage it. My current best solution is to develop and norm solid engineering practices with your org.

This is easier said than done of course. “Solid engineering practices” are subjective. There are lots of resource out there (which I intend to write future blogs about) as its a very lucrative industry . It is also mostly a social activity, which is a struggle for me and my kind. But, if you never develop a silo you can’t ship a silo!

So.. good luck with that lol.