According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});

It's pretty straight forward, but has the serious downside of relying on internal implementation details of Nest. If Nest decides to change how module paths are stored, they could do so at any time without bumping major version numbers.

There are two plugins I'm really excited about. The first lets you post to Mastodon directly from Obsidian. The plugin manages media attachments and thread separators (complete with character counts synced to your server's settings!):

https://github.com/elpamplina/mastodon-threading

A screenshot of this thread as seen in Obsidian.md, showing thread separators and character counts.

The other is a plugin which sends the current note's raw data and attachments to a webhook:

https://github.com/Masterb1234/obsidian-post-webhook

Put together, these drastically reduce the amount of code I need to write in order to fully own my stuff, while keeping is centralized, easy to access, and not locked in some piece of shit VC-backed silo.

Before my time at Splice there had been a few attempts at building a design system. In the last attempt styles had been audited and SCSS modules were created to encapsulate them. A backlog captured everything to be migrated. These styles were copied to a standalone repo and npm module to make them consumable by any application. There was even a sort of storybook-like preview site to show these styles decoupled from an application.

What Splice had at that point was a collection of tools, but nothing tying those tools to processes, nothing driving individual contributors to adopt them or participate in their development, nothing connecting their development to their visual origins within the design team. These things had the potential to help solve some software challenges associated with building user interfaces, but they did not add up to a system. What was lacking was a broader view of the challenge at hand, and without that broader view success was elusive.

Aside: I Wasn't Hired to Build a Design System

I was hired into Splice's Search & Recommendations team. I started on a Monday and the following Friday the team was dissolved as part of a broader engineering reorganization.

😬

My first project on my new team was to build the new "Pack Card" implementation, and the task was surprisingly difficult. CSS was everywhere: leaking out of global stylesheets associated with an embedded legacy AngularJS app, sprinkles of Bootstrap, a collection of SCSS selectors in a "splice-styles" directory, and in individual CSS files associated with specific Angular components.

Splice's current design system was motivated by the situation I found myself in. There was no obvious entrypoint into how to build Pack Cards in a coherent way. I could have just crammed some CSS together and picked up the next ticket - nobody would have batted an eye – but stapling yet more onto an incoherent system is something I have a hard time bringing myself to do.

I took this as an opportunity to not just solve the immediate need of shipping the "Pack Card" redesign, but to explore the problem space and prove out a solution to Splice's systemic challenges for designing and delivering user interfaces. The problem wasn't how to build the Pack Cards, it was to figure out how to make it less painful to build at Splice in general.

Design System Tunnel Vision

In many organizations the "design system" describes a set of artifacts which are owned by either of the Design or Engineering teams. The owner is primarily concerned with their own team's needs – other teams must extract what they can from that to inform their own work.

For instance, if the design system is wholly-owned by the Design Team, then the artifacts are often a set of PDFs (or PSDs, or Figmas) describing in great detail every aspect of how things should be implemented, be it brand colors, font families, font sizes, design patterns, or anything else all the way to conventions around language and tone.

In that context, the engineering team takes these specifications and builds implementations which reflect them. The code implementations may or may not encapsulate the design system in a set of tools to reduce work. The focus here is first and foremost on the specification, so let's call this approach the "Design Specification" approach to Design Systems.

On the other hand, if the design system is wholly-owned by the Engineering Team, then the artifacts are usually some sort of component library (maybe a some React or Swift components, maybe some CSS specified in BEM, etc). The focus is on engineering efficiency, and ensuring that engineers can implement designs consistently without having to constantly rebuild things, and without having to have designers watch over their every move. Let's call this approach the "Component Library" approach to Design Systems.

Both of these approaches provide value to an organization. They reduce variance, they limit duplicated effort, they improve communication across teams. They're also both incredibly limited: they reenforce Design and Engineering silos instead of fostering continuous dialog and collaboration; they speak to one team's needs at the expense of the other's. And on the negative side, they often work as leverage for telling the other team "no" without providing a way for that team to feed requirements into it.

Design System as Organizational Process

What if instead of describing a set of artifacts, we took "Design System" to mean the system by which design goes from concept to delivery, including a continuous feedback loop between Design and Engineering which facilitates product evolution? This changes everything about how both teams approach their work.

"Design System" is now a higher-level concept which encompasses all of the artifacts previously identified, but also describes how those artifacts relate to each other, each individual's role in driving their evolution across each layer, and how to affect change on those artifacts in a coherent and collaborative way.

This approach to a Design System is more comprehensive and spans more of the organization, and consists of multiple complimentary "layers".

Layer One: The Pattern Library

The Pattern Library is a set of technology-agnostic Design and Functional Specifications which describe repeatable patterns (colors, fonts, sizes, buttons, forms, cards, etc) spanning the "atomic design" nomenclature. Primarily owned by the Design Team.

The pattern library will describe design patterns broadly, going into detail about how they are applied differently based on context. A carousel design pattern might specify the overall look and feel, along with the kind of content it can contain, how "previous" and "next" behaviors work, how it behaves in mouse vs touch contexts, and how it may change based on context, such as if it's on web, iOS, or Android.

The pattern library usually exists "pre-implementation" in something like Notion and/or Figma, where general discussion and exploration of the pattern can take place, and where individuals can be directed for their own education about what currently exists and why.

That "why" is critical because the pattern library helps us constrain the number of design patterns, and this improves communication with the customer. For example, if we have a Video Card design pattern, that card is the design language within our product which our customers see over and over again, and for which they will gain an understanding and set of expectations. If any product team needs to show a video they will be able to rely on those expectations for interacting with the customer. If they go their own way and make arbitrary decisions about how to display a video, then they're going to have a much harder time communicating with the customer. This is why constraining design patterns is good.

The pattern library helps us inspect the natural inclination to assume our use cases are unique. When new use cases arise they can be held up against the pattern library either be thoughtfully rolled into existing design patterns, or become new design patterns if they truly are unique.

Layer Two: Component Libraries

Component Libraries are concrete implementations of the Pattern Library in specific technologies (React, Angular, Flutter, Swift, Webflow, etc). Primarily owned by Engineering Teams. There should be nothing in a component library which doesn't also exist in the pattern library.

Component libraries are the bridge between the pattern library and an organization's various products. They contain functional implementations of design patterns which can be assembled to realize product features efficiently. When an engineer is tasked with building a feature, this should be their first stop, and when they come up against the edges of a particular library's capabilities, they will contribute back to it to more fully realize a component as a fully fleshed-out design pattern.

Component libraries are the layer at which engineers participate in the design system feedback loop to bring engineering knowledge back to the pattern library. Engineers will have deep experience in their particular stack, and they can use that to evolve a design pattern to be more fit for purpose, rather than bolted on. For example, the engineer may raise up requirements around assistive devices (ie, alt text, focus, etc), or contextual UI expectations associated with a particular device (ie, native iOS select interfaces, etc).

This is also an opportunity to push back on design pattern creep. If a feature's design specification looks a whole lot like another component in function, if not form, an engineer should be encouraged to bring that back to the pattern library rather than just implementing anything they're handed. This brings us to...

Layer Three: The Feedback Loop

Pattern and Component Libraries are both points within the design → engineering → design feedback loop. The Feedback Loop is what makes it possible to gain the most value out of a Design System because it allows designers and engineers to participate in the evolution and refinement of the artifacts coming out of the system.

In a well-functioning feedback loop designers and engineers are learning from each other, gaining empathy for each other's concerns, and related work outside of their local silos. The same is true between engineers across stacks as well – if iOS and Web engineers are both contributing feedback to the same design patterns they will have more opportunities to understand how those design patterns apply across different technology contexts.

One mechanism for facilitating the feedback loop is to have a regular (optional) cross-disciplinary Design System Office Hours where engineers and designers can bring questions, solicit input, and help others.

Layer Four: Accountability

Individuals will always prioritize what they are accountable for – this is often the thing which kills an organization's design system aspirations.

Design systems produce capabilities, not features. They slow down the delivery of the current feature (the team has to first identify and extract the capabilities), but they make all subsequent features vastly easier to deliver because the team now has a set of capabilities to draw from. If a team is only ever accountable for the delivery of the current feature, then the incentives just do not exist to build a design system. This is where the organization's leadership comes in.

Design and Engineering management must build participation in, and contribution to, the Design System into their evaluation frameworks. Not just "what features did you deliver", but "what capabilities". During the review cycle designers and engineers should be able to point to an array of others who have been able to deliver on top of their own work within the design system, and they should be celebrated for it.

From Proof of Concept Component Library to Organizational Directive

Getting from that first desire to make it easier to build things like Pack Cards, to transforming the way Splice approached the idea of a design system, and software delivery in general, was not a straight line. I honestly kind of fumbled into it with a lot of luck and good timing.

I produced that first component library proof of concept in collaboration with my peers. It worked great, but it was still just a singular tool, not a system. A few months later the engineering reorganization continued as a new CTO and Director of Engineering were brought on. Their primary focus was on making it possible for Splice to more predictably deliver software, and unlike many new leaders, they opened the table to any and all ideas for how to achieve these goals. This ended up being the opportunity we needed.

An RFC was written up, senior leadership approved, and individual contributors were largely on board. Myself and then Design Director Jay Schaul were tasked with leading the initiative and helping people from each of our respective domains begin the process of starting to think and work in this way.

Three key choices ended up being effective leverage for getting the organizational process up and running, acting as an onramp for individuals from across the organization into this new process:

1. On the design side, Jay created the working framework for the Pattern Library which consisted of an organized area of Figma, where existing work was brought into the pattern library structure, and a "proposal" area of Notion, where design patterns could be proposed using a starter template with standard areas to be fleshed out by contributors.
2. On the engineering side, I created an Angular library where components and reusable CSS could live and be consumable from any of our web-based applications, ported our nascent SCSS modules into it, moved our Pack Cards into it, and stood up an instance of Storybook to make it possible to view everything in the component library and to operate as a component development environment decoupled from the applications.
3. Jay and I started optional bi-weekly design system check-in meetings (aka, design system office hours).

Progress, Setbacks, and Bus Factors

Much progress has been made: we're able to build features faster and more consistently, to reuse code across client applications, and to release those features across clients without significant additional effort.

But we're far from done. We have yet to fold in our mobile team, and as of late it feels like our feedback loop is fraying at the edges. The work now is to operate on the system to address its shortcomings. Much like we evolve design patterns, we will evolve our Design System to account for new organizational requirements without having to start from scratch.

Someone once said to me that for a process or system be sustainable it must not depend on the presence of any particular individual. No employee is forever. We need to continue to build the organizational muscle memory to make our Design System process more resilient to staffing changes, and less reliant on specific individuals reiterating and reenforcing the underlying principles. If we can achieve this, then Splice's Design System will truly be a success.

Acknowledgements

Splice's design system is built from the contributions and work of many others across the organization. Notable contributors are Jay Schaul, Kim Goulbourne, Leigh Caplan, Andrew Cornett, Lara Warman, Florida Elago, Jessica Eldredge, Matt Claypotch, Sue Lockwood, and more.

"Our home-rolled CI is too slow, let's switch to Jenkins!"

But then Jenkins just runs a bunch of shell scripts outside of source control, build artifacts aren't taken advantage of, and a single build runs npm install four to six different times.

"Our users are getting runtime errors – let's adopt Sentry!"

But then nobody ever checks Sentry, nobody is accountable for error metrics, and our users continue to get runtime errors.

"Nobody knows what work is in flight or how long it's going to take to ship a feature. Let's switch to [GitHub Projects/Jira] from [Jira/GitHub Projects]!"

And after the switch teams still can not communicate dependencies or estimate delivery timelines.

"[Payment provider] is always going down, let's switch to [other payment provider]!"

But the new payment provider fails in exciting new ways, and the payment error queue is as bad as it's always been.

We often reach for tools to address the failures we encounter when we should start by examining our systems.

Maybe it isn't that our CI is too slow, but that it is performing the wrong tasks at the wrong time and needs to be broken apart into a system of contextually-appropriate workflows which support team velocity?

Maybe there are so many runtime errors in production because nobody is accountable for those metrics? If a team's success takes into account error metrics, then the system of accountability will change how individuals prioritize their time and select their work, leading to trade-offs which will favor fewer runtime errors in production.

It's easy to go through the motions of agile without understanding how all of the pieces fit together into a system with specific characteristics. Maybe the kanban tool we already have would be more useful if our Project Managers and Lead Engineers went through agile training and actually understood the purpose of all of the rituals, and could see meaning in cumulative flow diagrams?

Maybe our current payment provider is fine, we just have to rethink the system we've built around it to be asynchronous, resilient to failure, and automatically retry instead of calling straight through to the third party? Every third-party provider is going to have downtime, and our systems need to be designed with that as an intrinsic characteristic to be managed locally.

Switching Tools is Expensive 💰💰💰

The idea of shiny new tools without all of our current problems can be enthralling, but switching tools is often counter-productive.

New tools require time and money to adopt (adjacent tools will need refactoring, integrations will need to be migrated without downtime, people will need training, and at the end artifacts of the previous system will almost inevitably persist), and all too often the failing characteristics of the system remain: build tools are still slow, teams continue to ship runtime errors, delivery timelines are a mystery, and somehow the payment error queue is longer than ever.

Requirements → System Design → Appropriate Tools

Image by Seattle Subway

I'm not at all saying that switching tools should be avoided, just that tools alone will not magic away the failings of the systems they operate within.

Transportation systems are a good example: a fleet of sleek self-driving electric cars sounds amazing, but they will never solve traffic congestion or get us closer to vision zero because Teslas and Volts do not address the failings of transportation systems. To address our transportation system's failings we need focus on the transportation system itself, much like Seattle Subway is doing, and then let that system design guide us in our infrastructure investments.

If our organization's product delivery system has teams working on features with dependencies across teams, then our kanban tool needs to support linking epics across teams. If our technology stack cannot guarantee that runtime errors are not shipped to production, then our system for accountability must include monitoring and measuring the rate and frequency of these errors, and evaluating our team's against those metrics as indicators of team and individual success.

We start by understanding our requirements.

Those requirements should lead to a thoughtful system design exploration.

And once we have crafted a system which will support our needs, only then can we select tools which fit into and support that system.

Our release process was stressful, time-consuming, and error prone.

This was the process: notify the team in Slack of your intention to release so nobody else would start their own release, merge your pull request, tag master via GitHub Release, copy the GitHub Release url and paste it into the Customer Relations Slack channel, find the correct "build" job, click through a few screens, click "build" and wait for it to build (about 15 - 25 minutes), deploy the build to dev, find the correct integration test job, trigger it and wait for it to complete (10 - 15 minutes), run integration test failures by QA to make sure they're not blockers (QA may be busy so you might have to wait until someone has time) ...

Are you still with me?

...find the production "deploy" job, click through a few screens, select the correct build artifact from the original job, click submit and wait for the deployment to complete, copy the url of the build log and paste it into the GitHub Release, do a smoke test in production, give an "all clear" message to the team so they know they start their own release if they need to.

That's if everything went well. If something went wrong then it would expand and possibly start all over.

This process would run up to three or four times per day by various engineers on the team, and it was error prone. We could easily click the wrong build job link, sending the incorrect build artifact into Production – it happened more than once. The process took forever so engineers would spend half of their day babysitting a release instead of working on product features.

And the team would spend a lot of time to coordinating:

Solution

Automate build, testing, and deployment as a side-effect of source control interactions and build timers. Get humans out of the process of releasing so they can focus on building the product and participating in peer review.

The new process: Open a pull request, get peer approval, get QA approval, merge, move on!

Every day at 9AM:

When the build is ready, the day's release marshal is notified and they're given a link to the "Proceed" screen. The process is async, so if they're busy they can leave the paused build until their schedule allows. They can hold the build altogether if there are any risky factors (such as a larger ongoing incident). If everything's in order, they click "Proceed" and get back to work.

Humans are responsible for writing high quality code, reviewing code from their peers, and merging completed work. Automation is responsible for everything else.

Outcome

• The release process went from a mountain of manual steps to one (clicking "Proceed")
• Time to release went down from 45 minutes to 15 minutes, but humans are only involved for about 3 of those minutes
• Removed all opportunities for user-error
• Reduced stress
• Increased engineering productivity
• Changed how other platforms within the organization approach CI

Every engineer contributing to our projects came with a wealth of experience which informed their preferences around code style and formatting. Each arrived at their preferences through trial, error, success, and failure. Unfortunately each arrived at slightly different conclusions. This was not a sustainable long term approach for a growing team.

The organizational problem with allowing divergent style preferences within a team is that it results in less productivity and a lot of avoidable low-level conflict. People change code unrelated to their work, nitpick pull requests, and expend energy on minutia instead of improving the product.

The technical problem is that this behavior introduces avoidable bugs as code is unnecessarily churned while pull requests become difficult to review as much of the patch has no bearing on the feature being built.

The emotional problem is that people get attached to their own ways and don't want to be forced to follow arbitrary rules which they don't agree with.

Solution

In order to support a large number of engineers contributing to a shared set of projects while maintaining consistent style and formatting we had to lean on automation. But before we could lean on automation we had to agree about the problem.

I opened an issue to discuss the problem, laying out the challenge as I saw them. I proposed that we adopt ESLint configured with an extended¹ version of AirBnB's ESLint configuration (the extensions would be made to bring it inline with the team's general preferences). The proposal explicitly stated that there was no de facto "rule decider"², but that anybody could propose modifications. Whether or not to adopt the proposed rule changes would be up to consensus.

There are trade offs. People would have to ensure their code is linted before it can be merged, which is a potential point of frustration. This is an understandable fear: nobody wants errors coming out of nowhere keeping them form getting their work done. The trade off is that what they get out of this compromise is that nobody will nitpick their code – if it passes the linter then style and formatting critique is off the table. A big upside to this is that we can automate the linting work away through editor integration and auto-formatting.

The team agreed. I helped people get their editors configured to automatically surface (and often fix) lint issues. I published @simple/eslint-config, and integrated it into all of our web applications (no small task).

Outcome

The outcome has been overwhelmingly positive. Pull Requests no longer get stuck in review for days. There is no back and forth churn of style and formatting as different people work on a file. When linting issues make it into a Pull Request (which is rare, given the widespread use of editor tooling), our CI reports those failures back to the Pull Request before it even gets into peer review.

Addendum

The Prettier auto-formatting tool has come along in the time since we adopted ESLint. I believe we would have used Prettier for much of what we use ESLint for today. We may yet do that, but our current ESLint setup has addressed all of the productivity issues so there's no pressing need to change tooling yet.

Notes

1. We couldn't go with the stock version of AirBnB's configuration for two reasons. The first was that just didn't agree with some of them. The second was that some rules would have required a level of refactoring which would have had a significant chance of introducing bugs, and in doing so hurting user experience while taking engineering resources away from product development. If the some of the stated goals were increased productivity and happiness, then adopting the default configuration wholesale would have worked against those goals.
2. Why no de facto rule decider? My experience in every organization has been that when someone assumes this role they abuse it and end up holding back the entire team. People get caught up in the dogma of their craft and use their clout to push back on changes which make their rule set irrelevant, such as shifts in the industry away from the technology associated with their expertise.

Our primary banking web application reached a point of unsustainability in that it had intractable architectural problems with no realistic path to improvement.

Client-side performance could not scale. The application's client-side performance was sufficient for users in their first couple of years of active use. As their account's data grew (up into the multiple megabyte range) performance degraded. The application was built with a critical architectural flaw: it assumed the entirety of a user's account data would be available within the client. Many features relied on this assumption (transaction search, reporting and graphs, transaction sorting, linking to a transaction from support messages, etc), and to break this assumption would require fundamentally re-thinking these features.

Bespoke in-house framework. The client-side application was built on a bespoke in-house framework similar to Backbone. Unlike Backbone it was undocumented, unmaintained, and had many fundamental problems.

• "Model" components made no distinction between "collections" and "objects", and had no schema. This made them confusing to work with as half of the API was irrelevant based on the data structure of its content.
• The HTTP transport component was chatty by default, so would fire off unexpected requests constantly. It used attributes of the associated "model" component to infer how to make requests, making it difficult to work with HTTP services which didn't operate under those same assumptions (which the case more often than not).
• "View" components were tightly coupled to "Model" components (a FooView would render a FooModel), making it difficult to build reusable UIs. For example, a transaction list View might contain multiple kinds of transaction-like Models, but would be hardwired to a single Model type. This kind of coupling required complex overrides of core framework logic to work around.

e lack of documentation made for a steep learning curve. New engineers would spend weeks running into confusing issues before feeling anywhere near confident in working on features.

Obsolete technologies. The framework was built on top of MooTools, which heavily modifies the native runtime environment by extending native classes and exposing a large number of globals. Using MooTools as the core made it difficult to use most other open source client-side technologies (which I'll talk about later). Aside from the technical issues, MooTools made hiring difficult. Nobody building state of the art web applications wants to go anywhere near MooTools.

MooTools used its own package manager called Packager, which was built with PHP. Packager was awful to work with so we wrote our own drop-in replacement in Ruby called Plums. Only one person on the team really understood how Plums worked, and nobody knew how Packager worked. Whenever a module packaging error occurred people would struggle for hours to resolve the issue.

I can't describe how bleak it is trying to build a state of the art product on these tools when open source alternatives are blossoming around every corner. The problems our in-house tools were solving had been solved over and over, and in better ways, than our own.

Poor separation of domains. The middle-tier of the banking application was a Ruby application. The Ruby application would then do its own work and eventually make requests to the backend services.

The fundamental problem was that the Ruby application should have been a lean proxy to the services, but was instead doing far too much work. Engineer would commonly re-write client requests at this layer before sending them to services, and also rewrite service responses before sending them back to the client. This was sometimes done to accommodate the inflexibility of the client-side framework, other times to get something done sooner rather than waiting on the backend team to have time to build it out in the services layer.

A side effect of this middle tier is that important details would often get dropped between the client-side and the services. For instance, HTTP headers set in the client were almost never sent to the services, which made many requests less efficient because things like if-modified-since couldn't be taken into account. A common issue is that the middle-tier endpoint for a specific request wouldn't have taken every HTTP response code from the service into account, so basic things like 429 Too Many Requests (aka rate-limiting) responses would appear in the client application as 500 Server Error, or get swallowed altogether and appear as 200 with an empty body.

The organizational impact of this middle-tier was largely unacknowledged but immense. We essentially had two APIs: one official API with documentation and processes for proposal, critique, and acceptance, the other completely ad hoc and undocumented. Instead of bringing API needs to the backend team, this team was self-servicing half-baked workarounds.

Humans and Feelings. Everything about the application was bespoke – many people in the organization had put months and even years into inventing it from scratch. Navigating how to speak to all of the above issues and proposing to replace it while honoring their work was a daunting task.

Solution

the things which were recurring pain points we all dealt with. We regularly had to write complex workarounds for using the same UI component in multiple places in the application. The process could take hours and often resulted in a long-tail of odd UI and state bugs. I built a very small proof of concept application which showed how React could render any component in any nested structure any number of times, and state changes would propagate to every instance based on the central state tree. That one little example made most of the team understand how much better things could be.

Explaining the Problem to the Company. Once the team was on board I put together a talk, "The Future of Our Web Applications", which was presented at our quarterly engineering conference. I spelled out in detail how much time and effort we were putting into dealing with bespoke technologies instead of shipping user-facing features. I outlined each of the things which were failing us, what we would replace them with, and how that would lead to more Product focus.

Incremental change and then rebuilding in parallel. Our goal was to incrementally rebuild the application in place so that we could continuously ship to users without blocking Product work.

We sliced all of the standalone screens and rebuilt them, so navigating between pages would be navigating between client-side applications. The vast majoring of the banking application, unfortunately, existed in the transaction sidebar (bill pay, ACH transfer, instant payments, support chats, etc). In order to incrementally rebuild this features we'd have to ship one sidebar feature at a time in the same client-side environment as the MooTools application. We ran into a wall of incompatibility. Some of the native browser objects React depended were modified by the MooTools runtime, so React would throw errors while rendering. Our in-house MooTools router fought with React-Router for control in unexpected ways. Making progress became impossible.

After getting stalled for about two months we made the decision to rebuild all of the sidebar features in one go and ship them together; our users would stop getting updates for a while, but they'd get the fully-functioning rebuild much sooner. This would also allow the entire team to return to working on new features sooner, given the slow pace of the incremental rebuild work.

Move functionality to the services. Many of the application's fundamental problems were solved by replacing our tools with React (+ Redux), etc., and using npm instead of Plums. The issue of a poor separation of domain was a different story. Nobody on the team wanted to continue making the same mistakes within our middle-tier Ruby application. As we worked through the rebuilt UI we collaborated closes with Data and Backend engineers to move functionality into the services. The Search feature got a dedicated backend service (so now mobile clients get the same search results as the web app), new transaction paging endpoints were created, all new middle-tier endpoints were lean pass-through proxies straight to the services, and some features got dropped because the cost to replicate wasn't worth the tradeoff.

Outcome

The new React + Redux application has proven to be a huge boon to productivity and overall happiness. We ship features and changes faster and with fewer bugs. Because we're using standard tools we're able to take advantage of the wider ecosystem, which has led to the adoption of ESLint, Webpack, charting libraries, validation libraries, and so on. These tools have helped us to ship higher quality code while focusing solely on user-facing features in support of the larger product vision.

According to git docs this is configured with the core.excludesfile.

To see if this is set:

git config --get core.excludesfile

If so, any pattern in that file will be globally applied. Unfortunately, I didn't check that before monkeying with things, so I have no idea if I've changed my system accidentally.

From the docs (which I finally found and read):

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

On my system:

• $XDG_CONFIG_HOME is not set
• $HOME/.config/git/ignore does not exist

So I think this means that I had no previously-configured global ignore patterns? Maybe???

In any event, I've now created $HOME/.config/git/ignore and configured git to specifically use it:

git config --global core.excludesfile $HOME/.config/git/ignore

I'll find out in the coming days if new files start showing up in my various repos when I run git status. 🤷

People don't like to leave a party
unless the house is actually
on fire. Even then, if the flames
are far enough away
to be pretty, they'll finish
their drink, take one more pass
at the hors d'oeuvres.
How things happen has always been
unclear. Hurricanes begin
in a place where no one lives.
Agents of the government start
to wear masks. Fascism is
a word my neighbors won't use
yet. They are following
the law, they say, and the sirens
are coming for someone else.

– CHARLES RAFFERTY

Prepare:

• Become a rapid responder in your community!
• Join with your organized neighbors!

RouterModule.register({path: '/foo', module: MyModule});

I'm in the middle of building some functionality which needs module router path prefixes, and from a centralized location. Unfortunately, Nest does not provide a first class API for retrieving them. With DiscoveryService it's possible to get path components from @Controller() and method decorators, but not RouterModule path prefixes.

I spent some time digging through Nest source code. There is a GraphInspector class, and that looked promising, but I couldn't find a way to make it useful here. There's also a RouterExplorer class, but it's internal.

In the end I was able to make this work by looking at the metadata Nest stores on modules. I eventually found metadata with the promising name of "__module_path__<bunchofstuff>", and on that is the path prefix!

What follows is my hack of a solution for gather module path prefixes using this bit of knowledge. First, let's declare a minimal module structure to prove out a solution:

import {Controller, Get, Module, OnApplicationBootstrap} from '@nestjs/common';
import {
  ModulesContainer,
  NestFactory,
  Reflector,
  RouterModule,
} from '@nestjs/core';

@Controller('controller-1')
class Controller1 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller1],
  providers: [],
  exports: [],
})
class Module1 {}

@Controller('controller-2')
class Controller2 {
  @Get('index')
  index() {}
}

@Module({
  imports: [],
  controllers: [Controller2],
  providers: [],
  exports: [],
})
class Module2 {}

The solution will involve:

• Waiting for the module graph to be completely built
• Traversal of the module graph
• Identify each module with __module_path__... metadata
• Pull out the associated path, filtering down the set of modules to only those with path prefixes

@Module({
  imports: [
    // Here we see each of our modules imported and registered:
    Module1,
    RouterModule.register([{path: 'module-1', module: Module1}]),
    Module2,
    RouterModule.register([{path: 'module-2', module: Module2}]),
  ],
})
export class RouteDiscoveryAppModule implements OnApplicationBootstrap {
  constructor(
    // Inject the global ModuleContainer so we can traverse modules:
    private modulesContainer: ModulesContainer,
    // Inject Reflector so we can get metadata:
    private reflector: Reflector
  ) {}

  onApplicationBootstrap() {
    const modules = Array.from(this.modulesContainer.values());
    const modulesWithPathPrefix = modules
      .map((module) => {
        // Use the native `Reflect` API to get all decorator metadata
        // on the modules's `class` (eg, metatype):
        const metadataKeys = Reflect.getMetadataKeys(module.metatype);
        // Look for a metadata key so we can use it to get path metadata:
        const pathKey = metadataKeys.find((key) =>
          String(key).startsWith('__module_path__')
        );

        // If we found a path key, pull the path out:
        const path = pathKey
          ? this.reflector.get(pathKey, module.metatype)
          : undefined;
        return {path, module};
      })
      // Filter out all modules with no paths:
      .filter(({path}) => Boolean(path));

    console.log(modulesWithPathPrefix);
  }
}

// Run the program:
Promise.resolve().then(async () => {
  const app = await NestFactory.create(RouteDiscoveryAppModule);
  app.listen(3001);
});