How Yarn Workspaces Accelerated LinkedIn Code Delivery
Here’s a familiar story: A monolithic code base grows and grows to meet the demands for new features and an enlarging user base. Unclear ownership, increased build times, a disproportionate number of testing per application are but a few of the headaches that ensue.
So the story goes for how the monolith outlived its useful lifespan. And one by one, unrelated product areas were decoupled and a distributed system was built in its place. Due to a continually growing codebase, the team for LinkedIn Talent Solutions, migrated the site’s front end from a monolith to a distributed system, then to a monorepo. Instrumental in this migration was the use of Yarn workspaces, according to a blog post.
“For the past few years, we had been eyeing the potential of workspaces, a technology offered by several package managers including npm, pnpm, and yarn (our current package manager of choice) that enables first-class support for a new type of monorepo,” wrote Jordan Hawker, Yarn Staff Frontend Engineer.
Yarn workspaces significantly decreased the time it took to ship changes and upgrade packages, however, the Commit-to-Publish timeline crept up again. LinkedIn Talent Solutions eventually migrated back to a distributed system while continuing to unitize yarn workspaces which allowed them to remain at a 97% time savings.
An Unreliable Dependency Graph
An unreliable dependency graph caused a few significant problems. It led to version upgrades and migrations with boilerplate changes repeated up to 70 times depending on the number of impacted packages. yarn-link, a popular dev tool that connects local packages enabling developers to run gore across projects with unmerged changes, aided developers in end-to-end testing was made unreliable.
The dependency graph also made a large impact on the Commit-to-Publish timeline, the time it takes pull requests (PRs) to reach their deploy pipeline. A single change required multiple tightly-coupled PRs across the ecosystem. Features went through several cycles of tooling pipeline. and the deployment pipeline could take as long as ~39 hours. Developers were waiting multiple business days to ship a change to production.
Enter: Yarn Workspaces and a New Type of Monorepo
Unlike the monolith of the past, workspaces house multiple distinct projects that cross-reference one another within the same repository. This eliminated the need for yarn-link while maintaining clear code ownership and build isolation. All code changes within a package are immediately available to the application and other consumers in the workspace.
There was no longer a need to version or publish independently after moving their repositories to workspaces a huge benefit of this meant LinkedIn no longer needed to run test suites with every shipped change. By co-locating the code within a single repository they were able to run package tests at the same time as application tests, ensuring compatibility with one another. This approach could cut the time automatically upgrading a package to less than a day.
Creating a Yarn Workspace
LinkedIn engineers created a script to automate the migration of each repository. With the repository name, the script cloned it into a temporary directory and removed any unnecessary files (e.g. .gitignore, .npmignore, yarn.lock). Next, it leveraged git mv to move the times to their new workspace destination and added the cloned directory as a temporary remote then merged the external library into the application’s git history with the —allow-unrelated-histories-flag.
The new package is registered by adding it to the application repository’s root package.json, making sure to declare any additional dependencies that were previously transitively required. LinkedIn also adopted a strategy of syncing dependency versions across the entire workspace, ensuring every library was built and tested against the same packages deployed to their vendor bundle in production.
Rethinking Their Builds… One More Time
But that was not the end of their story because the codebase continued to grow. In just one year, the length of a single test grew from ~ 45 minutes to nearly 100 minutes. With all of this previous experience behind the LinkedIn engineers, the engineers already knew the current trajectory was unsustainable.
And back to a distributed system, they went! Test runs were spawned on operate machines for each library and the application itself since every package already supported being built and tested in isolation. This approach still had a bottleneck — it was always going to be the slowest individual build in the system.
In this particular system, it was the application suite. To reduce impact, all unrelated application build steps were distributed. Execution times were cut by 50%. Even as more workspaces were migrated into the repository, build durations remained consistent, even with a slight decline as they continued to increase their capacity for distributed builds.
The second half of this solution was a change in how tests were executed. Core applications and tests for migrated packages were included in the repository. The core application tests were executed in every build but tests were no longer run on specific packages unaffected by changes. Tests will run on affected packages and any packages affected by that package.
Today
Presently, 28 repositories are migrated into Yarn workspace and a sizable impact is seen in developer productivity. The original goal of 95% improvement, originally considered ambitious, in code delivery was met by the end of the six-week project as the recent six-week trailing Commit-to-Publish P90 for the application was 70 minutes meaning there was a 97% reduction in the timeline. There is no longer a need for over 2,000 monthly version upgrades which significantly reduces the strain on automated tooling.
Additional benefits include improved code discoverability by co-locating code within a single repository offering more opportunity for code mod application and simultaneous migrations across multiple libraries. Dependency freshness is more reliable due to the alignment of all packages to the same version. LinkedIn cites the workspaces and consistent developer environments to engineers gaining confidence and peace of mind.
The developer satisfaction survey results are illustrated in the image below.
Conclusion
The team behind LinkedIn Talent Solutions credits Yarn Workspaces with the evolution of their application architecture without having to sacrifice the benefits of their multi-repo strategy, restructuring their test configuration to leverage distributed builds yielded more than a 50% improvement in test durations for pull requests without any regressions in coverage as they worked to further reduce the amount of required testing. Reducing the Commit-to-Publish times by 97% made great strides toward their focus on accelerating code delivery.
