The Slowest GitHub PRs In Recorded History

The quickest method of deploying a code change I've ever encountered was shared with me by a staff engineer at an industry-leading messaging company.

She revealed that in their early days, her company would rsync new files directly to their production servers without even needing a restart. Thanks to the advantages of interpreted languages, the new code would be executed on the next loop that engaged it. This maximized deployment speed, and saved on rolling deployment server costs. (For the record, her company has since moved on from this deployment technique.)

I believe this to be the upper bound speed of real-company-production-server-code-change. You write, save, and rsync. File save takes 1 ms, let’s say, and 10 ms for the rsync to an EC2. Call it an 11-second code deployment (and probably hours to write). There are obvious downsides to the practice, but it undeniably maximizes speed.

This story made me wonder - how slow could one release code? It’s like the programmer's alternative to “Fast” by Patrick Collison.

The winners for slowest PR

As of September 2023, one of the slowest PRs (publish-to-merge-time) in Graphite’s 15 million PR dataset was a change titled “update in line 20” with a total time of 1857 days. The PR adds three lines, resulting in the all-time-high 1.6 years a line.

I expected the top slowest merged PRs to be nonsensical, but that’s not the case - all were real changes in name-brand repos. PRs like:

• “rely on non-rectangular patch paths rather than bboxes for legend auto-placing” - Matplotlib, 5.1 years

• ”Update Readme with composer command” - Imgix, 5 years

• “Create LICENSE” - Datadog, 4.2 years

(Note, these examples are from open source repos, but Graphite’s systems see similarly slow PRs in closed source codebases)

These brutally long PRs are fascinating to me. In a world obsessed with speed, we have these sometimes trivial changes getting stalled in the review flow for half a decade before still deploying. It’s like a package getting lost in the mail but still making it to the destination years late.

Move slow break everything anyway

If the upper bound (until I find better) for slow PRs is ~5 years, what about projects? What are the slowest-to-release software projects ever?

• [2 years] IBM's OS/360: This operating system for the IBM System/360 mainframe computer was developed from 1964 to 1966. It was an incredibly ambitious project for the time and ended up taking much longer to complete than expected, with costs ballooning to over $5 billion (in 1960s dollars). One reason for the overruns was the sheer complexity of the project. The OS/360 was one of the largest software projects ever undertaken at the time, and the technology to manage such projects was still in its infancy. Frederick P. Brooks Jr., who managed the development of OS/360, wrote a seminal book on software project management titled "The Mythical Man-Month" based on his experiences with the project.

• [~10 years] The WinFS filesystem, initially planned in the late 1990’s, made it to beta in 2005, and was eventually canceled around 2006. The idea behind WinFS was to provide advanced data organization and management capabilities to the operating system. Unlike traditional file systems that organize data hierarchically in files and folders, WinFS aimed to store data in a way that users could search and organize by metadata and relationships. Essentially, it would work more like a database than a traditional file system, making it easier to find, relate, and manipulate data in complex ways. Eventually, aspects of WinFS made their way into Microsoft SQL Server, .NET Framework, and other Microsoft technologies.

• [32 years] GNU Hurd: The Hurd is a collection of servers that run on the Mach microkernel to implement different features, intended to replace the Unix kernel. The development of the Hurd began in 1990, after the GNU Project, started by Richard Stallman in 1983, had produced all of the other major components of the operating system. As of today a 1.0 version of GNU Hurd has still yet to be released.

• [2 years] Apple's Copland: Copland was a project initiated by Apple in 1994 to create a next-generation operating system, a successor to the Mac OS. It was intended to introduce features such as preemptive multitasking and a modern microkernel architecture. However, due to numerous delays and internal issues, Apple canceled the project in 1996 before it was ever released. In the aftermath of the Copland debacle, Apple decided to acquire NeXT, the company founded by Apple co-founder Steve Jobs after he had left Apple in 1985. The NeXTSTEP operating system developed by NeXT formed the basis for Mac OS X (now macOS), which eventually became the successor to Mac OS that Apple had sought to create with Copland.

• [15 years] The video game "Duke Nukem Forever” was announced in 1996 and released in 2011 - taking a total of 15 years. The first major snag came with the decision to switch game engines. The Quake II engine, which was initially used for the game, was discarded in favor of the Unreal Engine in 1998. As new technologies and game engines emerged, the development team was tempted to incorporate the latest features, causing the project to be perpetually stuck in the development phase. This cycle of perpetual updating without completing became something of a running joke in the industry, with Duke Nukem Forever receiving Vaporware Awards from Wired News for multiple years.

• [5 years] SCO's Project Monterey: Around 1998, IBM, SCO, and Sequent designed a project to create a unified Unix for the Itanium architecture. Monterey was designed to be a high-performance, scalable, and industrial-strength operating system for business-critical applications. It would include a 64-bit system capable of supporting very large databases, while also maintaining backward compatibility for 32-bit applications. However, SCO eventually filed a lawsuit against IBM, alleging that IBM had moved proprietary code from Project Monterey to the Linux kernel. After the filing of the lawsuit in 2003, the project was never released.

• [5 years] Microsoft Windows Vista: This is one of the most infamous examples of a delayed software project. It took more than five years to develop, which was considerably longer than any previous Windows operating system. Originally codenamed "Longhorn," Vista was announced in 2001 but wasn't released until 2006. It underwent significant changes during its development, including a complete "reset" in 2004 to focus on security features.

Leaving stagnation in the past

There’s a commonality across these slow software projects: they’re relatively old. Modern engineering projects have for the most part learned their lesson. Ship faster and iterate, less the project dies before ever making it into the hands of users. Compare Duke Nukem Forever to the game Baulders Gate 3, which has been in development since 2018, but has leveraged early access since 2020 in an effort to develop publicly and more iteratively.

While it's tempting to marvel at the tortoise-like pace of certain PRs and software projects, the trend is clear: the code review process is speeding up, both in open-source and proprietary ecosystems. This acceleration isn't just a fluke; it's the result of innovations in CI/CD workflows, build tooling, code review, and other DevOps practices. And the state of the world is still evolving!

Tools are being created everyday that help push engineering productivity even further; tools like Graphite. With emerging features like in-Slack reviews, AI-powered comment suggestions, and stack-merge functionalities, Graphite is set to continue the trend of reducing iteration times while making engineers' lives easier. While we may still occasionally encounter multi-year PRs that laugh in the face of urgency, the future of software development is undoubtedly set to be faster.

To get started with Graphite, signup for our free waitlist today and leave multi-year PR's in the past... where they belong.