The Future of npm

Thursday, January 23, 2014

I have been involved with Node.js for over four years. I was the 8th contributor to npm not long after we all decided that npm had won the package manager for node discussions. Anyone who knows what kiwi is knows what I am talking about. I say "we all" because back then I could actually count the people who were intimately involved with the community. Now, only a short while later I see a massive community of developers who both love and depend on these things I've had the privilege to help grow to what they are today.

Yet even with all of the great changes, one has to remember the values and fundamentals that got us here. The key one of those being openness with a sprinkle of anarchy. Pushing the decisions to the edge of the graph: to the community members themselves. Node grew because it empowered individuals to do more with less. To encourage fundamental concepts like writing modules and piping streams instead spelunking through a monolithic, obtuse API. Help developers put the building blocks together and their applications will almost builds themselves.

How these building blocks are distributed is as important as the functionality they provide. That "how" is npm, which has had two years of 10x growth in a row with no signs of slowing down. The first 10x year is what caught my eye at LXJS in 2012 when I spent my first quality time with Jeff Jackson and Jason Smith who made up Iris Couch: the company that ran the npm registry at the time.

That growth is what motivated the company I founded, Nodejitsu, to acquire Iris Couch in May 2013. Between May and November 2013 downloaded from the public npm registry tripled from 42 million downloads to 153 million downloads. And that growth has precipitated even larger changes, as growth often does. Since the morning in mid-November that led to #scalenpm, when Jason Smith and I were woken up by the third in a series of really bad npm registry outages on our way to CouchDB Conf in Vancouver a lot has happened. These are all facts:

  • The Node.js community outpoured support to make the registry better by donating $326k to Nodejitsu through #scalenpm.
  • The public npm registry is now fronted by Fastly and Joyent's Manta service and still backed by the CouchDB servers run by Nodejitsu.
  • The npm project has lost (hopefully just temporarily) one of its most prolific maintainers.
  • Isaac has stepped down as the head of the Node.js project to start npm Inc. with very little details about what it will do.
  • Nodejitsu has announced our private npm registries for teams backed by smart-private-npm, and CouchDB at http://registry.nodejitsu.com

This rapid set of changes brings to a head the question: "what is the future of npm?" I don't have all the answers to that and my efforts to help bring transparency have not received much response from the npm team. What I do know is that Nodejitsu remains committed to the community through open source, which was the central motivation for announcing http://registry.nodejitsu.com

But it's more than just a commitment to Open Source that is in our DNA.

  • Every major npm outage since #scalenpm has been related (either directly or indirectly) to Fastly or Manta. So the changes have not gotten us more stability.
  • Relying on proprietary vendors like Fastly and Manta for such an important piece of infrastructure goes against the open source philosophy that built it, leading towards vendor lock-in.
  • Nodejitsu’s running of the public npm registry is supported by a well understood and sustainable business model of hosted and on-premise deployments that has been replicated in the past by other open-source projects. Not by mysterious yet to be determined revenue streams.

We are trying to make clear what is undoubtedly murky around npm. Next Monday (the 27th), the January numbers for #scalenpm will be public, where we've already been able to optimize some costs. A central one of those costs being over 140TB of monthly bandwidth that putting a CDN in front of has not reduced dramatically because it is mostly CouchDB replication.

This bandwidth data shows that replication is a vital feature for the registry of the world’s production JavaScript. The replication solution that npm Inc. is currently working on involves relying even more heavily on proprietary technology (specifically Manta) for replication and distribution of the npm registry. This has two really negative side-effects for CouchDB and a fully Open Source npm stack:

  • The public registry at http://registry.npmjs.org will be metadata only, making CouchDB replication a second-class citizen.
  • It completely ignores finding a solution in CouchDB for those who want to run a "full fat" (or even partially fat) on-premise registry since the public registry maintained by npm Inc. will be a highly specialized Frankenstein of proprietary technology.

What is run in production at scale will always be the most streamlined solution. So these production decisions will force anyone that wants to replicate npm flawlessly themselves to purchase Manta from Joyent. This will only cause a larger disconnect between npm and the community it is supposed to service.

That is why I knew http://registry.nodejitsu.com had to exist. Beyond the obligation to give back to the community that supported #scalenpm, there had to be an alternative for those who believe the npm registry should always be made from 100% open source components. The servers at http://registry.nodejitsu.com will are powered by CouchDB, smart-private-npm, and overwatch (our CouchDB multi-master replication monitor).

The architecture of registry.nodejitsu.com

This is not without it's own challenges: CouchDB has to get better at replication and handling attachments. A 1TB registry.couch file will simply not work given the problems we see with it right now at ~200GB. I have had several exciting discussions with Jan Lehnardt (Vice President of Apache CouchDB at the ASF), Brian Mitchell, and Sam Bisbee (of Cloudant) about how this can be accomplished that make me very excited about the future of Couch.

I do not begrudge npm Inc. for what they are trying to do. They are a new company trying to find their way experimenting with ways to solve a hard problem. While a noble effort, it does not bring the stability and confidence into npm that it deeply needs now as it crosses the chasm into the Enterprise. I am also not implying any sinister ulterior motives from Joyent around Manta. It is an exciting technology that I am eager to leverage as a way to gain greater insights into the public JavaScript source code stored in the public npm registry. Joyent has also been a strong, impartial steward for the Node.js project, and a fantastic partner to Nodejitsu.

I’m simply concerned about the road npm is heading down because I see the dead end of vendor lock-in it leads us to. I hope you will join us at http://registry.nodejitsu.com.

npm config set registry http://registry.nodejitsu.com/