Npm 5 changes to npm link.

Npm has always had the ability to do npm link. This utility is meant for developers who are working on a module that is designed to be used in other apps or modules. Basically it allows you to symlink the module you're working on into the node_modules folder for another app or module so that you can make changes to the module without having to keep reinstalling it over and over again into the parent app just to test the module. The process is a quick 2-step process.

Step 1)

$ cd path/to/yourModule
$ npm link

This creates a symlink from your module's directory to the npm's global node_modules directory. If you were developing a CLI tool then this step is all you need to do and your CLI tool should now be usable in the terminal. However if your module is intended to be installed into another app then there's one more step.

Step 2)

$ cd path/to/yourTestApp
$ npm link yourModule

This creates a symlink from the global symlink you created in step 1 to yourTestApp's node_modules directory. In npm 2 and lower this worked almost flawlessly. You could continue working on your module and then just restart your test app and instantly test the changes you made to your module without having to install the module again. You could even run npm install in your test app and npm would detect symlinks in your app's node_modules and skip over them so as not to overwrite the symlink.

I've been following an issue on the npm bug tracker for a while now. Basically, when npm 3 released they made a somewhat major change to the way dependencies are installed. By default npm 3 and higher now flatten the dependency tree as much as possible. So if you have a dependency and that dependency has five other dependencies npm would hoist those five other dependencies up into your app's root node_modules directory instead of installing them in your dependency's node_modules directory. There are exceptions to this, such as when two dependencies depend on different versions of the same dependency, but for the most part the dependency tree gets flattened into the top-level node_modules directory.

This flattening solved a whole lot of problems in other areas but it introduced one new problem. If you npm link a dependency and then run npm install in your test app, your test app will steal your linked module's dependencies and hoist them into its own node_modules directory. This causes all sorts of headaches because since the linked dependency isn't actually inside the test app's node_modules it has a hard time requiring its own dependencies because from the module's perspective there is no parent node_modules directory. What used to be a really convenient way to test a module in tandem with an app became a really tedious and frustrating hassle.

The issue persisted through npm 4 but now we are on npm 5. After reading through the issue and all its comments I began to realize something had changed in npm 5 so I decided to experiment on my own.

First I recreated the original issue using npm@4.2.0.

You can see in the clip above that running npm install within module B does indeed steal module A's lodash dependency and hoists it into its own node_modules directory.

Since people were still complaining in the thread about npm link being broken in npm 5 I went ahead and tried to reproduce the error in npm 5.

At first the results I got were very bizarre to me. Running npm link ../A worked fine; a symlink was created and A's dependencies seemed in tact. But then running npm install ../A after that surprised me. It kept the symlink intact but it nuked A's node_modules folder. I thought I had duplicated the original issue but then I looked in B's node_modules folder and lodash was nowhere to be found. At this point I was very confused, but I noticed something different in npm 5 that I hadn't seen in npm 4 or earlier.

I noticed that when we ran npm install ../A it modified package.json, but with something I hadn't seen before: "A": "file:../A". See, in npm 4 and earlier if you ran npm install ../A it wouldn't treat it any differently than it treated a package from the registry. The only difference would be that it would copy the module from the file system instead of the npm registry. I remember this because if you did npm install followed by a file path it would add the fully qualified file path to package.json: "A": "/Users/chev/code/sandbox/A.

The fact that it now adds a relative file path to package.json gave me a hunch.

Sure enough, running npm install ../A by itself in npm 5 without running npm link created a symlink to the relative path. It did not copy the files like npm 4 and earlier would do. That's when the light bulb clicked on for me. They've change the way they expect npm link to be used, they just haven't told anyone. They haven't even updated the documentation. For the record, I do think the behavior of A's node_modules folder getting deleted when the symlink is overridden by the new symlink created by npm install ../A is a bug.

After a few more experiments I realized that they now intend npm link to be used for temporary symlinks. If you need to debug a module used by your app real quick then you can go into your app and run npm link ../path/to/module. It will overwrite the installed module with the symlink, leaving you free to start debugging. However, if you run npm install after that then npm will read from package.json and install the original package from the registry again. npm install no longer detects if a module in node_modules is a symlink and skips over it like it used to in npm 4 or earlier. There are only two possible conclusions from here. Either this is a bug, or it's intended behavior.

Normally I'd totally say that this new behavior of npm link is a bug, but after discovering the changes to npm install ../file/path I now think it's intended behavior. I think they meant for npm link to become a quick way to create a temporary link to a module for debugging. If you want a permanent link to a module then you just run npm install ../path/to/module and the link reference is stored in package.json. I believe the thinking is you would only have these links stored in there during active development and once its time to release you'd change them to actual registry references when your associated modules are published to npm.

I tried to convey these findings in the comments on that issue but I felt like people weren't completely understanding me and that we were talking past each other quite a bit. Some people did bring up some good criticisms that I agreed with though so I thought I'd finish this post up with a list of pros and cons for this new behavior.

Pros

Storing the permanent symlink references in package.json gives us the ability to completely delete our node_modules folder and run a fresh npm install and have those symlinks restored. In earlier versions of npm your symlinks created by npm link wouldn't get blown away by npm install, but if you had to completely delete your node_modules for some reason then you'd have to redo them all after install.

Similarly, storing relative path symlink references in package.json provides the ability for others to clone your project and associated modules for an instant development setup. This would be great for actively developing a module with a plugin architecture as the in-development plugins would automatically be symlinked into the main module's dependencies so long as the relative paths were maintained.

Cons

There is a risk of accidentally committing these references in package.json and cause your build to fail. Previously, links created by npm link would just live in your node_modules forever as long as you didn't delete them, but node_modules is in most projects' .gitignore files which conveniently left those symlinks out of the repository so you don't taint other peoples clones with them. Now there is the risk of accidentally committing those references when you didn't intend to.

If you don't intend to commit these relative path symlink references but you want to maintain permanent local symlinks then you have to add a build step to remove them.

Personally I feel like the pros outweigh the cons with these changes. At first I was one of those complaining that the issue wasn't fixed, but thanks to Paul Sanchez pointing out this symlink creation on npm install I now think I like the new behavior better. Though I do think they could, you know, document such changes so that people are aware of them, but I digress.

The biggest complaint I've seen from the few who understood what I was saying is that they don't want to override their package.json references just to have symlinks that don't disappear when you run npm install. I don't disagree with that as it is kind of annoying, but there are definitely workarounds. You could write a build script to swap them out with the real references on commit or build.