JSJ 266 NPM 5.0 with Rebecca Turner

00:00 2453
Download MP3

On today’s episode of JavaScript Jabber, Charles Max Wood and panelist Joe Eames chat with Rebecca Turner, tech lead for NPM, a popular Javascript package manager with the worlds largest software registry. Learn about the newly released NPM 5 including a few of the updated features. Stay tuned!

[1:58] Was the release of node JS 8 tied to NPM5?
  • Features in NPM5 have been in planning for 2 years now.
  • Planned on getting it out earlier this year.
  • Node 8 was coming out and got pushed out a month.
  • Putting NPM5 into Node 8 became doable.
  • Pushed really hard to get NPM5 into Node 8 so that users would get NPM5 and updates to NPM5.
[2:58] Why would it matter? NPM doesn’t care right?
  • Right you can use NPM5 with any version of node.
  • Most people don’t update NPM, but upgrade Node.
  • So releasing them together allowed for when people updated Node they would get NPM 5.
[3:29] How does the upgrade process work if you’re using NVM or some node version manager?
  • Depends. Different approaches for each
  • NVM gets a fresh copy of Node with new globals. NVM5 and Node 8 are bundled.
  • For some, If you manually upgrade NVM you’ll always have to manually. It will keep the one you manually upgraded to.
[4:16] Why NPM 5?
  • It’s night and day faster.
  • 3 to 5 times speed up is not uncommon.
  • Most package managers are slow.
  • NPM 5 is still growing. Will get even faster.
[5:18] How did you make it faster?
  • The NPM’s cache is old. It’s very slow. Appalling slow.
  • Rewrote cache
  • Saw huge performance gains
[5:49] What is the function of the cache?
  • Cache makes it so you don’t have to reinstall modules from the internet.
  • It has registry information too.
  • It will now obey http headers for timing out cache.
[6:50] Other things that made it faster?
  • Had a log file for a long time. It was called shrinkwrap.
  • NPM 5 makes it default.
  • Renamed it to packagelog.json
  • Exactly like shrinkwrap package file seen before
  • In combo with cache, it makes it really fast.
  • Stores information about what the tree should look like and it’s general structure.
  • It doesn’t have to go back and learn versions of packages.
[7:50] Can you turn the default Packagelog.json off?
  • Yes. Just:
  • Set packagelog=false in the npmrc
[8:01] Why make it default? Why wasn’t it default before?
  • It Didn’t have it before. Shrinkwrap was added as a separate project enfolded in NPM and wasn’t core to the design of NPM.
  • Most people would now benefit from it. Not many scenarios where you wouldn’t want one.
  • Teams not using the same tools causes headaches and issues.
[9:38] Where does not having a lock show up as a problem?
  • It records the versions of the packages installed and where NPM put them so that when you clone a project down you will have exactly the same versions across machines.
  • Collaborators have the exact same version.
  • Protects from issues after people introduce changes and patch releases.
  • NPM being faster is just a bonus.
  • Store the sha512 of the package that was installed in the glock file so that we can verify it when you install. It’s Bit for bit what you had previously.
[11:12] Could you solve that by setting the package version as the same version as the .Json file?
  • No. That will lock down the versions of the modules that you install personally, not the dependancies, or transitive dependancies.
  • Package log allows you to look into the head of the installer. This is what the install looks like.
[12:16] Defaulting the log file speed things up? How?
  • It doesn’t have to figure out dependences or the tree which makes it faster.
  • Shrinkwrap command is still there, it renames it to shrinkwrap but shrinkwrap cannot be published.
  • For application level things or big libraries, using shrinkwrap to lock down versions is popular.
[13:42] You’ve Adopted specifications in a ROC process. When did you guys do that?
  • Did it in January
  • Have been using them internally for years. Inviting people into the process.
  • Specifications
  • Written in the form of “Here is the problem and here are the solutions.”
  • Spec folder in NPM docs, things being added to that as they specify how things work.
  • Spec tests have been great.
[14:59] The update adds new tools. Will there be new things in registry as well?
  • Yes.
  • Information about a package from registry, it returns document that has info about every version and package json data and full readme for every version.
  • It gets very large.
  • New API to request smaller version of that document.
  • Reduces bandwidth, lower download size, makes it substantially faster.
  • Used to be hashed with sha1, With this update it will be hashed with sha512 as well as sha1 for older clients.
[16:20] Will you be stopping support for older versions?
  • LTS version of NPM was a thing for a while. They stopped doing that.
  • Two models, people either use whatever version came with Node or they update to the latest.
  • The NPM team is really small. Hard to maintain old NPM branches.
  • Supports current versions and that’s pretty much it.
  • If there are big problems they will fix old versions. Patches , etc.
[17:36] Will there ever be problems with that?
  • Older versions should continue to work. Shouldn’t break any of that.
  • Can’t upgrade from 0.8.
  • It does break with different Node version
  • Does not support Node versions 0.10 or 0.12.
[18:47] How do you upgrade to NPM?
  • sudo npm install -gmpm
  • Yes, you may not need sudo. depend on what you’re on.
[19:07] How long has it been since version 4?
  • Last October is when it came out.
[19:24] Do you already have plans for version 6?
  • Yes!
  • More releases than before coming up.
  • Finally deprecating old features that are only used in a few packages out of the whole registry.
  • Running tests on getting rid of things.
[20:50] Self healing cache. What is it and why do we want it?
  • Users are sometimes showing up where installs are broken and tarbols are corrupted.
  • This happens sometimes with complicated containerization setups makes it more likely. It’s unclear where the problem actually is.
  • CaCache - content addressable cache. Take the hash of your package and use it to look up address to look it up in the cache.
  • Compares the Tarbol using an address to look it up in the cache.
  • Compares to see if it’s old. Trashes old and downloads updated one.
  • Came out with the cache. Free side effect of the new cache.
[23:14] New information output as part of the update?
  • NPM has always gave back you the tree from what you just installed.
  • Now, trees can be larger and displaying that much information is not useful.
  • User patch - gives you specifically what you asked for.
  • Information it shows will be something like: “I installed 50 items, updated 7, deleted 2.”
[24:23] Did you personally put that together?
  • Yes, threw it together and then got feedback from users and went with it.
  • Often unplanned features will get made and will be thrown out to get feedback.
  • Another new things ls output now shows you modules that were deduped. Shows logical tree and it’s relationships and what was deduped.
[25:27] You came up to node 4 syntax. Why not go to node 8?
  • To allow people with just node 4 be able to use NPM.
  • Many projects still run Node 4. Once a project has been deployed, people generally don’t touch it.
[26:20] Other new features? What about the File Specifier?
  • File specifier is new. File paths can be in package json, usually put inside pointing to something inside your package.
  • It will copy from there to your node modules.
  • Just a node module symlink.
  • Much faster. Verifiable that what’s in your node modules matches the source. If it’s pointing at the right place it’s correct. If not, then it’s not.
  • Earlier, sometimes it was hard to tell.
[27:38] Anything else as part of the NPM 5 release? Who do you think will be most affected by it?
  • For the most part, people notice three things:
  • 1st. no giant tree at the end
  • 2nd. Much faster
  • 3rd. Package lock.
[28:14] If it’s locked, how do you update it?
  • Run npm installer and then npm update
  • Used to be scary, but works well now.
  • Updates to latest semver, matches semver to package json to all node modules.
  • Updates package lock at the same time
  • Summary in Git shows what’s changed.
[28:59] Did Yarn come into play with your decisions with this release?
  • The plans have been in play for a long time for this update.
  • Yarn’s inclusion of similar features and the feedback was an indicator that some of the features were valuable.
[29:53] Other plans to incorporate features similar to yarn?
  • Features are already pretty close.
  • There are other alternative package managers out there.
  • PMPM interesting because when it installs it doesn’t copy all the files. It creates hard links.
[30:28] Does PMPM and Yarn use NPM registry?
  • Yes! Other than CNPM. The NPM client used in China.
  • CNPM Registry mirror behind firewall. Have their own client to their registry. Their registry is a copy of ours.
[31:15] What about RNPM?
  • I wouldn’t be surprised.
[31:45] “Won’t you come and say something controversial about your competitor?”
  • We all want it to be collaborative.
  • When we were writing our new cache, we also helped Yarn with their cache and sped things up tremendously.



Rush Limbaugh’s children’s booksTinker CrateKiwi CrateNPMEpisodes on My JS Story.


Gravity FallsBoard Games



Links to keep up with NPM and Rebecca

Twitter @rebeccaorgNPMjS on Twitterblog.npmjs.com


JSJ 266: NPM 5.0 with Rebecca Turner

CHARLES: Joe has his own recording booth that’s why you hear opening and shutting. Of course when we started podcasting, Joe’s recording booth is his car so… [Music][Have you ever felt like you’re falling behind or the programming world is moving so fast that it’s impossible to keep up? Then, there’s the issue of where to go to ensure you’re up to date. The answer is to join a community dedicated to discussing the latest in Javascript. Would it be nice if you’ve got Javascript Jabber all day? Well, you can, kind of. We’ve created a Slack community for Javascript Jabber. That means you can collect with our listeners and guests on a platform you’re most likely you’re already using. Plus, we set up a keeping current channel that pulls stories from across the web to help you know what people are talking about. And coming soon, we’ll be holding monthly webinars and round table video chats to connect with experts, the community and each other. So come and join us at javascriptjabber.com/slack.] CHARLES: Hey, everybody and welcome to another Javascript Jabber. This week on our panel, we have Joe Eames. JOE: Hey. CHARLES: I’m Charles Max Wood from Devchat.tv and we have a special guest this week and that’s Rebecca Turner. REBECCA: Hey. CHARLES: You have been on for a little while. Although, we did just do the My JS Story, do you want to give everyone a brief introduction? REBECCA: Sure. I’m the tech lead for the NPM team, which is me and Kat Marchan. And we’re going to produce the actual NPM command that you run on the command line. CHARLES: Awesome. I think it’s fair to assume that we’ve talked about NPM in the past. Most Javascript developers are aware of what it is, if they not use it every day. I think that’s the function of what their job is. So let’s just dive right into this. NPM released version 5. REBECCA: Yeah. We’re very excited about this. CHARLES: Now, one question that I have is we’re talking to the Node.js folks on Friday about Node.js 8. Were the two tied together at all because it seemed like one happened right around the other one? REBECCA: Well, you know, the features that are in NPM 5, we’ve been talking about going on two years now. We started concrete planning to get them out in the first couple of quarters this year last November. And the first year is like, well, we knew Node.js 8 is coming out and it would be great but we don’t think we can hit that deadline. And then, Node.js got pushed out a month. Suddenly, it became something we can actually do. We pushed really hard to actually have it to a point where we can get it into Node.js 8, so that Node.js 8 users will get NPM 5 and get the updates from the NPM 5. CHARLES: Why would it matter? I mean, isn’t NPM pretty agnostic to the version of Node.js you run? REBECCA: Actually, NPM doesn’t care. if you manually upgrade your NPM, you can use NPM 5 and 8th version of Node.js. But most users of Node don’t upgrade their NPM binary by hand, they upgrade their Node.js. CHARLES: Oh, right. I got you. So essentially, what you’re saying is, you wanted it out before Node.js 8 so that when people upgraded Node.js, they got an NPM 5. REBECCA: Right, exactly. JOE: Now, what about if you’re using NVM or some Node Version Manager on Windows? Then, how does the upgrade process work then? Do you have to manually upgrade? REBECCA: Oh, it depends on which one you’re using. They have different approaches to this. Like with NVM, you get a completely fresh copy of Node.js with new globals for every version of Node.js. So if you switch to Node.js 8, with NVM, then, you get NPM 5 with the version that’s bundled with… version of Node.js 8. With the older version with some of the other version managers, I’m not familiar but I know that some of them preserve their globals. So if you manually upgraded your NPM, then, you will always have to manually upgrade NPM because it will keep the one that you manually upgraded to. JOE: Awesome. CHARLES: Makes sense. So why NPM 5? I mean, isn’t NPM already awesome? REBECCA: Sure! But the biggest thing is that it is really super-fast – it’s so much faster than previous versions of NPM. It is night and day faster. If there is a lot, depending on what your project looks like, 3-5 times speed up are not uncommon. So you know, what was a minute install taking 20 seconds, it’s really a substantial improvement. CHARLES: That’s nice. I’m pretty used to package managers just taking forever any way. They just go slow. Sorry, it has to go on the internet and download crap that’s slow. REBECCA: Sure. If your internet is slow, it’s still going to be slow because we can’t fix that. We’ve done this as much as we can in other areas. And there’s actually… there’s still a lot of room for improvement. I expect NPM 5 can get even faster. CHARLES: So what do you do? I mean, what did you do to make it faster? REBECCA: Well, it turns out that NPM’s cache dates back to Node.js 0.2 or something. It’s really slow. None of us were aware of just how slow it was. It was so slow that if you are in a fast network, it was sometimes no faster. It was appallingly slow. And so the big thing that we started working last November was rewriting the cache. When we dropped that in, that’s when we saw this huge performance gains. CHARLES: Now, what is the function of the cache? REBECCA: Cache is just so you don’t have to re-download modules as you get them off as NPM installs your project. So if you install a project, and then, go install something else that uses the same modules like your old test frameworks, then it will install it from the cache, instead of having to download it off the internet. CHARLES: Okay. I was just curious because sometimes they also cache registries, information and things like that. REBECCA: We cache that, as well. It’s also the registry information. And that’s actually one of the big changes there. Previously, we have some bespoke rules for how long we’ve to wait to hand out your cache. And we now obey whatever the HTTP headers are. For registry metadata, if somebody upgrades this module and somebody upgrades the version, you won’t see the update for five minutes because that was the website is telling us to do. CHARLES: Gotcha. Were there any other major things that you did that made it faster? Or was it all just the cache? REBECCA: Oh no. There was one other major thing which was… we had a lock file for a long time. Although, we didn’t call it a lock file. We called it shrinkwrap. This was a point of confusion for people but what NPM 5 does is it makes that the default. And we renamed it so it’s called package-lock.json. And it is in fact, actually, exactly like shrinkwrap file that we had previously. But that, in conjunction, with the improved cache, makes the install process that much faster because the lock file essentially acts as a cache. It installs its ideas about what your tree should look like. So if your lock file… doesn’t have to think that again. Doesn’t have to go, “Alright, which versions of which packages already got that file?” CHARLES: By the way, making that default, thank you. REBECCA: Yes. Yes, it has been a huge improvement for everyone. CHARLES: If people want to turn that off, can they? REBECCA: Yes. They can set packagelock false in their npmrc. CHARLES: Okay. JOE: So why decide to make it default the old way, I mean, is everybody not served, as well, with the way that, I guess, lock files didn’t… I’m not sure what to say here… the way the lock files didn’t work? Is that bad for everybody? REBECCA: What you’re asking is, if it’s default now, why isn’t it default before? JOE: Yeah. REBECCA: Originally, we didn’t have any lock file when we were first working at it. And like a lot of things in NPM, incrementally, we got things added to it. So when the shrinkwrap was added, which is for all intents and purposes, a lock file, it was initially, actually, a separate project that was then, folded in to NPM. That means it wasn’t core. People who have been using NPM, didn’t expect to have it there. They’ve gotten used to… they didn’t use them. But I think, most people are better served by having them. I think that it’s a great decision to always have a package lock. There aren’t many scenarios where you don’t want one. CHARLES: Yeah, it eliminates a lot of issues with… “Oh well, I got a package that’s a different version from the one I expected,” or “I do an npm install in my machine. And then, my co-worked does an npm install in their machine.” Since it’s pulling a package version that I already have in my machine, I have a much older than theirs so stuff breaks on my machine. And then, they fix it. Then, it breaks on their machine. And, yeah… REBECCA: Yeah, it ensures that all of the person in a project are using the same tools. CHARLES: Yup. Well also on appointment, you know. It eliminates issues there too. And I’ve had all of those headaches in many languages. JOE: So if we didn’t have a lock, can we maybe talk about a scenario where this actually shows up? if you’re probably one of those people that are out there that have just been taking your basic Node and have it run, maybe they aren’t doing so much the way of devops, right? And they haven’t seen this problem. Can you kind of explain and talk about why, what that does and what it solves? REBECCA: So when you run npm install, it records the versions of all the packages that you installed in to your Node modules folder. And it records where NPM decided to put them. That way, when you get a new laptop and clone your project down, you’ll have exactly the same versions of everything that you’ve had on your previous machine. It also needs… to everyone that operating with. It has the exact version of everything, which relates to that whole thing where... Cinder is a statement of intent for humans and humans screw that up. You’re not supposed to introduce breaking changes and patch releases but that doesn’t mean people don’t. This protects you from that. I would say that’s the biggest piece. The fact that it makes npm able to be much faster with it is a bonus. But other thing that it allowed us to do is that we now store the SHA sum of the package. Actually, we store a SHA-512 that’s available of the package that was installed in the file so that we can verify it when you go to install. So that you know that it’s bit for bit identical to what you had previously. JOE: So can’t you just solve the problem by setting the package version to be in exact version of the package.json file? Is that not good enough? REBECCA: That does an excellent point. So no, you can’t because… lock down the versions of the modules that you personally install. It doesn’t lock down the versions of all of their dependencies. JOE: Interesting. REBECCA: Because you’re setting the version of that and everyone uses Cinder for the dependencies, it doesn’t lock down any of your, what we call, transitive dependencies – the dependencies of your dependencies. CHARLES: Right. I think I’m spoiled on this a little bit because I came up through Ruby and used bundler. And this is what bundler does. REBECCA: It’s also... one of the things that I like about it, apart from an npm developer point of view, is that the package lock allows you to look inside the head of the installer. Like, this is the installer stating its intent, or like, this is what the install should look like. I’m just going to save this here and I’m going to make your install look like that. CHARLES: Now, one thing that you’ve mentioned is that the shrinkwrap being default speed things up. Is that because it doesn’t have to do as much dependencies or figuring out the dependencies? Or there’s more to it than that? REBECCA: No, that is what primarily what it does. That’s why it’s faster. It’s both that it doesn’t have to figure out the dependencies and it doesn’t have to figure out where they go in your tree. Shrinkwrap describes exactly what your node modules will look like - or the package lock. I should say that we still need the npm shrinkwrap command, but actually it is now renamed your package lock - npm shrinkwrap. The difference between the two files is that package lock can’t be published. Npm shrinkwrap can. So if you’re publishing an application and you want to lock down your dependency versions. Happy is just a shrinkwrap. And they want to build to go, “Alright, this version of happy, we’ve tested it with this version of our dependencies. We don’t want to have support issues from someone who used a newer version of one of those dependencies so we’re going to lock everything down.” You really don’t want to do this on your small libraries because that would make the tree… it will make much harder… you can actually create much more difficult situations. You get the dif-tree again but for application-level things or like big libraries, frameworks, it is a popular approach to using shrinkwrap to lock down specifically what your users might be using. CHARLES: So one other thing that I saw just looking at the blog post here and a few other things that I kind of look at is that, you’ve adopted specifications in an RFC process. REBECCA: Yes. CHARLES: When did that happen? Is that old news and I just missed it? REBECCA: Yeah, we announced this last January. We did some blog post about it and as we’ve been coming across things, we’ve been posting up RFC’s. We’ve been using them internally for several years. And this is mostly just been inviting the community into our process. Specifications thing kind of came out of that. It wasn’t something we’ve talked about previously but we have an RFC and it’s like, “Alright, now, I want to implement the RFC.” The RFC is a request for comment so it’s written in the form of, “Well, here’s this problem we want to solve. Here’s some possible solutions. The specification needs to be, “Well, which one did we actually choose?” CHARLES: Right. REBECCA: So with our specs folder in the npm docs… and we’re slowly adding things to that as we specify in detail how they’re going to work. It also allowed us to write spec test, which has been fantastic. CHARLES: Oh, I’m sure. I have another question. It’s completely unrelated. Joe, do you have anything you want to this before I move on? JOE: Go ahead. CHARLES: So one other thing that I saw is that it adds some new tools that are in fact, it says here, “new tools intended for use with upcoming registry features.” So are we looking at new things coming in the registry, as well? REBECCA: Yes. That is true. Gosh, I’m not sure exactly what that’s referring to specifically. Amongst the new registry features has been… we introduced a… so a normal package, when you get information about a package from the registry, it returns this document that has every version ever of the package. The full package is on data for every package. And that package on data also includes the full README for every version. And so, as you can imagine, this is kind of unbounded help that they can get. For some packages, it is absurdly large. So we have a new API that you can request a smaller version of that document. That helps us by reducing our bandwidth build and also meaning, you don’t have to download so it should be substantially faster. CHARLES: That makes sense. REBECCA: So relatedly, the introduction of SHA-512’s, historically, all packages have been hashed with SHA-1 on the registry. With npm 5, we now have to input SHA-1 and SHA-512. We do the SHA-1 because all the older clients need to be able to still install your package. But newer clients will use the SHA-512. CHARLES: Makes sense. You mentioned the older packages. Are you planning to deprecate or stop support for, say, npm 1 or 2? Maybe you already have and I missed that too. REBECCA: We don’t actually maintain…we spent a period of time trying to do an LTS version of NPM. And ultimately, we gave up on that because nobody install npm as an LTS. People are in two camps. They either use whatever version came with Node or they upgrade to the latest. Those are the only two models that we came across when we’re doing that. And also, our team is really small. We can’t actually maintain a whole bunch of different branches. Officially, we support the current version of npm and we do not support the older versions. If there’s a security problem, we’ll go to releases on those older versions. Or something particularly like, we’ve done patches there to support Node. Node decides that they want to do an updated LTS release with a new Node chip, then we’ll go to an older release of npm’s for that. CHARLES: Okay. REBECCA: But otherwise, no, we just maintained npm 5 because it’s the only npm that we’ll be seeing releases. CHARLES: I guess what I’m asking though is if I have a really old installation of Node somewhere, it’s running application that don’t really want to bring up the latest version, and I’m kind of too lazy to update npm. Are there other reasons why I do not want to… am I going to see problems with that because you’re not looking at that? REBECCA: We’re not planning… we do not currently have any plans to deprecate any of the old registry API’s or anything like that. Older versions should continue to work. That also means that the third party npm plans will continue to work, as well. So yeah, no plans to break that. I think we can still upgrade from… well, you can’t upgrade from 0.8 anymore because we thoroughly broke backwards compatibility with that. Well, actually, npm 5 does introduce breaking changes as far as what Node versions it supports. So we finally hit the bullet and Node 5 does not support 0.10 or 0.12. We finally got to use npm 4 features. It’s so lovely. CHARLES: Nice. REBECCA: You know, we’re like three years late to the party. It’s okay. CHARLES: Right. So if you want to upgrade npm, you just sudo npm install –g npm? REBECCA: That’s right. It depends on your system on whether or not you need sudo. If you’re using a version manager, you probably don’t. CHARLES: That’s true. I’m just used to working with it either on a Mac or in some kind of Linux terminal where I’m not root. That makes sense. JOE: I’m kind of curious. How long were you on version 4? Or how long has it been since version 4 came out? REBECCA: Version 4 came out last October. JOE: So has it been terribly long. It kind of like 8 months? REBECCA: Yeah, about that and 7 days. JOE: Is that the typical cycle? Do you already have plans for version 6? REBECCA: We do! We’re really moving towards more releases than we’ve done previously. Some of this has to do with finally deprecating features that are used in 3 packages and all the registry. Thankfully, a question… we have the tools now to easily ask those questions, too. So we regularly, run tests of like, “Can we get rid of this? Does anyone use this at all?” We recently found out that one of the more esoteric features of npm is in fact used by a couple hundred packages on the registry. It gets the undocumented behavior and they rely on them. It’ll break if you change it. JOE: Interesting.[This episode is sponsored by hired.com. Every week on hired, they run an option where a thousand tech companies in San Francisco, New York and LA bid on Javascript developers providing them with salary and equity upfront. The average Javascript developer gets an average of 5 to 15 introductory offers and an average salary offer of a $130,000 a year. Users can either accept an offer, go right into interviewing of a company, or an item without a continuing obligations. It’s totally free for users and when you’re hired, they also give you a $2,000 bonus as a thank you for using them. But if you use the Ruby Rogues link, you’ll get a $4,000 bonus instead. Finally, if you’re not looking for a job but know someone who is, you can refer them to Hired and get $137,000 bonus if they accept the job. Go sign up at hired.com/javascriptjabber.] JOE: I have another question. One of the things mentioned is the self-healing cache. What kind of deal is that? I’ve used npm for a long time. I’ve never felt like I needed a self-healing cache. Can you explain what that is, why we want it and why you did it? REBECCA: Well, you wouldn’t really know if you did because it wasn’t before. As supposed it… wait, won’t usually show up as you would have broken install it previously because it will gone to get a tarball out of there, and the tarball would have been corrupted. I’ve never personally let this happen to my cache. With some regularity, they’re showing up if this is a problem. Usually because they’re using complicated software environments or other things, it makes that more likely. Maybe it has something to do with overlay file systems, and something writing to it under the real file system and overlay at the same time. I don’t know. What this actually means is that the new cache is based on a module called ca-cache. And that stands for content-addressable cache. By content-addressable, what we need is we take a cryptographic hash of you package and we use that as an address to look it up in our cache. And then, when we read that tarball out of our cache, we verify it against the content address that we’re given. And if it doesn’t match, then, we throw it away and download it again. If there’s any corruption in there, it just transparently tosses out the old cache entry and gets a new one. JOE: I think that makes sense. So was that a very common problem? REBECCA: Well… JOE: You said it never happened to you. REBECCA: Like I said, it’s hard to measure how common is this problem. It’s common enough that like I said, with some regularity issues of people who run it. And as I said, they are mostly people using complicated containerization setups so there seems to be something about that that causes this to happen. We don’t really know what. JOE: Gotcha. So how long was that on the plan to take care of that? REBECCA: That just came out on the cache rewrite. That came for free because we’ve been planning to do this content-addressable cache. When we did that, simply say, a free side effect of that. You have to make like cool little gifts of manually corrupting our cache… JOE: Alright, so the other feature that’s listed is new information output, right? REBECCA: Right. So this is something that we’ve had on the table for a long time. It was making the result that you’ve unpack from the npm install be more useful. The result that you got… npm always, historically, giving you the tree of what you just installed. Npm 2 and earlier gave you a separate tree for every module install. And npm 3 just gave you a single tree. Well, either way, it seemed like a great thing when you’re installing modules. These days, I installed just my base setup environment and I have 300 modules in my tree. That tree is… just blame that in the output is not useful. We’ve got rid of that and turned it into a summary. And actually, we’ve already accepted a user patch that now also summarizes, specifically, what you asked from the command line if you ask for anything there. So if you do npm install, name a thing and it will tell you, “Yeah, I installed that thing for you and this is the version we found for that.” And then, it also says, “I installed 50 items, updated 7, and deleted 2.” JOE: I see. Do you have a lot of big-handed to the exact design of that? Was that based on your own personal experience as to how it ended up getting designed? REBECCA: Yeah, I mean I just got do something together and put it out there. And then, we got feedback from users and we iterated on it. JOE: Okay, cool. REBECCA: And that’s one of those things that they have these experimental branches, where it will get a brainstorm and put something together. It won’t have any tests and it won’t have deeper product… Just push it out there. This release we got to actually… natural release process. One of the other related things that came out of that is probably new to you if you’re just upgrading npm 5 is ls output now shows you modules that were… It shows you the full logical tree of here’s everything and how it relates to each other. Some places here, we didn’t actually install module there. It’s just conceptually there. So that’s a little marker next to the item in the list. JOE: Right. That’s cool. CHARLES: So I have a question. You mentioned you were able to come up to Node 4. Why not just go all the way to Node 8? REBECCA: Because we want Node 4 users to still be able to update their npm. CHARLES: Oh, I guess that makes sense. Well, there are wider systems like AWS Lambda, for example, that still use Node 4 so that makes a lot of sense. REBECCA: Yeah. The percentage of users on Node 4 is still pretty substantial. I haven’t seen the latest numbers recently but I know that 0.10 and 0.12 together are under 5% now. That’s a very recent change. It’s because once something is deployed to operations, no one wants to touch it because it works. I wouldn’t want to upgrade it if it was mine. CHARLES: No, that’s true. REBECCA: Something else that we haven’t talked about is the file specifier. It’s not something that people use that much. But in some people, it’s pretty key to their header packages setup. You can put file pass in your package json. Usually, put this as pointing to something inside your package and it would copy from there into your Node modules. We’ve changed this and now, it makes a simile. CHARLES: That makes sense. REBECCA: Yeah. It’s much faster, of course, because we’re just making a simile but we can also verify that what is in our Node modules actually matches the source because it’s just a simile. If the simile gets pointed out in the right place, then, it’s correct. If it’s not, it’s not correct. When it was a bunch of files copied from a folder that maybe were mutated by the post install. It was really hard to tell. That came out of our sitting down, going well. If we make package locks, you know, standard, everybody has to have them. Then, we really have to like, “These cases have to make sense and have to not be in a case where it’s like, ‘well, I changed it and now, it’s not updating in my Node modules, what?’” That one, I hope to, I know, Yarn-landed bare version of the RFC for that feature. I don’t know if they’ve actually gotten in yet but they’re going to be following that. There’s semantics too as I understand that. JOE: Cool. Is there anything else that we haven’t talked about? Part of the npm5 release? What do you think would be most impacted and most affected by? REBECCA: I think the first thing that any users can actually… the first thing that most people notice is they don’t get that giant tree at the end. The second thing that they notice is that this is much faster. And then, the package lock is kind of a distant bird. Unless, you’re actually aware of these things and comfortable with them. Most users are, “Huh, well there’s this file here now.” But everything else seems better, so that’s great. CHARLES: So if it’s locked, how do you update it? Do you just update your package.json and say, I want something else? REBECCA: Ordinarily, you’d run npm install, npm update. For a long time, npm update has been kind of scary but it works pretty well, now. So you’ll run npm update and it will update to the latest Cinder. If you don’t get any arguments, sudo update to match the Cinder from your package.json all the modules in your modules. CHARLES: Okay. REBECCA: And then, it updates your package lock at the same time. And of course, you get an easy way to review that because your package lock is checked in when you go to add that to your kit. It has a summary of change. CHARLES: Joe, did you get to ask your Yarn question yet? JOE: Oh no, I totally forgot about that. I wanted to ask. So Yarn has been a thing. People have been talking about it a lot lately, right? And so, when you guys were deciding to do this release, you look a lot what Yarn was doing. Did that affect these decisions on what to put into this release? REBECCA: Like I said, our plans for this release go back about a year and a half. Two to four Yarns release. It’s not to say that Yarn didn’t have an impact. Things like having a lock file by default was something where we’re sitting there and talking about… we’ve been planning on making the npm shrinkwrap good enough that you’d produce it all the time. Once you have that, then, you can actually have a lock file by default. But Yarn was an excellent indicator that that was something valuable to people. Does that answer your question? JOE: Yeah, yeah. It does. Other than that, do you have any other plans incorporating features that Yarn has? REBECCA: Not aware of the substantial… I don’t think that there’s that big of a difference in our feature clarity. There are a lot of alternative package. We’ve also been working pretty close with the developer in pnpm. Pnpm is interesting because it makes it… when it does installs, it doesn’t copy all the files into your Node modules. It makes hyperlinks. It is, by and far, the fastest single package manager for npm. CHARLES: Now, the Yarn, pnpm, and all the rest of these use the npm registry? REBECCA: Yes, yeah. Everyone uses the npm registry. The only caveat to that is the npm, which is the npm client that’s primarily used in China. They actually run the registry here behind the firewall for obvious network reasons. You don’t want to do package request over the firewall. So they have their own client that talks privately to their registry. And then, that registry is our full copy of our registry. And they actually have something where you can publish to their registry or our registry. You don’t have to publish ours. It’s not 100% clear to me but, yeah, there are exceptions to that. CHARLES: Right. What about rnpm which is React Native? Is that completely different registry? REBECCA: I’m not familiar with it. CHARLES: Okay. I just never looked so I don’t know. REBECCA: I would be surprised. Most people are well… we could run our own registry and then, pay all the cost supporting that and have an operations team, or we could just use npm. CHARLES: Alright, well, are there any other things that you want to talk about, Joe? Or things that you think we should have brought up, Rebecca? JOE: Nothing on my side. REBECCA: I got also a few more words about Yarn. Amongst the end users, there’s sometimes say, people are like, “I found this exciting thing and now won’t you come and say something controversial about this competitor of yours?” But the fact is, all of us on all development teams will prefer to speak collaborative. And to that end, one of the purpose that we found in writing our cache, when we were looking at Yarn for the last month of development, we found that they haven’t implemented that. We actually PR-ed that. 20% speed improvement for Yarn, it was substantial. CHARLES: That’s one thing that I think is cool is maybe somebody wanted to work just a little differently or do something different. Or in this cases, their end… Yeah, that’s great that everybody, you know, you are contributing to them, their contributing to you. Yeah, that’s awesome. Alright, well, let’s go ahead and do some picks.[This episode is sponsored by Ruby Remote Conf. Ruby Remote Conf is a 2-day completely virtual conference hosted by, none other than, Charles Max Wood. If travel expenses are an issue where you just can't afford to be away from home for 2 days. Then, join us! It’s virtual. This conference is focused on people who are new to programming who want to learn what the pros know, or just get a leg up, and getting a job, and getting into the programming community. We will have speakers from all over the programming community to help you stay current in a Slack room where you can connect with speakers and other attendees in real-time. We’ll also have a live round table video chat for attendees and speakers. Plus, we’ll provide the top recordings to you within days from the conference.] CHARLES: (33:21) I’ll go ahead and throw out some picks. So my kids, they like to spend time with dad. It’s kind of fun. And so there are a couple of things that I really enjoyed doing with them. One of these picks, I hesitate a little bit to pick just because people have political stances, you know, they’re going to have feelings. But they’re terrific books and if it bothers you that I pick a book by Rush Limbaugh, then just close your ears for a minute. He’s got some children’s books, which are historical novels written for kids. It just tells the story. History teacher takes middle school kids back in time to the American Revolution. They’re really great. I’ve been reading those two. It’s Rush Revere and the Brave Pilgrims. I believe it’s the first one. And there are, I think, 5 total books. My kids are just love them and they know who all these historical figures are, they know what they are famous for. Anyway, they’re tremendous. The other thing, which if you have… more technical or maybe mechanical engineering bent that we’ve been getting from my kids, they get a Tinker Crate and Kiwi Crate. The Tinker Crates are for the older kids and the Kiwi Crates are for the younger kids. And the last ones that I got from my kids, we have a 3D Viewer that they built with mirrors and then, they have little slots to slide the pictures in. And I built that with the older kids, and the younger kids they have this really, really, fat straws, and you put this phone thing in one end and you put pins on the other end. Then, they have an air bladder with a hose on it. There’s an apparatus you put it into so you can change the angle of the rocket launcher and launch the rockets. They also had kites, like really small kites, and something else in it. But anyway, so it’s all about air pressure and aerodynamics, just moving things with air. The other one was obviously about optics and light, and things like that. So if you’re into some of that science stuff and you have kids that are… I think the Tinker Crates are for 4th to 6th grade and the Kiwi Crates are 1st to 3rd grade. I’ve been doing the Kiwi Crates with my 2nd grader and kindergartener, who will now be in 1st grade and 3rd grade now that it’s summer. Anyway, if you’re looking for some things to do with your kids, those are things that I highly recommend. Joe, do you have some picks? JOE: Hey, so they launched their show called Gravity Falls. It’s been awesome, really like it. It’s cartoon for adults. So I’m going to pick that. I also want to pick board games, in general. The local store, retail store, near my house is going out of business, the board games store is going out of business, which is kind of sad. But it’s just the way that things work. It’s been a shift in what’s profitable in retail locations, especially when selling things you can’t compete with them. It will be too bad. I miss being able to go to the store and see what’s new about board games. I hope that doesn’t mean that the types of media that are easy for people to consume like video games and TV shows crowd out board games. So I want to pick board games and it’s a great way to spend your time. And that’s my final pick. CHARLES: Alright, Rebecca. Do you have some picks for us? REBECCA: Sure. You know, I actually like to pick two like side projects of Kat and me. We both been kind of just living development for the past six months and so other hobbies on the side. But a... these are kind of exciting. So, one is Kat put together a tool called NPX. In NPX, you type in NPX and the name of some command-line tool somewhere on NPM and it installs it and run the command-line for you. And it does this all transparently and if you happen to have it installed locally, it just uses one. It started as a way to run like the stuff that's in your node module that's been... like your life cycle scripts can run but you just want to run it from the command-line you can NPX them. And she was like, I could just install the module so now it is... completely changes your relationship with these little command-line tools. So you don't need to install rimraf if you want to remove something. You may run NPX rimraf and it will install it and run it for you. So, I think that's really exciting. We may actually bring that into NPM and ship it as an additional binary at some point because it's so good. But it was just a… like that was like a thing that she hacked together in a few hours and was like, "Look at this thing I made!" And it totally changes your relationship with these things. CHARLES: That's cool. REBECCA: Yeah. So the other thing is a little library I put together which I called Funstream. And it's about making streams... it gives streams the iterator functions that arrays have. So you can call map, filter, forEach, reduce, and it also makes it so that errors are passed down the chain which is one of the biggest frustrations working with streams ordinarily is that if you do a series of pipes, it just throws away all the errors. And any error that you get just causes the whole program to exit because it's an unhandled error event and node exits on unhandled error events. And to capture them, you have to capture them at each level individually. I miss it why people use tools like Pump. Well, there's another approach to that. It's also faster than using through or writing your own through stream using a series of maps is faster than using through. So, that's an advantage too but I’m having a lot of fun with that. It's where I go to cool down from NPM things. CHARLES: Nice. Now, I'm going to call out just a couple more things here. On My JS Story, I have interviewed Isaac, Rebecca, and Larry Boss - all from NPM. So if you want to get kind of a picture of who these people are behind the scenes, Isaac also gave a rather interesting history of NPM. So, if you're interested in any of that, then go check My JS Story. And just go see those. Rebecca, if people want to keep up on what's going on with NPM or if they want to keep up on you and your projects and what you're doing, what are the places to go do that? REBECCA: Well, they should follow me on Twitter at @ReBeccaorg. If they want to follow NPM, we've got @npmjs on Twitter. We also have blog.npmjs.org. I think it was actually last January, we started posting all of our release notes to the blog every time we do our release. We'll be getting back into our normal release soon which means every two weeks, there will be a new NPM release. CHARLES: Very cool. Alright. Well, we will go ahead and wrap this one up. Thank you for coming and we'll catch every one next week. REBECCA: Kudos! Thanks for having me.[Bandwidth for this segment is provided by Cachefly, the world's fastest CDN. To view your content fast with Cachefly, visit cachefly.com to learn more.]

Sign up for the Newsletter

Join our newsletter and get updates in your inbox. We won’t spam you and we respect your privacy.