084 JSJ Node with Mikeal Rogers

Download MP3



00:59 - Mikeal Rogers Introduction


Next Week

Huxley with Pete Hunt


JAMISON:  I actually have a cat sitting on my lap right now. Can you guys hear purring noises? MIKEAL:   [Chuckles] No. JAMISON:  Okay. JOE:  Is your cat purring? JAMISON:  Yeah. Yeah, she is. This is Grace Hopper. The finest technologist in the land. [Hosting and bandwidth provided by the Blue Box Group. Check them out at BlueBox.net.]  [This episode is sponsored by Component One, makers of Wijmo. If you need stunning UI elements or awesome graphs and charts, then go to Wijmo.com and check them out.]  JAMISON:  Hey everybody and welcome to JavaScript Jabber episode 84. I am Jamison Dance and filling in as a host for Chuck who is, I don’t know. He’s not here. We have Joe Eames. JOE:  Hey there. JAMISON:   And we have a special guest, Mikeal Rogers. MIKEAL:  Hello. JAMISON:  Mikeal, do you want to introduce yourself for the people that haven’t heard of you? MIKEAL:  Sure, sure. I’m Mikeal Rogers. I write a lot of JavaScript code and do a lot of stuff with Node and I run a bunch of conferences. And I’m the CTO of a little startup called Getable. JAMISON:  So one major question is what percentage of people pronouncing your name say it wrong? Because it is not spelled like you write. MIKEAL:  Intentionally or unintentionally? [Chuckles] JAMISON:  I don’t know. Either way. MIKEAL:  A good percentage of people that know me well mispronounce it on purpose to mess with me. JAMISON:  Okay. [Chuckles] MIKEAL:  [Chuckles] But most people, first time, they mispronounce it. And then they immediately ask, “Where is that from?” because it is a very interesting spelling. And the answer is just that my parents were hippies and so they just decided to make up a spelling. And it has not ethnic base at all. JAMISON:  [Chuckles] So I live in Utah. And there’s this trend sweeping the state to name your child like [inaudible] but just with the spelling tweaked. So you were way ahead of the game on that. MIKEAL:  Yeah. I’m a hipster since birth, I guess. [Chuckles] JAMISON:  Yeah. You were a hipster before hipsters even existed. MIKEAL:  [Chuckles] Yeah, yeah. JAMISON:  So I feel like you were too modest. I would classify you as the first champion of Node. You run NodeConf, which is the biggest Node.js conference and you are super influential in talking about what Node does and what it does well. And you’re always in the module discussions championing npm. And I don’t know. You’re one of the faces of Node. MIKEAL:  Yeah. And npm doesn’t need much of a champion. [Chuckles] It’s doing pretty well without me or anybody. [Chuckles] Yeah, it’s a force to be reckoned with. Btu yeah, I guess I got into it really early. I would say that the first champion of it was probably Jan Lehnardt, at least in my eyes. There were a bunch of people that got behind it early on and started developing it. But Jan really has a knack for promoting things and getting people excited about them. So Jan, running JSConf EU and this was in 2009, he was running JSConf EU and had Ryan give the first talk on Node.js. And that was where most people found out about Node.js. And so Jan was [inaudible] and talked to me about it, because he’s an old friend from Couch one. We were at the CouchDB company together. And I was like, “Well that looks really interesting. I should check that out.” This is actually before I was at the CouchDB company. This is when I was still at Mozilla. And so I started poking around with it. And I was like, “Oh, it’s interesting,” and then I put it down. And then a few days later Jan asked if anybody had written an HTTP proxy yet. And I looked and nobody had an HTTP proxy in Node yet. It was actually that early. And I had written this HTTP proxy in Python as part of this testing tool called windmill. And it was a very, very fancy proxy. It did so much. It would spoof, it would basically make all of the traffic look like it came from the same domain to get around the same domain security policy and did all this fancy stuff. And then to make the test run faster, I had implemented over about three years every crazy hack that you could do to make an HTTP proxy faster. So then I was like, “I know how to write a proxy. I’ll poke around with that with Node. I should be able to finish this in a weekend.” And it took me maybe three or four hours to write a full proxy. And then when I ran it and tested it, it was already faster than my Python proxy. JAMISON:  Wow. JOE:  Wow. MIKEAL:  And this was Node 0.20 or something, 0.10. This was really early. This was when promises were still in core. And I was like, “Well, I guess I need to make a decision about whether or not I’m going to write Python anymore.” [Chuckles] MIKEAL:  And I decided not to write Python anymore. Because all of the things that I was doing, Node was already better at, even back then. And so just going in that early and being really comfortable with HTTP and knowing the spec really well, I was able to be influential in the early days and get few nice little patches in the core, particularly around streaming and the HTTP client. I’ve always been associated with core more than I’ve ever actually contributed to it. My contributions to core are pretty minimal. But my biggest contribution is actually probably a one-line patch that I did that changed the way that streams worked fundamentally. So there was a patch that I put in. if you’re familiar with Node streams at all, they’re this object that can scope to a file handler like a socket or a part of the file system. And the cool thing about streams is that you can pipe them together. So if I know how to read a file, then I also know how to read an HTTP request or an HTTP client response, or an HTTP server input. It’s the same sort of interface to do all that, which is really nice for when you write a module and you want to transform some data. It just works everywhere. Sow what I put in was a patch that [basically] just emitted an event for when, a pipe event on the source stream. So now a stream knew about its input and its output, rather than just its output, which meant that we could do a lot of the really fancy stream stuff that you see in request and other libraries where we look at the input and do something different, basically. JAMISON:  Sure. You just casually dropped that. But you wrote request, which is the HTTP library that everyone uses pretty much in Node. MIKEAL:  Yeah, yeah. That was probably the first library that I put up that people were using. And that was before npm too, so I have no idea how many people were using it. [Chuckles] Or what they were doing with it or anything like that, because it predated npm. And it went through several iterations as Node changed and also as npm went up and changed. JAMISON:  Do you want to talk about NodeConf a little bit? Its format and what makes it unique and cool. MIKEAL:  Well, yeah. Well, its format is ever-changing. I’ve invented a new format every year so far. The first year was a little bit more standard. But the next year was really unique and new. And this year was very unique and new. So what I try to do with NodeConf is just figure out what I feel the community needs at any given time and then create a format that helps us express that and brings that all to the forefront. So in 2012, Node was growing really, really fast. But we hadn’t put out into even the community leadership a lot of the basic knowledge about streams. And also what was really happening at that time was that everybody’s idea of Node was very small. So if you thought about it as a frontend build tool, that’s all that you thought that it did. And if you thought that it was for express apps, that was all that you thought that it did. And so what I really wanted to do was expand what people thought of as being Node. So we had some of the first Node hardware talks there. And the format that I created for that was basically in a theater. And it was two days long and there were three acts in each day. And there was a narrative. So the first day was about the world of Node. And the second day was about something else that I forgot. [Chuckles] And I think it was the growth of Node. I can’t remember. But basically, there’s this narrative structure so that there’s a beginning, and a middle, and an end. And there were only 20-minute talks. And so each talk in an act would blend into the next. And I was able to get the speakers, especially because this time nobody had ever asked any of these speakers to do a 20-minute talk before. Everybody was doing 40-minute talks at the time. so they really felt like they didn’t have enough time, which forced them to work together and say, “Well if you’re handling this, then I don’t need to cover it and then I can just pick up where you left off.” And it worked out really, really well, especially in some of the sections like the streams section and stuff like that. So that was great and that went super, super well. And then this year, I felt like that format was really good at opening up what people thought of as being Node but didn’t really make them feel enabled to use Node. If you were an express developer and then you saw the hardware talk, you were like, “It’s really cool that people do that,” and not, “I can do that.” [Chuckles] So I really wanted everybody that left NodeConf this year to not just think of it as being bigger but really feel enabled to poke at all the different parts of Node even outside of their comfort zone. So rather than do talks, what I wanted to do was more hands-on stuff. So in a session, or in what used to be a talk, I wanted people to actually write code and accomplish something. So we went up to Walker Creek Ranch, which is a summer camp in Marin County. This is actually my fourth year running a conference there. I used to run Node unconferences there, and four years back a CouchDB unconference. But this year, we were way bigger than any of those conferences. We were over 300 people. And we took over the entire ranch and we basically had eight sessions. And each session was about an hour long and it had between 30 and 40 people in each one. And then each group would mix up and then you’d go to another session. Let me explain that better. When you got to NodeConf, you got a schedule and it said which session that you were in at every different time, every different time slot. You didn’t know what they were about. And so each session, it would address a concept and then you would actually sit down and write something and accomplish something. So in the hardware one, you would actually write some code that at least blinked an LED or maybe moved a Servo around. In the copter one, you actually were flying copters by the end. In Isaac’s session, he actually got you to write a patch to core that would never be accepted. [Chuckles] But you actually did patch core and compiled it and stuff. And all of the sessions were like that. And there were between two and four people in each session collaborating and getting that together. And all of those people way ahead of time, months and months and months ahead of time, started collaborating on the content and format for that. And it turned out really, really well. Everybody walked away really feeling much more enabled than they were before. And that venue was very good at bringing people together and breaking down all the social barriers. And people bring out there family and their kids and everything. It’s a really great kind of community atmosphere. JAMISON:  Yeah, I actually went to that NodeConf. And we were talking about this before on the show a little bit. So the hallway track is a classic part of a conference. And when you’re at this ranch where you’re hanging out at the lake or playing Frisbee or just chilling outside, it’s the best hallway track ever. [Chuckles] JAMISON:  It breaks down the artificial barriers, like you said, between the open source celebrities and just the average people that are there to have fun. MIKEAL:  Yeah. JAMISON:  So I think it was sweet. I talked to lots of people that I wouldn’t have talked to at a normal conference, I feel like. MIKEAL:  Yeah. And there are a lot of really standout things that happened. I remember the first time that I ever ran something there, it was CouchCamp. And I didn’t even think about having the four square game. But I ended up bringing chalk so that people could do art and we had a kickball. And people just started, were like, “Oh, we could do four square,” and four square blew up. [Chuckles] People love playing four square. Geeks love four square for some reason [Chuckles]. And so Substack turns out to be a four square drunk master. [Laughter] MIKEAL:  Like a beer on one hand and just dominating this game. [Laughter] MIKEAL:  It’s pretty great. You’d never think it. JAMISON:  That’s sweet. Yeah, it was a good time. And I feel like it changed the way that I look at conferences in the future. I don’t know. That sounds too breathless. And it was fun. MIKEAL:  I think that when conferences are good, what they are is an experience. So it’s not like an event or something. You don’t consume it. You experience it. And the good conference organizers always think about it from the point of view of an attendee that shows up and makes it from the first day to the last day. And everybody has a different take and a different spin that they put on that. But all of the good conference [inaudible] think about it that way. I think that there’s a bunch of conferences run by corporations that don’t think about it that way. And you’re just [Chuckles] you’re just the product or the consumer of all this content. But that’s certainly how me and all of the JSConf family people have always looked at it. JAMISON:  Sure. MIKEAL:  And certain people have taken that to totally new levels. This year, RealtimeConf was insane. Adam Brault and the whole &yet crew. I had a narrative structure with the 20-minute talks. He actually didn’t just create a narrative structure, he created a novel. Literally, there was a novel that came out that you had to read before the conference. There was also an audio book for it and a graphic novel for it. And then where that story, it’s this dystopian future novel, where that ends, the conference begins. And the conference takes on that story. And then there are actors. And I was a speaker and I had a name that was my character name and I had to give the talk as my character. It was amazing. [Chuckles] But yeah, definitely took the experience portion of conferences to a whole new level. JAMISON:  This is pretty timely, too, because Joe is actually part of a group organizing ng-conf, the Angular conference. And I know he’s also put a lot of thought into making it an experience. JOE:  Yeah, mostly it was, “How can we make this just like NodeConf?” [Laughter] JOE:  Seriously, that was one of the biggest fighting points or discussions points, was like, “Oh NodeConf was so awesome. And we played four square.” [Laughter] JOE:  I remember the four square story. “We played four square. Can we play four square at ng-conf?” [Chuckles] MIKEAL:  Yeah. So you should steal good ideas. You should be merciless about stealing good ideas. But you always have to think about why you’re adding them. I’ve been to a few conferences that have just taken these ideas and it was like a potpourri of conference stuff and not really like a clear reason for why this was happening or that was happening in a given time. And then you really just have to think about, “Yes, you want four square. But where do you put it and why? And what are people thinking when they get there?” And there are all these little social hacks that you have to do. So much is like, the trip from taking people from one place to the next, if you just change the way that you make that journey, people are going to have a different experience when they get to the end of it. Like RealtimeConf started, everybody was staying in this hotel and we did breakfast at this hotel then we had to walk four blocks away. And we’re about to go from breakfast and planes into this crazy narrative novel. And so what happens is everybody gets their flag for their country that they’re from. And then everybody stands behind a color guard and a marching band shows up. And then the marching band, this high school marching band, is playing and we literally marched there. And then before we got into the doors, there were actors playing protesters that were protesting the conference and some of the things that were happening. [Chuckles] MIKEAL:  And if we wouldn’t have walked, if we wouldn’t have had that experience when we walked there, if we just showed up and there’d be this protest, we’d be like, “What the hell? This is weird and kind of cheesy.” But it takes you over when you start it like that. JAMISON:  Sure. We don’t just want to make people feel bad they didn’t go to NodeConf. [Chuckles] JAMISON:  I think the value of this is that the JS community is amazing and that there’s awesome stuff happening that you could try to be a part of. I also want to talk about Node specifically, because we have a large audience of people that aren’t Node developers necessarily. They’re more frontend JS developers. Why does Node matter to these people? Maybe they don’t write backends or maybe they do but it’s in a different language. Why do they care about Node? MIKEAL:  That those people actually are using Node. [Chuckles] It’s the same way that people that were doing CSS and HTML and some jQuery didn’t call themselves JavaScript developers because the just knew how to plug jQuery stuff around. They didn’t identify as JavaScript developers. But most frontend developers now are using Node for something. They’re using Grunt for some kind of task. Or they’re using some kind of [dole] process for their CSS or whatever. And all of that stuff is being built in Node.  And you should think about it the same way as you did jQuery or JavaScript. Node is becoming something that you can just take for granted. So if you need to accomplish a task, you don’t have to be writing a backend. You could just be writing some kind of system process or you want to scrape some website or you just want to accomplish a task that isn’t in the browser, you should really reach for Node. Because the language and also the patterns are all there and they’re all really familiar. And in addition to that, there’s just this unbelievably large module community. So anything that you need to do is just a matter of finding the modules that you need and then stringing them together. JOE:  Yeah. So I’ve got this course with Pluralsight, Angular Fundamentals. It’s been seen by a lot of people. And one of the most common pieces of feedback is in our course, we use a Node backend. And people saying, “Oh, why do I have to know Node?” To give you a little bit of background, there’s probably a fairly heavy number of .NET developers who are watching this course. The dark matter of the corporate developer universe, right? MIKEAL:  [Chuckles] Yeah. JOE:  So I think there’s still a fair amount of people, like you said, there’s a lot of frontend developers that are probably using some Node but don’t realize it. But there’s probably a lot of web developers that actually haven’t learned Node, haven’t used it yet. These corporate Java, .NET type developers haven’t used it yet and still haven’t seen the vision of what is the value and power of it. And through my courses, I keep trying to convey that, “Node is awesome. There are all these things and you should be learning it. If you don’t know it right now, you’re falling behind the times. Even if you don’t use it. Even if your company says you cannot use Node, you’re falling behind if you’re not up to date with Node and doing it, at least know a little bit of basics.” Like you said, that idea that you have paradigms already that we’re familiar with because we know JavaScript. Just about everybody that does any web is going to know a little bit of JavaScript. MIKEAL:  Yes. JOE:  So what are your thoughts on that? MIKEAL:  Well I would say that Node is probably the most accessible platform of all of its contemporaries. And that’s not just because it’s in JavaScript, although that certainly helps. And JavaScript is a language that’s proved to be more accessible than any other language on the planet. More people turn into web developers that are not formally than any other language in the world, or any of the other platform and programming interface in the world. and Node, you have the benefit from that being in JavaScript, but also Node is always thinking about things from the point of view of, “How do we make this as simple as possible, as easy as possible, as accessible as possible?” So you don’t have to, when you go to learn Rails, you really need to go and learn all of these Rails patterns. And people like me obsess about Node patterns. And people like me and substack and these other people really obsess about these Node patterns. But when you start dipping your toe in and when you’re just starting out, everything is just obvious. You don’t need to learn how streams work for a little while. You don’t need to learn all of the little intricacies of the module pattern at first. You can really just start moving and go really, really fast actually, and build a lot of stuff, just piecing it along. And there are a lot of amazing learning resources now to even get you over the early hump of just dipping your toe in. So I feel like Node has to prove itself to all these people in all these different contexts and replacing a backend with Node. And we have a lot of literature and a lot of writing and a lot of benchmarks and things like that, that help people make that decision to either transition or to put a layer between their existing old backend and the new frontend they want to do. And there are a lot of people talking about that side of Node. I don’t [even think] people are talking enough about the, “Well I know .NET and I know HTML and a little bit of frontend JavaScript. What can I do with Node?” Well the answer is anything that you need to do. And it’s going to be easier and it’s going to be more [inaudible] and there’s going to be a much larger community of people also solving this problem and also working with you on it and a lot more existing work to draw from in solving the problems just that you would have as a frontend developer. JOE:  Right. MIKEAL:  That’s it. JOE:  Awesome. Very well said. MIKEAL:  Yeah. JAMISON:  Related to this, there was this article that the Groupon Engineering blog posted where they were running a Rails app to basically render their other frontend content and were just running into some problems with that at they split it up into a bunch of tiny Node apps that will render HTML and talk to several different APIs. And the use case for Node was that you’re doing lots of I/O and you’re making lots of requests at the same time to different backends and the async I/O stuff is just perfect for that. That’s the [inaudible] use for Node too. We run a bunch of little services that all talk to each other and it’s just so nice to be able to use async and just do async.parallel and make these five requests and join them all together and then come back to me when you’re done. And know that that’s not going to block and have to deal with threading to make that fast and stuff. MIKEAL:  Yeah, yeah. Node, the people in Node core especially, and people like me, we tend to obsess about performance. So we start talking about why is this certain use case with this buffer allocation not as fast as it should be? When you really start to break it down and put Node next to its contemporaries, it’s really fast. [Chuckles] It’s just unbearably fast compared to a lot of its competition. And even if you take out the async I/O stuff and you just start doing some processing, it’s actually a lot faster than a lot its contemporaries. JAMISON:  That’s amazing, because it’s reaping all the benefits of the browser wars, right? V8 gets faster, then Node gets faster for free. MIKEAL:  Yeah. People don’t talk about this enough, but the JVM is quite fast, especially if you can tune it and tweak it. And a lot of that comes from the fact that there were dueling JVMs for a long time. IBM had one and Sun had one. And there were lots of people trying to do better JVMs and competing with each other. And that tapered off or just died out. I don’t see a lot of innovation there anymore. There’s one JVM now and there’s new innovation happening there in terms of features and even some performance, but there’s not competition anymore. And the browsers are still competing for the fastest VM. There are more VMs now than there were five years ago. And all those performance gains are readily accessible. JOE:  But wait. I saw a YouTube video that says that it’s slow. MIKEAL:  Oh, that guy’s hilarious. [Laughter] JAMISON:  That guy [inaudible], man. MIKEAL:  You need to link to that. He is some kind of comic savant. He’s some sort of Andy Kaufman-esque in a natural [Laughter] MIKEAL:  I’ve never seen somebody so upset about something that other people did and then also so upset for a bunch of reasons that are just factually inaccurate. [Laughter] MIKEAL:  It’s amazing. You should put that in the show notes. JAMISON:  I never would have heard of him if not for that. He’s a [inaudible] MIKEAL:  [Laughter] Yeah, that man is hilarious.  I love his JavaScript is slow, that JavaScript is slow. Oh, god. [Laughter] MIKEAL:  It’s not like they’ve spent ten years getting super faster. JAMISON:  Another thing that’s been happening lately is just more talk about modules. I know there was some discussion about AMD versus CommonJS basically and that [inaudible] post [inaudible] in the show notes. We’ll do that. But there’s this basically a blog post as a gist by James Burke who’s the RequireJS guy talking about ... MIKEAL:  Yeah, I’m in there [queuing] to comment and stuff. JAMISON:  Yeah. Yeah, so AMD. He’s talking about why it’s the best for the web and then there’s some back and forth between some people talking about CommonJS. Do you want to summarize that and talk about you position on it? MIKEAL:  Yeah, I can. Well I think that AMD versus CommonJS is a false premise. That’s not really an argument or a conversation that matters. What you care about if you are a frontend developer is what tool do you use to modularize your own code and to get valuable assets from an ecosystem? How do you get other people’s code and use it well? And how do you get access to more of that? So what tooling do you use for that? And also, what tooling do you use for minification, CSS translation. There’s all this different frontend tooling. And there’s a lot different way of doing that kind of tooling. And a lot of people are competing in that space. Different tools have decided to go with different module formats. Most of the tools end up taking whatever module formats get popular enough to care about. So Browserify is a very popular modular written by substack that is in Node. And what Browserify does is that it basically emulates the Node module environment enough that you can take a lot of modules that are in npm that are written for Node that nobody even thought would run in the frontend that do and they work great. Or you can write a new module using the standard Node patterns in npm to publish and install it locally and it won’t even run in Node. It only works in the browser. And you can package all of that up and use those same patterns in the browser. So the code that you write looks more like Node’s module pattern, when you’re writing your application code and when you’re using this tooling. But the Browserify way of doing things is the Node way, which is we just write all these tiny little modules that do all this stuff. And so there’s a Browserify AMD compatibility thing. There is a Browserify component compatibility thing. So all the different ways to package code in different module formats to use, Browserify will consume and use. RequireJS was written to the AMD specification and has decided that it’s not going to support any other module format. And the reasons that it has for that is basically a list of features that they have implemented as part of the tooling that require that module format. In truth, you could do some fancy stuff to wrap up other module formats, but they don’t really want to hear that. That’s not the conversation they want to have. They want to have this conversation about why aren’t you using AMD. And I don’t really care if people use AMD or if they use the Node module format. I do care that people that are in the frontend community have access to the entire ecosystem of modules, which is primarily written to the Node pattern. You can’t have a conversation about JavaScript modules without tacitly admitting that Node has won this. By any measure that we have, the Node module format is 40 or 50 times larger and more accessible and there are more modules published than any of the dominant frontend module patterns. So that’s my take on that whole argument. [Chuckles] JAMISON:  So RequireJS can work with CommonJS modules. It’s hacky though, like you said. You have to do some code manipulation. Merrick should be here. But I think you have to change your code basically. MIKEAL:  Yeah. JAMISON:  Run it through a converter or stick some extra code in there for it. MIKEAL:  Yeah. And on the Browserify side, you can require an AMD module exactly like the regular Browserify Node patterns. In fact, they refer to the AMD module as deAMDify. [Chuckles] MIKEAL:  So basically it takes out the AMD stuff and normalizes it into a Node module. And you could do the reverse. It wouldn’t actually be that difficult. You could definitely wrap up a Node module. There are certain little things that may not work or maybe a little weird module.exports, but you can make it work. JAMISON:  Sure. Another part of this discussion goes into, you mentioned Bower and npm. I think that’s another part of the discussion. Where do you put modules? And some of the complaints against using npm for browser modules are static asset compilation or packaging, basically. So you can put anything you want in npm, like you put binary files in there, whatever. But the package.json format that npm expects doesn’t have any spec specifying like this is where my fonts live. This is how you consume them. MIKEAL:  Right, right. But at the same time, there are a lot of properties on package.json used by a lot of tools that aren’t specified by npm or in the package.json spec. So for instance, Browserify has several properties. If you want to, say overlay a module to be used in a browser that’s different from in Node, there is property in package.json that you use for that. That’s not specified anywhere in npm. It’s just a thing that Browserify made up. If you had any tooling that you wanted to do special stuff with those assets, you could just add it to package.json. I believe Grunt actually does some of this as well. I’m not sure off the top of my head what they are, though. But it’s really common to just extend package.json and add whatever properties that you care about. JAMISON:  It’s common to extend it. I feel like the problem comes in where you want to enshrine those as a standard. Node doesn’t want to put browser-specific stuff in its package.json format. Why do they have an incentive to support a fonts property when Node doesn’t need that? But if you really want to use npm as the browser package management, package repository I mean, you need to solve that problem in a generalized way. MIKEAL:  Well what we have to realize is that Node and npm are separate. And what Node needs and what npm needs can be different. And that’s understood. From day one, Isaac has said you should totally put frontend assets in npm. And npm is for JavaScript modules. Just put in whatever you want. Second, he said pretty bluntly that you can extend packages with whatever you want. We’re never going to, Node’s never going to decide that it needs a font property and [Chuckles] do different semantics with the font property that you decide to create. There’s an incentive for you to extend it and there’s all of the ability in the world for you to use that. And if enough people use it, it becomes the de facto standard. There are a couple of properties actually in package.json that started out as people just setting metadata and then we decided that that should be in the website. The way that the Node community likes to work, especially if there’s, so with package.json and with the npm repository, there is this single point of truth. So the data for this module is at this namespace. And then there is this JSON file for it. And what those properties mean, Node has some definitions for that it elevates on the website. For everything else, Node’s mantra and the npm mantra is the ecosystem should figure this out on its own. Allow thousands of flowers to bloom and then whichever ones make it and become really big and people get really into, those are the ones that we’ll say, “Okay, you’ve won,” and we’re going to start to elevate some of those properties and start to look at this metadata a little bit clearer. So I guarantee that if enough people were using the font property in package.json that something would go into the npm website that said, “Oh, this package has fonts. Here’s what the fonts are.” JAMISON:  Have there been any frontend-specific things that have made it into package.json yet or is it just not standardized enough yet? MIKEAL:  Package.json is just a place for any metadata about the module right? I wouldn’t even think about it as a standard. So there was a common JS standard for package.json that was primarily written by Isaac and a little bit me before we left CommonJS. And after we left CommonJS, it’s just like, “Okay, we’re not doing standard bodies shit anymore. We’re just going to build stuff. [Chuckles] We’re just going to build stuff and we’re going to see what works and we’re going to experiment a lot. And whatever works, works.” And so the package.json format, there are properties that npm uses and then there are properties that Browserify uses and any other tool in npm can decide to use or not. So if you want to create a new property with some good data on it, go for it and encourage other people to do the same. There are also new packages like Autumnify that are trying to take a modular piece of not just JavaScript but also some templates and some CSS and package that up into a module that can be reused. And that’s also creating a bunch of new package.json properties that it uses internally for some metadata. So yeah, the answer to this really is just start adding to package.json. And if your tool that consumes them can just read the package.json and do whatever it needs to. JAMISON:  Sure. JOE:  So for anybody that may not be familiar with all the topics that we’ve discussed, we should take a quick step back and do a couple of things and define, talk about the different popular module systems that are out there. And then maybe a little bit of the history of npm and the package.json and the purpose that it serves, some basics on that. MIKEAL:  Sure. How far do you want to go back? I know all the history. [Laughter] JOE:  Give us the, what is it, the elevator talk? MIKEAL:  So when Ryan first built Node, he took a module format from a group called CommonJS, which were primarily developers of Narwhal. So Narwhal was trying to be a Node thing but built on the JVM with Rhino JS. Node really took off and Ryan, earlier than everybody, divested from CommonJS. And me and Isaac were pretty involved still. JAMISON:  So CommonJS existed before Node, right? MIKEAL:  Yes. JAMISON:  It was a standard attempted to provide a module system for JavaScript? MIKEAL:  Okay, so CommonJS was a mailing list that thought that it was a standard body. [Laughter] MIKEAL:  And in many ways early on this was a very good thing. It could move a lot quicker than a standards body because it was just a mailing list of people and we just talk. And people were mostly in agreement for the early part of it. And then as more people were added, the disagreements piled up. And now the problem with thinking that you’re a standard body is that, “Oh my god. This stuff matters so much. We have to be really, really serious. And oh god, we have to think about every single use case ever imagined and enshrine it into the spec.” So that’s what CommonJS was. And what Ryan did was he took the modules standard. He took what was called modules 1.0 I think. Or maybe it was 1.1. And then I wrote an extension to it so that we could support CommonJS modules in CouchDB. So CouchDB also still to this day actually has support for standardized CommonJS modules. Then Isaac wrote npm. And package.json sort of was a thing in CommonJS and in the Narwalish community, but he had extended it so dramatically and needed so many things out of it that he just took it over. And at first those things were making their way into specs. And then eventually they didn’t. And even the first version of the npm registry, all of the REST interfaces and what they did, those were described in a spec that he wrote for CommonJS that nobody really took [inaudible] of that. And then slowly but surely, me and Isaac divested from CommonJS because while CommonJS was arguing about very tiny little things, Node was just blowing up. And we really wanted the freedom for Node to do the best module system possible for modules. We just wanted it to be the best way for you to write a JavaScript module, should be over here. And we were more comfortable taking the momentum in the ecosystem that was happening and all of the people writing all of this stuff to figure that out and experiment and see which one wins than we were with a bunch of people on a mailing list. And that worked. There are over 40,000 last time I checked, something, modules, in npm. There are about 100 a day that get added. It’s the fastest-growing ecosystem of any platform in the world now. So that ended up working out really, really well for us. And CommonJS stayed around. Then they went into this rabbit hole where they wanted asynchronous module loading. When they started that, we were still there. And we were thinking about that problem. And around that time, Substack did his first version of Browserify. And so Browserify took all these amazing modules in Node and made them accessible in the browser without changing the module format. And so we said, “Oh, okay. That problem’s solved. That will get better.” Yes, it doesn’t do certain fancy features that you can do with the standard of AMD, but it’s proving that there’s a path to get all of this stuff into the browser. So we’re not going to worry about that. We’re just going to optimize some [inaudible] system for modules, not worry about these browser edge cases. This is the philosophy of Node in general, is that everything should be a tiny module. Concerns should be separated. AMD as a standard, which is asynchronous module definition, it’s a module definition that RequireJS uses and it’s fairly popular for certain module formats. And Dojo moved all of its plugins and all of its modules to it overnight. So it got a big boost from that. That module standard has all of this asynchronous loading boiler plate in it. And the reason is because they want to be able to do this feature and they wanted that feature to be in the module format itself. Now you can do asynchronous module loading of Browserified modules if you use these certain patterns or if you use this certain tooling or if you decide to do this certain thing at this time, to import this asynchronous loader. You can do it. It’s just that most people don’t. And we didn’t elevate that problem to putting in that boiler plate for everybody that ever wrote a module. And they did and that’s fine. That’s their prerogative. But yeah, and that’s about where we are today. The module standard continued to evolve. Around that time, we also moved to all Node modules are installed locally rather than globally. So if you’ve used Python or Ruby or actually any platform, when you install a package, it gets installed for the runtime. When you run Python, you can get every Python module that you’ve ever installed. That is the opposite of how Node works. So when you install an npm package, it installs it into your local repository. So you have a project and you npm install your package for that project and then it installs it just locally. And so moving to that meant that there were certain complications with, okay let me back up a little bit. One of the things that this does is it allows us to get this holy grail of package management where you can have two different dependencies and they require conflicting versions of the same module. So one hasn’t been updated, hasn’t updated its jQuery dependency forever, and one is on jQuery 2.0, right? Well they just get their own version of jQueries since they need their own version of jQuery to work. And they don’t conflict with each other and they don’t override global stuff. And that’s a really, really important pattern when you want to build out a huge broad ecosystem of modules. Because if people can’t install these safely and rely on them to work, they won’t use modules and they won’t publish modules. The harder that you make it to do things right and the harder that you make it for things to not blow up, the less people are just going to write modules. And the npm philosophy has always been we need to make sure that the system encourages people to write more modules, [Chuckles] encourages people to publish more modules. It should always be easier to use a module than it is to write one, or to write a new one. And it should always be easier to publish a module than to not publish a module. So that’s the npm philosophy and where I feel like we’re at now. And npm stands for Node Package Manager. And it started out as just a package manager for Node, but now you can install it for, you can use it to install frontend tools. And those frontend tools can digest things in Node’s module format without even really being used in Node. I hope that explains. JAMISON:  That does explain. MIKEAL:  [Chuckles] Okay. JAMISON:  That’s [inaudible] info. [Laughter] JAMISON:  We are close to wrapping up. Do you have anything that you want to talk about that we haven’t talked about yet? Do you have any question that is just [inaudible] up for you that you wish we would have asked? MIKEAL:  [Laughter] No, no. I think that we talked about a lot of the good Node stuff. There are just so many things happening in the Node community that it’s hard to put your finger on one or two of them. Oh, oh, you brought up the Groupon thing. So there are two stories of running Node in production right now and I feel like people are starting to conflate them. So one is the Node in enterprise story and the other is new startups building new things in Node. And the way in which they build these things are actually really different. The Node in enterprise story is interesting. It’s basically that there’s a team of people at a big company and they want to move a lot faster and they want to write a mobile app and a desktop app a lot quicker than the current infrastructure allows. So what they do is they don’t touch the existing crazy backend that’s in .NET or Java or whatever it is. They write a little Node app and it talks to that backend and then it serves out their new frontend. So that’s what Wal-Mart does. That’s what I think AirBnB does that. LinkedIn does. It’s what all these companies are doing. And they’d have that problem. The new startup on Node is very different. So the new startup on Node is what I would call, it has an aversion to infrastructure. So you just set up Node processes and you try not to go with other things. You probably have a database because you need a database. But you really don’t want to run a lot of extra external services because everything that you run that isn’t Node is just another thing that’s harder to debug than it is to do Node. And it’s another thing that has another deployment process and it has another crazy log file format that you need to parse and worry about. It’s just another thing to worry about. Whereas if all your services are just Node processes, you just run them. And the database is the last thing that people are running outside of that. But most of these people are writing their load balancers in Node. They’re writing things that used to do operations, to put in Redis, just all in Node. And then we are finally writing databases in Node. The community is beautiful and amazing and totally not something you should put in production yet. [Laughter] MIKEAL:  But basically there’s a really beautiful modular community around building a database, which I really think is the future of database development. Because if you look at the way databases are developed now, they get to share very little code between each other. And they’re even able to use very little code from the ecosystem that they’re using, their programming language. So levelDB is actually used by a couple of these databases like Riak, or at least a fork of levelDB is. And so levelDB is like lowest level part of the database possible. It’s the thing that writes to the disk. And then you build every extraction that you want, whether it’s SQL or Riak or whatever. You can build on top of that primitive. So what we did in the Node community is that we modularized everything on top of that. Even the interface to that levelDB is what’s called level down. It’s like an abstract level down. And then on top of that is levelUP. And then this huge ecosystem that’s build on top of levelUP. So there’s a multi-level to get multiple versions of that. There’s ways to do versioning. There’s ways to do map reduce indexes. There are all these little plugins and all these little modules. And then even the level down layer is actually modularized. So you can see which performance characteristic you work the best with. so there are three primary forks of levelDB as well as there are some stuff that works in the browser and on top of index database and one that works on top of local storage. And so you can take this huge ecosystem of databases and then run it either in the browser or you can run it on these different file writers and see which one is faster for you. It’s really, really interesting. And I wrote a little database and I wanted to make it faster. I’m implementing most of what CouchDB does. So I wanted it to be faster than CouchDB for this particular benchmark. And I was like, “Oh, well I probably need a blooming filter.” If I was writing any other database, I would need to write a bloom filter. Instead, I had to pick which bloom filter module I wanted to use out of the five that are in npm. [Chuckles] And so that was five lines of code and all of a sudden it’s twice as fast for this particular use case. And it just makes writing all of this and makes the ecosystem way more diverse and a lot, lot simpler. So rather than having the giant database project, there’s just a bunch of little modules everywhere. So yeah, I guess that’s what I wanted to talk about. JAMISON:  Sweet. MIKEAL:  And JSFest. [Chuckles] JAMISON:  We can get to that. Do you want to make that one of your picks or do you want to talk about it now? Or we can just move on. MIKEAL:  Oh sure, I can just [inaudible] my picks. That’s fine. JAMISON:  Okay, cool. Well let’s move on to the picks. Joe, do you want to go first? JOE:  Yeah. I’ll do my picks. I think I’m only going to pick books today. The reason being I went on a trip to Ecuador and so I got to read a lot. Because while I was on a trip to Ecuador, I got stuck in Panama for two days in an airport, and also in Miami for a day in an airport. So I got to read a lot. And so I’m going to pick a couple of books. I already picked in a previous episode, I think I picked ‘Steelheart’, a new book by Brandon Sanderson. So I’m going to pick a classic by Brandon Sanderson, ‘Mistborn’ which I read in its entirety while I was, I’d read a little bit of it before but I read that [inaudible] whole thing. ‘Mistborn’ was I think his first real breakout novel. And it was great. It was an awesome novel. I enjoyed it every bit as much as I enjoyed ‘Steelheart’. And for my second pick, I’m going to pick Malcolm Gladwell’s ‘David and Goliath’ which is about an analysis of disadvantages and whether or not those are really truly disadvantages or not. So it’s great. and it starts out with this amazing story about this basketball team of 12-year-old girls that nearly took their entire league competition with a bunch of girls who didn’t know how to play basketball, most who had never played before, and they were the shortest and the worst-skilled players. But they nearly took it because their coach didn’t know basketball. It was a great book. I really enjoyed that. So those are going to be my two picks. JAMISON:  Sweet. I’ll go next. So I just have one pick. It’s also a book. It’s called ‘Willpower: Rediscovering the Greatest Human Strength’. It’s the kind of title that you would look at and toss into the garbage can as a cheesy self-help book. But it’s pretty good. I’m about halfway through it. So not done with it all the way. But it’s an interesting summary of some psychological research into willpower and what helps you build it up and what depletes it. So it talks a lot about decision fatigue. So if you have to make lots of decisions, it depletes your willpower towards the end so that you are more likely to just choose the default, choose the easiest path. But it talks about the role of nutrition in willpower and blood sugar levels and stuff like that. It’s been pretty fascinating. So that’s my only pick. Mikeal, do you want to talk about yours? MIKEAL:  Sure. I have a couple. So my first ones are learning resources. A new book came out by Henrik called ‘Human JavaScript’. It’s available online. It’s one of the better books on learning JavaScript. So for people that want to grab a book, I would do that one. The next one is a website called NodeSchool.io. NodeSchool is awesome. At NodeConf, everybody created different slides and materials to do their workshop. But substack actually wrote this whole framework for doing a choose your own adventure style learning exercise. So he wrote this thing called steam adventure, which is a choose your own adventure learning of streams. And people have taken that and written four more now. Or no sorry, three more than just that one. And they’re all available at NodeSchool. So one is done just learning Node. One is on functional programming. And the other one’s on the levelDB stuff that I talked about earlier. And now we’re actually doing some NodeSchool in-person events. So there’s on in London on the 13th and one at GitHub at San Francisco on the 21st of November 2013. These will actually be a basis for the workshops that we’re going to do at JSFest. Okay, so then I have some conference-y stuff for my other picks. So one is JSConf.com. At this point, the JSConf family has really gotten huge and blown up. So there are, I think, seven different JSConfs. It’s not one person or one corporation or anything running these. These are all run by individuals and we just help each other and have a little backchannel community that gets these all going. And then there’s a broader JSConf family of events. So you see things like CascadiaJS and NodeConf and even the upcoming JSFest. So if you go to JSConf.com you can see which ones are up and which ones have tickets available right now. And last is JSFest. So JSFest is my new conference. I built this conference with accessibility in mind. So the idea is that there’s this huge long tail of JavaScript developers that are new that really don’t feel all that comfortable going to a JSConf or a NodeConf because those are smaller, intimate events and it can just be a little bit overwhelming. And the content isn’t as newcomer-friendly as it could be. So JSFest is really like a way to bring everybody out and get everybody excited and energized. And the way that we’re doing it is a festival format. So we have all kinds of different events going on, all different dates. So there’s always something going on. Every event is individually ticketed. Some are just fun things. Some are more traditional conferences. Some are unconferences. Some are hackathons. So the stuff that we’ve announced so far is two main events that are narratives. So they’re very similar to the format that I talked about for NodeConf 2012 where there was a narrative structure. And there’s going to be an indie web camp. And there’s also DHTMLConf. Tickets just went on sale today. So DHTMLConf, just go to DHTMLConf.com to really understand it. You’ve really got to take in the visual presence to understand DHTMLConf. [Chuckles] JAMISON:  It’s beautiful. JOE:  Yeah, it’s totally awesome. [Chuckles] JAMISON:  I think he means totally rad. MIKEAL:  [Laughter] Yeah, yeah. The animated gifs are pretty great, too. I also love that I actually used Bootstrap. So that center tag with the triple MC Hammer and marquee, those are all responsive. [Laughter] JOE:  That’s so awesome. JAMISON:  It goes down to one MC Hammer on the phone? [Laughter] MIKEAL:  So those are what have been announced so far. But we’re also working on a ton of other cool events. So again, Node copter, we’re talking about doing a WebRTC camp. I want to do a TC 39 town hall so we can get members of the JavaScript standard community and then we just get normal developers there. And they can ask them questions. We want to do a robots event. We want to do a live NodeUp. We want to do some distillery and brewery tours just for fun. We’re talking about doing a silent disco. So this is crazy. This is like you get a DJ and rather than playing out of speakers, it just plays into an FM transmitter and then everybody has headphones on tuned to that FM station. And I think we’re going to build the FM things with Arduinos and some JavaScript. [Chuckles] And so if you walk by, there’s no music playing, but everybody’s dancing together [Laughter] MIKEAL:  And it looks like a big show. [Laughter] But it’s totally so JAMISON:  You’ll just see all these people breathing really heavy and like, “Oh that sounds amazing.” MIKEAL:  Yeah. I want to try to put together an art show. There are all kinds of cool stuff and we’re still adding things. We want to add security events and maybe some FirefoxOS stuff. There are a lot of different ideas that we have and we’re just seeing what we can do. But it’s going to be a really, really fun time in San Francisco. And you can go to JSFest.com to check that out. JAMISON:  Sweet. Well thanks so much for coming on the show Mikeal. It’s been great to talk to you. MIKEAL:  Yeah, thanks. JAMISON:  It’s been really good to get some insight into Node stuff in general and its influence on the web. MIKEAL:  Great. Yeah, thanks. JAMISON:  Alright. We will sign off. See you guys later.

Sign up for the Newsletter

Join our newsletter and get updates in your inbox. We won’t spam you and we respect your privacy.