143 RR Passenger Enterprise with Tinco Andringa and Hongli Lai

Ruby Under a Microscope by Pat Shaughnessy! We will be interviewing Pat on February 27, 2014. The episode will air on March 6th, 2014. No Starch was kind enough to provide this coupon code your listeners can use to get a discount for Ruby Under a Microscope. Use the coupon code ROGUE for 40% off! (Coupon expires April 1, 2014.)

Next Week

Passion

Transcript

JAMES: I don’t know. What do you do, Avdi? Do you just hang out on the beach all day? [Laughter] JAMES: I’ve seen your ice pictures recently, which is why I said that. AVDI: Yeah, I just hang out on Hoth and just have some sun. Hunt down the occasional wampa.[Hosting and bandwidth provided by the Blue Box Group. Check them out at BlueBox.net.] **[This podcast is sponsored by New Relic. To track and optimize your application performance, go to RubyRogues.com/NewRelic.]**[This episode is sponsored by Code Climate. Code Climate automated code reviews ensure that your projects stay on track. Fix and find quality and security issues in your Ruby code sooner. Try it free at RubyRogues.com/CodeClimate.]**[Does your application need to send emails? Did you know that 20% of all email doesn’t even get delivered to the inbox? SendGrid can help you get your message delivered every time. Go to RubyRogues.com/SendGrid, sign up for free and tell them thanks.] **CHUCK: Hey everybody and welcome to episode 143 of the Ruby Rogues Podcast. This week on our panel, we have Avdi Grimm. AVDI: Hello. CHUCK: James Edward Gray. JAMES: Hello everyone. CHUCK: I’m Charles Max Wood from DevChat.TV. And this week, we have two special guests. We have Hongli Lai. HONGLI: Yes, thank you. CHUCK: And Tinco Andringa. TINCO: [Chuckles] Hi. CHUCK: I tried very hard not to mess those names up. HONGLI: Sounds very good. Thank you for hosting us today. CHUCK: Yeah. So, do you guys want to introduce yourselves really quickly? HONGLI: So, I’m Hongli Lai. I’m one of the co-founders of Phusion. And at Phusion, we make Phusion Passenger, the application server for Ruby. Tinco is our first employee. TINCO: I’ve been at Phusion since the beginning, sort of, quite a short while after the beginning. And at the moment, I’m leading the Union Station product, which is a metrics or a web application metrics app that you can use to keep your Ruby on Rails applications in check. JAMES: Wow, cool. CHUCK: Cool. HONGLI: And I have a lot of experience with the open source world even before I started Phusion. And we met together at the university, so we’ve actually known each other for a long time now. CHUCK: Oh, cool. Now, you guys are based down in the Netherlands, right? TINCO: That is correct. JAMES: So Passenger. Tell us about Passenger. HONGLI: Passenger is, hmm, yeah. Where should I begin? I should start from the very beginning, explaining what Passenger is, like you are five? JAMES: Sure, yeah absolutely. In fact, I would prefer if all of our guest from now on assume that I am five, because it’s [inaudible]. [Laughter] CHUCK: I act like I’m five. HONGLI: Okay. So, suppose you have a Ruby app. You want to put it on the internet so that other people can use it. How do you do that? Well, there has to be some way for the browser to talk to your Ruby app. But your Ruby app or Rails app cannot do that by itself. And if you come from a PHP world, then it would be very natural to put your app on Apache and then Apache would take care of that connection between your browser and the app. It doesn’t work like that with Ruby because Apache does not support Ruby by default. Neither does Nginx. And that’s where Passenger comes from. Passenger fills in that role and ensures that a web server that is Apache or Nginx can understand what your Ruby app is saying and that it can establish a connection between the browser and your Ruby app so that it can actually do something. That is Passenger in a nutshell. And Passenger is unique compared to other Ruby app servers in that it’s more of an integrated whole. It’s more of a holistic solution. It takes care of more things, has less moving parts, and it just tries its best to be damn simple and really easy to use and to be as least a hassle as possible. TINCO: Yeah. And because Passenger, like other Ruby application servers, implements the Rack standard, it can host and Ruby on Rails application without modification. HONGLI: Yeah, or any, Sinatra or Padrino or whatever other Rack framework you use. CHUCK: Does it support any other, I guess I don’t really know if there are other web frameworks that run on Rack. But does it support any other systems? HONGLI: When it comes to Ruby, since five years ago everybody supports Rack. Before Rack was invented circa 2009 or something, you had Rails, you had Merb and then you had a bunch of other frameworks that all had their own interfaces. Then came Rack and everybody supported Rack after a year or so, so there is now nothing that does not support Rack in Ruby space. CHUCK: Okay. Now one other thing, I’m going to go off on a little bit of a non-Ruby tangent for a minute. When I install it, it asks me if I want to install it for Node and a couple of others. HONGLI: Yes. JAMES: Python, Meteor, yeah I actually wasn’t aware of this. HONGLI: Yes, that’s because this is new. Since half a year ago, we started supporting Node.js and Meteor as well. And actually, Python support has been there since 2008. It was introduced as a proof of concept. Passenger was designed from the beginning to be able to handle not just Ruby. And then we added Python for fun just to show that it’s possible. But we never really marketed that. And we never really documented that. And it’s just since recently that we started to really push Passenger as a polyglot web server. JAMES: That’s interesting. So yeah, how does that work internally? Is it still basically the same engine with a little bit different interface on what it’s talking to or what? HONGLI: I don’t understand what you mean, sorry. TINCO: Well it’s sort of, there’s a loader that’s written in Ruby for the Ruby side of things, but the internals are all in C++.So it’s basically a small Ruby or Python or Node.js app. It loads the app. [Our part] loads the app and then it sets up the protocol to talk to Passenger itself, which is in C++. HONGLI: Yeah. The Passenger core, say 90% of the code, it’s in C++. And that part is the same no matter what language you use. And the language-specific parts that establish the connection between the language agnostic core and the actual language, that’s just a very small part. JAMES: Wow, that’s cool. CHUCK: So when I’m talking to people about Passenger, I start hearing about things like workers and threads and all kinds of stuff. And I haven’t really taken the time to look at how Passenger runs my Rails app or Rack app internally. Can you talk a little bit about that, about workers and spawning and all that stuff? HONGLI: Yeah, sure. I think I should start with what a process is because I notice that a lot of times, people get confused by the distinction between process and thread. So, if you go through the basics, let’s say you have a command line and you’ll type ls or something, that starts a program. And that program, that instance of a program is called a process. A process by definition and by how it is implemented on UNIX does not share memory with any other processes. And if one process crashes, then it does not affect any other process as well. So, processes can crash independently from each other. And if you look at the context of Rails, then when people say worker, then what they actually mean is a Rails process, or a Ruby process. And a process is just that, an instance of your Ruby program responsible for handling requests. And traditionally, Rails apps were only able to handle one concurrent request per process. Things have changed now. I will talk about that a little bit later. But it’s still true for many kinds of deployment setups, one concurrent request per process. So, what people do is they spawn multiple processes, multiple instances of the app, each one handling one request concurrently. And then they put all of that behind a load balancer, say Nginx. And then Nginx selects a process which is free to handle things so that you can make use of multicore and can have more concurrency and that sort of stuff. So, when people say that Ruby has a global interpreter lock, that’s got nothing to do with processes. And it does not mean that Ruby cannot use multicore. It can. You just have to use multiple processes. And what I just said about spawning multiple processes, putting it behind a load balancer, that’s actually what Passenger already does for you internally. So, if you use something like Unicorn, then you have to do all this manually and Passenger just takes care of all of this automatically without you having to do anything. And a thread runs inside a process. A process can have multiple threads. And threads can share memory with each other as long as they are running inside the same process. So, if you have a thread from process A, then it cannot share memory with a thread from process B because processes don’t share memory. And a thread is capable of providing extra concurrency. Nowadays, pretty much all Ruby frameworks have excellent support for multithreading. So, you can actually start an instance of your application that can handle more than one request at a time by having multiple threads. And how those threads are used, it depends on the application server. For example, a Passenger, the open source version, is a strictly multi-process application server, so it will start multiple processes, each one which are single-threaded, while Unicorn has the same design. It is also strictly multi-process, while Puma for example is multithreaded. And in Passenger Enterprise you also get the multithreading capabilities. There are benefits as well as disadvantages to using threads. So, if you’re interested, I can talk more about that. But it all boils down to multiprocessing is easier to do. Less things can go wrong. But it also uses more resources. While multithreaded it can potentially, not always, but potentially give you more performance. But it’s also hard to do, can give you more bugs, et cetera. CHUCK: Yeah, we had quite a discussion about threads and processes with Jesse Storimer a while back, so I’ll just put a link to that in the show notes so that people can go and listen to that if they want a more in-depth discussion of all of that stuff. HONGLI: Sure. JAMES: You had a really good description of it though, in the various tradeoffs. So, like you said, it’s more expensive to spawn up more processes resource-wise. You may need more hardware quicker whereas threads are generally cheaper and stuff. So, you can get farther. How have recent advancements in Ruby, say Ruby 2.0 and Ruby 2.1 where garbage collection and such got significantly better, or the new copy-on-write friendly stuff, should make multi-processes maybe a little less expensive, right? HONGLI: Out of all of those things, the only thing that makes multi-process less expensive is the copy-on-write garbage collector. JAMES: Right. HONGLI: Well actually, we did something like that back in 2008. Maybe some people still remember that with…. JAMES: Ruby Enterprise Edition. HONGLI: Yeah, yeah, exactly. Our findings back then was that on average, you can save 33% memory if you spawn your application in a specific way. And that is by using the smart spawning method in Passenger, or the preload app setting in Unicorn. So, with Ruby 2.0 and 2.1 you can get those kinds of advantages too. But it’s still relatively heavyweight compared to threads. Threads are still a lot lighter. JAMES: Sure. So, let me explain that a little, for people that don’t know. I’m sorry, I forgot the Passenger setting, but the preloading in Unicorn, and what was the Passenger setting? HONGLI: Smart spawning. It’s on by default. JAMES: Right. So, what that does, you said that processes don’t share memory, but actually that’s not entirely accurate. HONGLI: Yeah, yeah. JAMES: Right. HONGLI: It’s not entirely accurate. JAMES: There is some sharing that goes on as an operating system optimization thing. HONGLI: Yes. JAMES: So when you spawn a process, by default a good, efficient OS like Unix will just put a new entry in the process table but it won’t bother to copy everything over. And what it will do instead is as you begin to change bits of memory in the program, then it will copy that memory over and change it in the new process. But in the meantime, it will just reference back to the original process. This is called the copy-on-write. So, when you change it, it copies it over. And we’ve talked about how Ruby’s garbage collector has gotten better recently and storing object flags and stuff in a separate place so that it wouldn’t invalidate everything that loads before. So, what Hongli is talking about is these web servers have a feature where they will preload your app, they’ll load up Rails and everything, and get the app running. And that loading of all that, a lot of that doesn’t change. Like you probably don’t change your model classes and stuff like that, just the instances of the models right? And so by preloading all of that and then forking from there, you have all this shared memory that doesn’t end up getting copied over as long as you don’t modify that memory. So, it makes it slightly cheaper and that those processes are referring to the same chuck of memory. So, that’s why advancements like Ruby’s copy-on-write-friendly garbage collector was so important. HONGLI: Yes, exactly. JAMES: So Ruby Enterprise Edition, that was your attempt to do all of this before Ruby was really ready for that. You built your own custom version of Ruby to make this kind of thing friendlier. HONGLI: Yes. It actually came out of a research project at the university, and also because back then we were extremely cash-strapped and we didn’t want to bother paying $15 extra per month for the extra memory. So, then we thought, “Hey, let’s optimize the Ruby interpreter.” CHUCK: [Laughs] That is so the programmer way, right? HONGLI: Yeah. CHUCK: Well, we don’t want to spend $15 per server, so we’re going to pay somebody salary to go in and change Ruby. HONGLI: Well, we were students back then, so the government paid our salary. JAMES: [Laughs] Nice. CHUCK: Oh, there you go. HONGLI: Yeah, but it was still not much. The $15 per month, it made the difference between whether you can eat well or not. [Laughter] CHUCK: Oh, there you go, whether you get the high quality or low quality ramen noodles. [Laughter] HONGLI: Exactly. JAMES: We talked about this before the show, but I’m on the Ruby committers and I actually remember the discussions when Ruby Enterprise Edition came out. There was a lot of discussion about whether or not those changes would be adopted into Ruby core itself. And obviously something like them eventually made it in, in later versions. But they didn’t take the changes straight up. It’s a complicated issue actually. A lot of people didn’t understand why they didn’t just apply the changes directly into Ruby. But Ruby is meant to be a general purpose programming language. And the changes made in Ruby Enterprise Edition definitely made it more efficient, but for a certain kind of process, like a process that forks for example, and then slightly less in the more general case. So, that’s what the debate back then was about and why it took Ruby so long to gain these features and stuff. So, it’s an interesting thing. [Inaudible] changes, but much later I would say. HONGLI: Yes. CHUCK: So one thing that I’m a little curious about, we’ve talked about processes and the advantages of going with processes, especially for MRI. But we keep hearing discussions about threading in JRuby and Rubinius, which have different locking mechanisms that aren’t as, they’re not the global interpreter lock. So, it is a little bit easier to do threading on those. Does Passenger take advantage of any of that? Or is it still spawning separate processes for each worker or whatever? HONGLI: Passenger, well it depends on which version of Passenger. The open source version of Passenger only supports one thread for process while the enterprise version supports multiple threads per process and you could have as many threads or as many processes as you want. You can totally configure that. So, if you run that on JRuby and Rubinius, then yes you can take full advantage. CHUCK: Yeah, I was talking to a potential client and he mentioned that I wasn’t living right because I wasn’t using Passenger Enterprise. And I looked at him and I… [Laughter] CHUCK: My Scooby sense lit up and I was like, “Enterprise?” [Laughter] JAMES: That’s interesting. So, Enterprise takes the combination of both. You can configure processes and threads so you can basically have the best of both… HONGLI: Yes, it’s hybrid. JAMES: Oh, cool. TINCO: Yeah, it’s actually a fun story that we, back in 2008 we didn’t take the Enterprise very seriously so we got Ruby Enterprise Edition as an ironical name for just something that saves a little bit of memory. JAMES: And now you have an Enterprise version. TINCO: [Chuckles] Yeah, now we have… HONGLI: Yeah, now we actually have an enterprise offering. TINCO: Yeah. JAMES: That’s hilarious. CHUCK: Do you want to talk about that for a minute? What are the advantages of the Enterprise Edition? HONGLI: So, we already had the multithreading thing I discussed earlier. There are also more advanced resource control mechanisms. Oftentimes, there might be problems with the application due to bugs or maybe problems in the OS settings or… JAMES: Whoa, whoa, whoa. You’re not talking about my applications [inaudible]. HONGLI: Oh no, yours are totally perfect. JAMES: [Laughs] CHUCK: He’s talking about mine, but I didn’t want to say anything. JAMES: Oh, alright, just to be clear, okay. HONGLI: We’re talking about ours. [Chuckles] JAMES: Sorry, go ahead. HONGLI: Okay. Sometimes, it happens that for example you have a memory leak somewhere or maybe you have an infinite loop somewhere because you made a typo. That sort of thing happens. Passenger has these safeguard mechanisms that can help keep your server stay alive and defend those things. You can put a timer on your request so that if it takes too long time, then your request is aborted by killing the process, and of course also automatically restart it. Or you can put a memory limit on things so that if something leaks then it doesn’t run out of hand. And then your process just gets [resetted] after that, those kinds of things. And you can have more fine-grained control over the number of processes you want so that you can put quotas on your server per app, in case you have multiple apps on your server. TINCO: There’s also the rolling restarts. HONGLI: Yes. Rolling restarts is, it allows you to have better continuity for your website. Like if you do multiple deployments a day or something, then by using rolling restarts, your app will not suffer any downtime or delays during a restart. CHUCK: Can I clarify really quickly? So when you say rolling restarts, are you talking about the process of basically waiting for the worker to finish the request that it’s working on and then restarting and then pulling the next request that it needs to handle? HONGLI: Yeah, something like that. JAMES: I think they were talking more about how Unicorn will spawn a new master process and then the old ones die off, which I think is what you were referring to Chuck. CHUCK: Yeah. HONGLI: Yeah. JAMES: And then a new request will come in on the new master, right? HONGLI: Yeah. When using rolling restarts, the processes will spawn in the background and only when that is done, it will be atomically replaced with an existing process. And that is very quick so that the user does not notice any of the restart time, which may take several seconds normally. CHUCK: Very nice. JAMES: That’s interesting. CHUCK: So, I’m looking at the list here. What do you mean by mass deployment? HONGLI: So mass deployment is, let’s say you have 15 or 50 websites [inaudible] directory. And you want to deploy them all. You want to put them all online. With mass deployment, you don’t have to write 50 virtual host entries in your web server configuration file. You just type passenger start and then passenger will automatically set up a web server for you that can server all those websites using the folder name as the virtual host name. JAMES: Nice. CHUCK: So, is it bypassing Apache doing that then? HONGLI: It is using an Nginx core internally. CHUCK: Okay. TINCO: Yeah. So, it’s using the Nginx API to dynamically add the virtual host entries. CHUCK: Do you have to have Nginx installed or is that included in Passenger Enterprise? HONGLI: It is included. CHUCK: Okay. Yeah, some of these others, too, I’m a little bit curious about. So, live IRB console? HONGLI: Yes. So, sometimes, we have an issue with a process and we don’t really know what’s going on inside the process. We want to inspect its internal state. And then with Passenger Enterprise you can attach an IRB console to that process and then you can run whatever Ruby code inside the context of that process to inspect the state that you want. CHUCK: Oh, wow! That’s handy. JAMES: Yeah, that’s nice. HONGLI: And also Ruby debug support of course. CHUCK: Of course. [Chuckles] HONGLI: Of course. JAMES: You said earlier that you think of Passenger as a more holistic solution than something like Unicorn. Can you talk about the differences? HONGLI: With Unicorn, you usually have to do, you have to take care of more things. There are more moving parts. Like with Unicorn, you have to set up multiple Unicorn processes and then you have to put those Unicorn processes behind an Nginx reverse proxy. So, those are two steps. And then the third step is to set up process monitoring to ensure Unicorn gets started when the system starts, gets restarted if it crashes, those kinds of things. And all of that is taken care of for you by Passenger. So, in Passenger, you just edit your virtual host entry inside your Nginx configuration file or Apache configuration file. You just say, “I have this virtual host and my app is here. Take care of all the other boring, mundane stuff for me,” and then it’s done. JAMES: That’s interesting. So, you’re saying that Passenger does its own internal process monitoring and load balancing and stuff like that? HONGLI: Yeah. All those kinds of stuff are internal. It takes care of that for you. CHUCK: Is there any way to make it do that for things like background processes of daemons that I’ve set up? JAMES: Because Passenger handles so many details like load balancing and managing the processes and stuff, does that make it more of a black box, more of a thing I can’t see inside of? Like with Unicorn, I can just list my processes and I can see there, “Oh here’s my master process. Here’s my worker processes.” If one of them has hung, I could just kill it right there, or whatever. How does Passenger make that kind of information available? Or does it? HONGLI: Passenger does make that kind of information available. It’s just that you don’t have to care about it if you don’t want to. If you want to, you can. There is the passenger status command line tool as well as the Passenger memory stats command line tool. And with those tools, you can see exactly what’s going on and which processes you have. You can even manually kill things if you want. Passenger will restart stuff. JAMES: Nice. CHUCK: So, one thing that I’m looking at here on the enterprise page is that it says error-resistant deploys. Now is that just the resource management? Or… HONGLI: No. Error-resistant deploy is a specific feature for the following scenario. Sometimes, you deploy a new version of your app, but that version has a bug in it that you didn’t really notice before. And as a result, your app will fail to start. And normally, if you deploy a bad version like that, it will result in total website downtime because then no new version of your app can be started. It fails already during spawning. And if you have the resistance option enabled and Passenger detects that, it will refuse to continue with the restart and it will actually set a flag on the application. Then it will remember the fact that it failed to start and will refuse to spawn any further processes and it will also refuse to shut down existing processes. In other words, it will try to keep the old version of your deploy running as long as possible so that the website can be available for as long as possible while you fix the problem. JAMES: Interesting. CHUCK: That’s really interesting. HONGLI: So, those kinds of features are really designed to maximize uptime. JAMES: Right. TINCO: Yeah. So, the common theme of Passenger Enterprise is that we want to provide a service to people who are building apps and do not want to focus on any sys admin stuff or as little as possible. CHUCK: Yeah, now that is one other thing that I’m a little curious about, and that is that with systems like PHP, you mentioned at the beginning of the show, the shared hosting and a lot of the other hosting options out there, there are just a ton of them. And with Rails, it doesn’t seem like there are quite as many because there’s some level of setup. Even with Passenger, you have to go and install it on the server because most of the time it doesn’t come with it. Do you think that Passenger or Passenger Enterprise may contribute to that becoming a little bit less of a hosting story and it just becomes a common thing to have Passenger installed as well as mod_php or whatever. HONGLI: Passenger is actually already installed on many hosting platforms, for example DreamHost has been using it since almost day one. We explicitly designed for DreamHost. And there are many other hosts out there. I cannot remember them by name, but I know there are many others. JAMES: In cases where it’s installed, you just have to activate that particular Apache or Nginx module, right? TINCO: Yeah, yeah. If Passenger is installed, you can just add the line passenger enabled on to your Nginx configuration file and it will work. JAMES: Cool, cool. TINCO: So it’s yeah, just a single line to activate it when it’s installed. But yeah, PHP has a huge advantage because it’s integrated in Apache and Apache is installed on so many machines by default. JAMES: Right. TINCO: It’s always going to be one extra step. And… HONGLI: Yeah, we do try to make it as easy as possible, for example by providing the Debian packages. So that you can just apt-get install the passenger module and then done. Everything else, taken care for you automatically. CHUCK: Eat your heart out, Red Hat. [Laughter] HONGLI: We are also working on rpms, but it will take a while. CHUCK: Nah, [inaudible]. I’m just kidding. JAMES: Do you have any sense of if Passenger is run more on Apache or Nginx? TINCO: We support both of them equally. HONGLI: It started out with only Apache and then later we added Nginx and now we have excellent support for both. TINCO: I think our customers are preferring Nginx lately. HONGLI: Yeah. It seems like more and more users and customers are switching to Nginx, definitely gaining traction. TINCO: Yeah. JAMES: Interesting. CHUCK: So, I want to talk a little bit about some of the features that have come into Passenger over the last several years. One feature that was really handy, and I think it came out in a response to RVM was that you can specify which Ruby interpreter you use per app. HONGLI: Yes. CHUCK: Which was something that I really missed when you first came out, because it was a global setting for your entire server. Was there a lot of work that you had to do to make that work or was it more or less just a few changes to the way that it spawns the app? HONGLI: Well, it was a lot of work but not because we had to support multiple Rubies. It’s like we started supporting multiple Rubies since Passenger 4 but Passenger 4 also included a lot of other internal changes. We started out with a lot of design decisions and assumptions and then after a few years, it turns out that a lot of those assumptions no longer hold. So, we had to change a lot of things. And [inaudible] was Passenger 4 has much better internals and much more features, much more language-agnostic and that’s also how multiple Ruby versions are implemented. TINCO: Yeah, basically we moved to an evented architecture and the whole system became so modular that adding features like that became easier. CHUCK: What features are you looking at adding now? HONGLI: For now, we are only focusing on stability. On longer term, the daemon management stuff is definitely something that we want to take a look at, as well as introducing rpms. TINCO: Yeah. We’re also busy trying to penetrate the Node.js market and the Meteor market. We have actually had some good response from the communities of Node.js and we’re working together with them to make it a cool experience for them. HONGLI: So, these are our plans for now. We don’t have anything super long term yet. CHUCK: You keep mentioning that you’re working on stability. Where are the stability issues with Passenger? Because I run it and I don’t have any problems with it, but generally I’m not running things that are all that complicated or get that much traffic. HONGLI: The stability issues are just whatever issues people report on our bug tracker. TINCO: Yeah. So, we have a lot of customers with completely different setups and different applications. And people come with the weirdest platforms so sometimes there are operating system bugs we have to work around. Sometimes, there are just weird applications that do stuff a bit different. And it’s mostly getting Passenger to work with every environment that our customers use. HONGLI: Yeah, Passenger is written in C++ and it’s pretty low-level software. So, a lot of things at that level can go wrong. And although the main use case is tested pretty well, a lot of people have strange I/O patterns or strange traffic [inaudible] that can trigger edge cases where things don’t work so well. And we just to make sure all those stuff works. JAMES: It’s really cool. I like the dedication you have to making it as robust as possible and just trying to make it so everybody has a solid experience on it. I think that really pays off when you’re using software. I know, when I’ve installed it and gone through, it’s just been so easy to get going with and put up. And even when I do have something wrong, which is pretty much always my fault, it pops up these great error pages and stuff telling me exactly what to do. CHUCK: Oh yeah, I love those error pages. HONGLI: Good to hear. CHUCK: I don’t love seeing them, but I love the information that’s on them when I get them. JAMES: Yeah. [Laughter] JAMES: It’s usually something like, “Hey dummy. You forgot to blah, blah, blah. Please go do this now.” [Chuckles] HONGLI: Yeah, we did spend a lot of time writing those error pages. Like we regularly search Twitter and Stack Overflow for people complaining about Passenger. They don’t always complain to use directly, but they complain anywhere else. And we actively search them, analyze the use cases where things go wrong and we put code inside Passenger to take care of that and to warn them. That’s just a lot easier than trying to answer the same question for the 600^th time. JAMES: Yeah. Yeah, it is. CHUCK: Well, one other thing that’s interesting about that, I think, is that since Passenger is open source, we’re pretty accustomed to our open source software being written by one or a handful of people who are out doing it in their spare time, not necessarily backed by a company that’s really dedicated to keeping it up and supporting it and making it better. And so going to Stack Overflow is a knee-jerk reaction I think in most of these cases anyway. But if people do want to report a bug or things, what’s the best way for them to get that information to you? TINCO: Yeah. So, we have the mailing list that we browse daily. We’re always on there so you can get a response from Hongli or from me rather quickly. And our paying customers, they have a dedicated email address that we respond to even more quickly when we get the time. Yeah. HONGLI: You can find it if you go to the Phusion Passenger website and then you click docs and support. Everything is there. TINCO: Yeah, we’re five guys now I think, or six maybe. HONGLI: Six. TINCO: Six. [Laughs] So, we’re growing a lot lately. And it’s really cool to be supporting a product in this way. Because people are sometimes, they are surprised that we respond so quickly. Like if you post a bug on our list, it will be, within a day, there will be a serious developer making a fix or whatever. And we’re experiencing that there’s a demand for this. There’s been this feeling in the Ruby community, or maybe at Hacker News community or whatever, that the Ruby community is shrinking or that it’s becoming less and less vocal. But we’re actually experiencing that the Ruby community is still as large as it was. But it’s also becoming more serious. There are bigger businesses. In our customer base, we see that. When we launched back in 2008, most users were single server or few servers except for the startups that hit it big, usually a lot of small people that used Ruby on Rails. And then the past few years, the customers just got bigger. And now we’re serving multiple enterprise customers. It’s become a really serious business, the Ruby on Rails world. HONGLI: Yes. It’s maturing a lot. JAMES: Yeah, that’s a good point. CHUCK: That’s good to hear. JAMES: I just want to say, I think it’s pretty cool. It’s neat, what you’ve built. It’s fun to use and play with, and thanks for doing that. It’s awesome. TINCO: No problem. Thanks for using. [Chuckles] HONGLI: No problem, yes. Pleasure is all ours. CHUCK: Alright. Well, let’s go ahead and wrap this up and get to the picks. James, do you want to start us with the picks? JAMES: Sure. I’ve got three. Let’s see. First, I do this tactic all the time and I’ve never written about it or anything, and then I saw that Ernie Miller wrote it up on his blog. So, this is the ‘7 Lines Every Rakefile Should Have’ in it, for your gem. And it’s basically just how to launch an IRB console from your Rakefile. And I do that all the time and thought that was cool. So, good blog post there. Another blog post I enjoyed recently, we talked a little bit about Sinatra on this show today. This was a post about how to structure Sinatra applications, which I think people often bring up as one of the complaints about Sinatra. In Rails, you have the provided structure, and how are you supposed to structure in Sinatra? And this article has several ideas in it. But really, the main one is that you can build these little modules or apps and then just mount them under another app and that gives you your hierarchical structure much like Rails. So, it’s a cool read, if you spend any time building Sinatra applications. And then finally, I’ve been playing some new iOS games. Adam Keys told me about Lost Cities on the iOS and he and I have been playing a few rounds of that. That’s great fun. If you’ve never played the card game, it’s really good and simple but it’s kind of maddening with the choices and you end up watching the time and trying to squeeze out as many moves as you can in the time allowed. It’s pretty addictive. And this is one of those few games where I actually think it’s a little bit better on the iOS because in real life, if you want to know how many turns you have left you have to count the number of counts left on the deck. Whereas on the iOS it just tells you the number all the time. And it keeps a running score, whereas in the real game, you typically count it at the end. And so it’s easier to see how you’re doing as you play. So yeah, Lost Cities on the iOS. It’s pretty fun. Okay, those are my picks. CHUCK: Alright. Well, I’ve got some picks. The first one is a game called Relic Runners. And it’s a lot of fun. Basically, it’s kind of a mix between Lost Cities and Settlers of Catan. [Chuckles] It is the best way I could describe it. It’s a lot of fun. But basically, you run from, or you have your game piece move from spot to spot and you’re building roads or trails. And anyway, what you’re trying to do is you move to the temples and they give you special abilities and then you eventually, when the temple is completely excavated then you have a relic that you’re trying to get. And so you have a way of moving that you can use to get those. Anyway, it’s a lot of fun. So, just take my word for it. It’s a kind of complicated game to explain on a podcast like this. But we really enjoyed it. And so I’ll put that up there. One other pick I have is, and I’ve picked them before on the show but they just made my day, or my week last week. It’s hover.com. Now hover.com is my domain registrar. I finally got to move everything off of GoDaddy. JAMES: Woohoo! CHUCK: GoDaddy just makes me crazy. JAMES: It’s a great day. CHUCK: Anyway, what they have is they have a valet service that will move all your domains over for you. So, I just went and changed my GoDaddy password and gave it to them. They went in. They moved all my domains over. You as the listener probably didn’t even feel it move when RubyRogues.com moved over. And that’s because they set up all the name servers. They copied all of the records over and just made it happen. [Chuckles] And I couldn’t be happier. So, I’m off of GoDaddy. Everything’s on Hover. Their interface is way nicer to use. And I just can’t say enough good things about them. So, go check them out. Finally, I’ve been trying this Focus@Will. It’s FocusAtWill.com. And what it is, is they have a science page. Now, I’m not a neuroscientist so I don’t know if it’s junk or not. But it really does work for me in the sense that when I’m listening to the music, and they talk about the tempos and the type of music and stuff, it basically shuts off the voices in your head so that you can focus on what you’re doing. And it really does turn off all the back chatter in my head so that I don’t get distracted as easily. And then I just have that clear thought process around what I’m working on. So, you can go check that out at FocusAtWill.com. And those are my picks. Hongli, what are your picks? HONGLI: My picks are two-fold. One is the Rails Composer. So, Rails Composer is a template for Rails. You can pass it to rails new and then it will allow you to pick a lot of components that are often used. For example, Device or OmniAuth, CanCan, Bootstrap, RSpec, all those kinds of things. And it just makes it a lot easier than for you to edit your gem files for the 600^th time. So, this is great. And the other pick that I have is the UX Crash Course. And I think this is really important for pretty much everybody who ever works in the frontend. User experience is so important. It’s not just about design or about aesthetics. It’s about making sure that people experience your app in the right way. For example, in the Passenger installer, also a lot of user experience design has been put into it. And the kinds of principles that this website describes there, they’re not necessarily related to a visual user experience. So, this seems to be a good guide. TINCO: So, I picked the third choice and it’s a keynote that I watched. I did some research. I’m refactoring the Union Station app at the moment. And there were a lot of models that were used in multiple applications that should be in one gem. And I did some research and I came across an old Uncle Bob Martin talk. It’s the keynote at Ruby Midwest 2011. So, it’s pretty old. But it’s very fun watching. If you haven’t seen it, you should look at it. He goes a bit abstract and a bit far, but the ideas are inspiring and it’s nice. And it’s Uncle Bob Martin. So, it’s fun. CHUCK: Great. Well, thanks for coming, guys. We all really appreciate you coming and talking about Passenger with us. HONGLI: Thank you for inviting us. TINCO: Yeah, no problem. It was fun. HONGLI: Yeah. CHUCK: Now before we get too far gone, I also want to remind everybody that we are reading ‘Ruby Under a Microscope’ and we will be talking to Pat Shaughnessy on February 27^th. JAMES: Woohoo! CHUCK: So, that means that you’ll get the episode March 3^rd or something. JAMES: I was reading it literally before I came on this call. I read the chapter on hashes. CHUCK: Awesome. [Chuckles] CHUCK: So yeah. So, go check that out. And thanks for listening and we’ll catch you all next week.