Charles: Hey everybody and welcome to episode 290 of the Ruby Rogues podcast. This week on our panel we have Brian Hogan.
Charles: Jason Swett.
Charles: Jerome Hardaway.
Jerome: Hey everybody.
Charles: I’m Charles Max Wood from Devchat.tv. This week we’re going to be talking about deployment. We’re going to have a geek fight, I think. It should be interesting anyway to see what the trade-offs are. I’m curious, just to go down the line and we’ll start with Jerome since I introduced everybody in the other order. What have you done with deployments? What have you used?
Jerome: For deployment, I’ve used Git, I’ve used Capistrano and I have used Heroku just for smaller projects, mostly [00:05:21]. But for professional projects, it’s usually Capistrano. Capistrano 2.0, Capistrano 3.0, those are the big ones. There was one project I had where I had to use AWS and I absolutely did not liked it. Those are the only things I used to deploy in Rails apps.
Charles: I hear you. How about you, Jason?
Jason: I’ve used pretty much exclusively Heroku over many years. Before that, I would just use BPS and do it manually. But in the last four or five years, I’ve used almost exclusively Heroku because I just liked to be able to not really think about it and just get it done and then never worry about it again.
Charles: Nice. How about you, Brian?
Brian: I still continue to use Capistrano 2.0 because of some of the features it has. the exception of places where I’m doing something with Heroku, that’s pretty much where I reach for all this code and grab a script that I’ve used for previous app and change a few variables. There, I don’t have to think about this type of command and its deployment. I can run the migrations and all the other things. It’s amazing how many of these things in the Ruby land have changed but something that’s like Capistrano still works today. I think that’s pretty fascinating myself.
Charles: Awesome. As far as my experience goes, I’ve done Heroku, I’ve set up things so I can just get push and it deploys. I’ve used Capistrano 2.0 and 3.0, most recently 3.0. I haven’t seen a need to stay on 2.0 but we can talk about that. I’ve done a little bit with Docker, not a ton, I’m not an expert but I have done a little bit with Docker as well for deployment. I’ve had AWS, but again, somebody else set that up, it was a royal pain in the rear, I’m with you on that one, Jerome. Somehow, they magic it up whenever I needed it magic up. Should we start with the simplest case and talk about Heroku then?
Charles: Heroku just seems to know what to do. You get push it and it does the build most of the time.
Jerome: Pretty much. You can definitely run in these nags. Just a quick note as far as that goes, I’ve worked on projects where they didn’t set up the production environment until six months into the project. That’s a really bad idea because if you do run in the snags and he likes six months in, then the problem might lie anywhere in that six months worth of work. It’s a really good idea to get your production environment setup on day one or at least as early as possible. If anything goes wrong, there’s a lot less code to debug.
Definitely, get that started as early as you can because you might run in the snags like what the asset pipeline and stuff. But chances are, anything you’re going to encounter has been encountered by somebody else before you and you can pretty easily find the answer [00:08:53].
Jason: That’s a really good point. One of the things that you got to remember when you’re using Heroku is you’re going to use the [00:09:00] database. That’s what they use and that’s what you should be using on your development boxes too because you’re going to end up with problems if you don’t. That’s based on personal experience. I won’t go into those details but I will say that, you really want to have as much of the production environment as you can in your development environment too and then continuously pushing to your production environment during that development process so you can make sure that those problems aren’t going to crap up.
Jerome: That whole concept of Dev/prod parity. Have your development environment and your production environment as similar as possible. There’s two ways we touch on there. One is the technologies and the configuration between the two. If you use PostgreSQL in production, then you should use it in development too or at least you can expect to have fewer database specific problem if you’re using the same RDBMS in production and development. That’s one is the technologies. But then also, just like the difference in code, if you’re deploying once a week, you’re going to have a lot fewer issues probably than if you only deploy once every three months.
Charles: Totally. The other thing is that, even on systems where it’s not Heroku, you also just have to keep in mind that there’s nothing worse than getting ready to release V1 and the whole company is watching and there’s issue with the deployment. The app works fine but for whatever reason, you just can’t get it to the server. That just sucks. It’s like, “Oh well, we didn’t account for this, that and the other when we actually built the app.” If you start out with your deployment setup on day one, then, you’re much better off because you’re going to hit those issues right away and you’re going to have a consistent setup and environment and set of concerns for the long haul.
Brian: It’s really exciting to jump into the code base on a green field app. It’s like, “Oh I want to get in here and write the code.” But, if you look at what a lot of professional teams do, they spend that they call iterations zero in some places where they set up their repository, they set up their continuous deployment environment, they make sure that everything is going so they can hit the ground running. They just all work it out of the way. Just getting all that stuff at one so that when you’re deploying that V1, you can do with [00:11:42] because you’ve already deployed it 10,000 times already. This last one is just for shell.
Jerome: Exactly. Another thought connecting to that, if you just dive in and start developing and then present it to the stakeholders, present your first little bit progress to the stakeholders, that might give them a false sense of the pace of progress that they can expect. Because if you just shown your development environment directly after a week with the development, you can actually deploy that to production without some work in between. They might set their expectations more optimistically than they really should based on that early, apparent really fast progress. But if you take that time in the beginning to set up all your infrastructure stuff, that’s going to be slower but it sets a little bit more realistic pace expectation right off the bat.
Charles: One of the things I’m just going to jump in with, I lied. There is something worst than getting your app ready and then not being able to deploy it. That is getting it deployed and then not being able to roll it back. If you have your deployment set up, tested and ready to go a lot of times, your setup so that you can just roll it back. Something’s broken in an embarrassing or insecure way, you can just pull it back out. No harm, no foul. Fix it up and then push it back again.
Jerome: Yeah, that’s another argument for the Dev/prod parity thing again. If you deploy a bad deployment and the last time you deployed was yesterday, the last time you did a good deployment was yesterday, then it’s probably not going to be a huge deal to roll back to them. But if you work for three months and then tried to deploy three months worth of work and it doesn’t go well, it’s probably going to be a lot harder to roll that back.
Charles: One other thing that I’m going to point out while we’re talking about this is, that I’ve been in a deployment hell where we basically had to go through the IT staff in order to get stuff deployed because we we’re dealing with government data and there were all this rules about who could access and how they could access and all that stuff. The deployment basically went through. If we have a new gem to install they had to convert the gem to an RPM which is a package for Red Hat Linux and then they would go and put it into a repository after they looked through all the code in the gem and then they would deploy it.
If you’ve got processes like that, that go on every time you deploy, you have to know about them upfront. This just straightens all that garbage out so that if you’re dealing with something like [00:14:34] or government data where there are regulations about the way that you can store it and deal with it and who can get access to the server, you want to get that out of the way first thing so that you just know. Okay, when we deploy it’s going to take two days because they have to go through and look at stuff.
Jerome: By the way, that reminds me of a real quick story that I want to tell. I worked at this start up once where we built this tool, we spent six months or at least the original deadline was six months. We started in January and the original deadline was July 1st. But it got push back to October something. But finally we’re finished sometime in October. They had this launch party because they hadn’t deployed it to production yet, they have this launch party where they got everybody together in the conference room and they showed this video of a shuttle launch on TV and they had noisemakers and they had, this was a terrible idea but they had fireworks that they lit off inside. It made the conference room all smoke and everybody had to leave. But they had this huge party for the production release. About a half an hour later, somebody is like, “Oh hey, we just got a call, it’s not working.” They had to roll it back and it was a total no go. I guess just like a case in point as far as deploy early and after you deploy early, deploy frequently after that.
Brian: If we want to talk about that, it’s probably a good time to ask, what do you do for continuous deployment? Are there certain tools that you all favor, that you all go to as sort of your tool set for getting that continuous environment going? Do you use continuous integration stacks that do a deployment? [00:16:44] the certain pushes to [00:16:45] or do you do it in a more manual approach? What do you all do?
Jason: I’ve used CircleCI in the pasT. I’ve set it so that when the test pass, it automatically deploys on some projects. But usually, to be honest, the projects I’m working on, like legacy projects that just don’t have the test coverage that would allow that kind of thing so usually that doesn’t come into the picture for me.
Brian: That’s more real world. We’re not all living in the fantasy land of the perfect web applications got the full test coverage that you can only [00:17:23] in the test that are all green. If you have those legacy apps, what kinds of things do you do with those in the best you can.
Charles: Yeah, I’ve taken kind of half measures on that. Another tool you can use, disclaimer, they sponsor the shows Snap by [00:17:42]. It was kind of the same thing. I’ve actually done it where if I push to a particular branch like the production branch and then all the test pass. In that way, I do have my sanity check on there but I’ve probably run it in staging your development and check it out manually first before I actually push those changes into the production branch if I don’t have automated test around it.
Though I’ve become more and more of a fan of just writing the automated test because if it takes me an hour to write automated test and it takes me a half hour to do the test manually, I only have to write the automated test ones to save myself three manual tests in order to start saving myself time and effort. I will start writing tests around the stuff that I am routinely checking. Sometimes you don’t have a full test suite around stuff then I just give it a quick sanity check. Maybe I’ll write a quick selenium test that runs through the main paths of the application just to make sure that they work for those pass on the production branch then I’ll have it deploy.
Jerome: I have a horrible legacy app with deployment. Believe me, the setup is just disgusting. The company didn’t want any testing, they didn’t want any of the new [00:19:10]. The current tools being used on how to make things better. Every time I did something, I had to do it old school. I’ll physically upload, go through [00:19:26] physically upload the files, clear the cache from the gem and then physically rebuild and restart the app every time. The app was so large, it would take 10 to 15 minutes for everything to go through. It was a complete nightmare.
I like CircleCI like [00:19:50] with Jason but there are a lot of nightmare scenarios so I think it’s best to know how to do it when it comes to deployment, the absolute horrible way that none of us are doing anymore. But, if you can always [00:20:11] these cooler, this better tools. [00:20:20] about the legacy apps you worked on this front line nightmares [00:20:24].
Jason: You’ve touched on something interesting there. This might be too much of attention but it sounds like that particular employer were officially anti testing, is that right?
Jerome: They didn’t care. They were an SEO and then ended up becoming like this legacy apps taking [00:20:47] partner. They did not care about any other tools or any of that nature. Just hurry up, get it out as fast as you can, keep these clients happy.. They didn’t care about doing it the right ways, [00:21:00].
Jason: I worked for a client once too. He was not just apathetic toward testing but he was officially anti testing. I was not allowed to write test. I just wanted to bring that question up real quick. What do you guys think, what should you do when your boss or when your client or whatever has a rule of don’t write test. Might been in this kind of just don’t work for those people but I wanted to know you guys think because that’s a pretty integral part. You don’t want to deploy an app that hasn’t been tested.
Brian: Can I answer by telling a little tiny story that my dad told me? He tune pianos, that was what he did for a business on the side while I was growing up. He would ask people and they said, “Hey, you tune pianos, would you tune my piano?” He would ask, “When was the last time you had your piano tuned?” If they said, “I had it done six months ago.” He’d be, “Oh, yeah, I’ll be over tomorrow to tune your piano.” But if you say, “We had it for 12 years. We’ve never had it tuned.” He would turn the job down because he knew that if he went in there and he tuned that piano, because the way pianos are made, this tuning would slip in a couple of days. They would think it was his fault, they would think he didn’t to a good job.
The lesson is, always stuck with me when it comes to software development. If I’m writing software and I don’t have a way to verify that the work I’m doing is good, when it breaks, they’re not going to blame Bob, the previous developer who wrote that terrible code, they’re going to blame Brian, the guy who actually changed that code. They’re going to blame me. That’s going to be a mark against my professional reputation. That’s going to reflect poorly upon me in that job and maybe in the future jobs. I try to very much avoid those kinds of situations. If I can’t, I will the write the damn test anyway.
Charles: I was going to say, I’ve been in that position and I just wrote the tests. I’ve had clients that said, “I don’t want to pay you for the time to write the tests.” Unfortunately, I’ve taken clients like that. I just write the test anyway and I just explain to them, this is the way that I know that I’m giving you value. Some of them don’t like that and they feel it’s a waste of time and others do. I’ve also had employers where basically we just have the conversation and it was like, “Look, you’re paying me to write this code and you’re paying me to write code that works and this is how I make sure it works.”
Brian: I have always had a good success by spinning it around and saying, “You’re not really paying me to write code, you’re paying me to solve your problem and the particular way that I solve your problem is with code that I write. It’s going to take me just as long to write this code with test or without test because if I don’t write test that I’m going to be staring at a web browser and filling in your forms repeatedly for you. You’ll be paying me for that time. You’re going to be paying me to solve your problem, let me solve the problem.”
Charles: I saw a quote in the [00:24:14] Newsletter that sent around, it was basically, people are paying you to solve their problem and the code is just a nasty byproduct.
Jason: That’s true. I read this other one some time that said something like you shouldn’t invest time in developing the skill of working with bad clients. It’s like your story Brian, with your dad turning those jobs down of tuning the pianos that had never been tuned. Just don’t work for those people.
Jerome: I like that. That’s a [00:24:57] motto.
Brian: It’s something that you do have to measure. You have to temper that a little bit because sometimes you don’t just have the choice. There are people who listen to this podcast, they don’t have the ability to just pick up and move and take a different job, and work with better people, they’re stuck in those situations so what kinds of advice can we give those kinds of people who are stuck deploying a Rails application through FTP, deploying an application that they don’t have confidence that they can roll back. What kind of advice can we offer to those people? It’s easy for some of us to say I’ll just get another job, I’ll just go work for different person. It’s important to understand that not everybody is in that situation. What could we do for those people?
Charles: I think ultimately, sometimes you have an unreasonable boss and there’s just nothing you can do. But again, just having the conversations as we’ve talked about some of these things where you explain the value of doing what you’re doing so you can mitigate some of the risks. Have a plan, even if it is through FTP and it’s not easy to roll back, just having a way to roll back or talking to them and just saying, “Look, sometimes some of this stuff happens so we need the automated tests just to make sure, just to give us a sanity check before we push something out.”
The other thing I’ve also been able to do is just say, “Look, let’s try it from up. If it turns out it’s a big fat waste to time then we’ll quit doing it. But if we see our bug rates go down or some other measurement of success go up, then we’d like to keep doing it.” A lot of time they’re willing to risk a month where you spend an extra 45 minutes a week or something like this that may or may not pay off.
Brian: There have been times where I’ve done those things. I’ll just spend the weekend off the clock of my own time setting up the stuff so I can deliver that a quick wing because I don’t want to be spending a week setting up a CI and lose a sometime other than month. I’ll just do that. I’ll say, “I’m going to set this of my own time and bring it in and now we’re going to use it and I can prove, I can demonstrate to you that this will save us time.” Sometimes you have to do that because it makes your life a little easier down the road just like you we’re saying a little bit ago about you do have to put that a little, even if you’re not going to get paid to write the test, you’re still going to write the test because it’s going to save you some time.
Brian: I have always thought that one of the problems that a lot of software developers run into, related this topic is, the feeling the need to be hero and react and respond so quickly to stakeholders creating an artificial picture of how long things take. Because if you’re delivering code really quickly without tests and then you start adding tests, externally, you’re going to look like you’re not as productive. I just wonder if that’s what the real problem is. We’ve done things sort of [00:28:18] and have haphazard for so long that the business owner’s expectation in a lot of cases are test slow you down because you’re doing that. We never really factor in. We never really account for the time.
We always talk about how we need our test suites to be faster and we can see those numbers. But we never really, I mean who tracks the time you spend in the web browser clicking through buttons and clicking through forms. They’re fine that your math was actually correct because you don’t have test that you’re verifying it, somebody is still verifying it. We never track that time. We never track it because it’s not as easy to see. I was wondering if that’s really the place to push back, is to show, hey, let’s account for all the time in the development process, not just the writing code and showing it to you on the screen, let’s account for all that time that we’re spending.
Jerome: That’s a really good point. I don’t know how you [00:29:11] because it sounds like one thing that you’re saying is like, if you write code without tests, then the short term, you can implementing things faster and demonstrate faster progress to the stakeholders. But then, if you start writing tests where previously maybe you weren’t, then the stakeholders might notice a slow down because it takes longer to write. Now, you’re not just writing with features but you’re also writing the tests and what do you do about that, is that what you’re saying?
Brian: Yeah, I mean, how do we start thinking about that because you are testing those features. If you’re going to show it to the stakeholder, hopefully you are doing some sort of verifications that when you show it to the stakeholders, it works. You’re probably pulling up in a browser, you’re probably doing that. Where is the mismatch come from when, oh, now we’re writing tests and now we’re slower.
You’re doing that testing either one way, you’re letting the computer do it but you’re spending a little bit of your time to write codes so that computer can do it for you or you’re doing it manually over and over again but you’re still probably doing. I hope that’s we’re doing as an industry [00:30:26] or actually taking the code we write and running it through and making sure that it works in some way.
Jason: Some people. I think there is like fours levels of testing. One is, you can have an automated test that verifies that it works. That’s the best. Or you can manually test it yourself and verify that it works that way. If there’s a bug you’ll find it at that level. Or if there’s a bug your stakeholder will discover it when your stakeholder is evaluating it or your users find it. That’s how you find out about the bug.
It becomes increasingly embarrassing and expensive as you go down that list. Maybe that’s one way to frame it to anybody who’s skeptical about the value or writing test. It’s like, “Hey, the testing inevitably happens. It’s just who discovers the bug and at what point do we discover it and how much it’s going to cost to find it and fix it.”
Brian: There are a lot of people who have some [00:31:38] about some project management software. One of the things that I have always loved is the project management software that sort of punishes your velocity for having bugs. I always like that. I always think that’s kind of a neat little like final pivotal tracker does that. You don’t get points for finishing off bugs. If you spend your whole week on bug fixes, your velocity [00:31:58]. I think that’s neat because it does show you that little bit of time upfront can save you time. Instead of fixing all these bugs, you could be working on new cool things.
Charles: I think that’s eventually where you get to is. Initially there’s a slowdown where you’re writing the tests and it’s not saving you a ton of time but as time goes on, you spend less time fixing bugs and more time actually solving other problems by building features. I’m going to push this back over toward deployment because we’re talking along the edge between testing and deployment but ultimately deployment is about getting the code under the server and making it run. I’m curious, do you guys typically do the server setup or do you have someone else to do it and then you just worry about getting the code out there?
Jerome: I do my own server setup. I’m afraid I don’t want to be involved in the situation where I don’t know what is going on based on the other person’s experience or what they do. Everybody, we preach these practices but people get their experience on different places and they don’t know what they don’t know. You don’t know where they’re background is coming from so end up running into this situation where the server is being set up by somebody who’s experience or their experience, as one way to say it, is vastly different from you or it’s like computer science [00:33:40] into the software.
They have their own unique way of doing things when it comes to a server setup versus self taught programmers who tend to think very differently than those who [00:33:54] the academic [00:33:55] when it comes to learning how to code and making their mistakes into that nature. That’s something that’s really, really important because when there’s a problem that you have to hurry up and fix, you don’t have time to go and ask a million questions and try to understand what that person did or try to find the person who did the set up. For me, server setup is very important to me and I handle that on my own.
Brian: I’ve been doing it with Scripts so that my environment is repeatable. As long as I can remember, I’ve doing it with Scripts even if it’s been, well, I have a physical box here and a [00:34:34] Script that it installs all the stuff in it. Just so it’s repeatable because I’ll forget important stuff, I’ll forget stuff. There are certain things that I’ve never been able to figure out how to automate so at least there’s a section in the Script that tells you, “Hey Brian, don’t forget to type this command to do something that needs to be done.” That’s how I do it.
I’ll learn new things all the time but what I can do is, I know the limitations of my knowledge related to the security in the servers. At least I can take my Scripts and run them by other people and they can say, “Hey, you forgot about this or you forgot about that.” That’s the thing to me. I don’t want to manually set up a server each time I have to an app. I want to process just like my code in my deployment to be repeatable.
I’ve heard of one super, super crazy example that a friend of mine says that they used to have their server guy every app deployment, he use to destroy the virtual machine, rebuild it from scratch because he came from a healthcare environment where they wanted to make sure there was no data left behind anywhere. They would always, every time the new version of the app, the deployment would literally [00:35:46] box, rebuild it with the code on it. That seems a little extreme but it’s honestly cool too when you think about it.
Jason: Welcome to the government and hospitals. That’s how they do it.
Charles: Well, that’s one of the advantages of Docker, is just that, your app environment is all encapsulated in your Docker container. It’s like, okay, we’re deploying the new version of the apps so you put all the new ones out there, you point your load balancer to all the new instances and then you just blow the old ones away which is kind of interesting. Then, basically your database instance is the only thing that’s consistently sticking around. That’s kind of interesting.
As far as server set ups go, for me, I’ve done everything from using Chef and Chef-solo down to writing scripts as Brian said. I’ve also just set up my own servers and deployed stuff to it depending on what I’ve got going on. I’ll say that the nice thing about Chef is that, the recipes for most of the stuff are kept up to date. They’ll do a lot of the server hardening and things for you. There’s actually a security recipe or two that I use and I’m trying to remember what they’re called. I’ll make sure I get links to those in the show notes. They restrict the access and stuff like that and make it really easy to get things set up.
I’ve run the whole gamut of things. Lately, it was just a toy project, a lot of times I’ll just spin up the server and go do a couple of things to install, phusion passenger on there and then we’re off to the races. But in other cases, if I think it’s going to stick around for a little longer, then I’ll actually go through the process of setting it up and hardening it up and things like that.
Brian: There’s an interesting topic. We talked about the process of deployment but what are you all using? What are you all using on your servers? Are you using Passenger? Are you using Unicorn? Are you using Nginx or are you using Apache? What are you already using?
Charles: I’m a Passenger, Nginx guy.
Jerome: Passenger and Nginx.
Charles: It’s easy.
Brian: Honestly, I’m doing Passenger and Apache rather than Nginx. Convince me if like the listeners for example if they’re on the same boat as me, convince me why Nginx and Passengers is such a great idea.
Charles: Ultimately for me and this isn’t a selling point in any way other than fact that I found that the config for Nginx is more friendly, in my opinion, than the one for Apache. This is for me, coming from the world where I actually deployed Linux servers with Apache on the regular basis when I was working in IT at the University I attended. I just find it cleaner and easier to use. I’ve heard all kinds of performance reasons to use Nginx but ultimately all the servers have just been fast enough for me. How about you, Jerome? Why Nginx?
Jerome: For me Nginx is cleaner, is easier. I believe, just like legacy apps are legacy practices [00:39:27] first thing I learned to picked up when I came to Rails. If it’s not broken, don’t fix it. I’m trying to figure out why go against something that’s already moving really well. I try to break that cycle and do something different.
Brian: Nobody here is doing any crazy load balancing with a bunch of Plumas or Unicorns or things like that. You’re sticking with Passenger and having good luck with that.
Charles: Yeah. My case, yes.
Jerome: Definitely yes here.
Jason: I just let Heroku do its thing. It works fine for pretty much everything that I do.
Charles: Yeah. The only trade off I’ve had with Heroku, I’ve never actually maxed out at Dino because I move stuff off to a server before I get to that point. But I have had them put the thread to sleep or kill it or whatever they do with it if it hasn’t been used for a while on some of my toy apps. That’s totally fine unless I have people who actually need to access it for business reasons and stuff like that. I know you can set up [00:40:42] and stuff so that it will actually ping it periodically, keep it awake. But if you’re going to go through all that effort, just set up a server.
Jerome: I think most apps that most people are doing are not like a really high traffic tape things. For me, they fall under two categories mostly. One is like in MVP for a start up or something like that. Usually, let’s be honest, you build it and then nothing ever happens. The production environment is basically just for the client to look at it and it gets very few, if any actual users on it ever. In the other case, it’s like an internal app for a small business. Just a small handful people are using it. Obviously, Heroku is going to be completely adequate for either of those cases.
Brian: Yeah. I got to say the majority of the applications that I’ve done in my career have been those internal applications with less than a hundred users. You always hear about, oh Rails can’t scale when the app [00:41:55]. That was the big talking point. It still tells me I’m not going to use Rails, it’s not fast enough. What are you building? Like literally what are you building? Because I have been doing this for a long time and I can tell you that most of the applications I’ve been encountered as a consultant, most of the applications I’ve been encountered as a fulltime employee. They’ve got 100 and maybe 10 simultaneous users because that’s what people need. They need their problem solved.
Jerome: Just a quick side note. Whenever I have to performance optimize something, the bottle neck is pretty much never the language or the framework. It’s usually like they’re making too many trips to the database where they have a really slow query that can be written differently. They pretty much never get to the point it’s the language or framework that’s a problem.
Brian: Yeah. There’s so many things going on in a web application that something must be really wrong if your choice of language is causing the performance issue.
Jerome: Yup. If you get to that point, if you have such huge traffic that you really have to dig really deep to get, then that’s probably a sign that you’re working for a really successful business that has the money to deal with that kind of stuff.
Brian: One more thing I’ve been itching to know about, how do you all handle for those sites that do get some traffic, do you throw things like [00:43:36], how do you handle those kinds of caching issues? Are you using other services to handle those kinds of things?
Charles: I’ve never gotten to the point where I needed to. Actually, how you usually just beef up the server itself get more memory or see if I can get faster or better more course. I’ll just use the built in caching with Rails in a lot of cases. That’s been good enough. I haven’t gotten to the point where I had so much going on that I actually needed something like [00:44:09].
Jason: I usually just use the built in Rails caching or denormalization of my database tables like for example if there is a reporting area of the application like the application I’m working on right now we’re doing this. I’m separating the reporting areas so we’re not querying the database in real time. I actually created a database view for that particular report and then I load that into a totally separate table that just exists for reporting purposes and then I query that table which is totally denormalized, a combination of the Rails cache and denormalization.
Brian: Rails took away one of my favorite feature which was the original page caching feature. I used to use that on a lot of [00:45:03] sites say to, for things that you didn’t have people log in for. You could whip together in a nice little CMS with that. With that being taken out [00:45:14] action caching on the other caching layers. I’ve had the resort of [00:45:21] things like that. [00:45:22] in that same boat.
Jason: No, I’ve actually never heard of [00:45:27] until now.
Jerome: I actually do the same way that Charles does it. You guys beef the server.
Charles: [00:45:37] have we tell about ten at night. Log in [00:45:41], make it better.
Jason: I was the only person that did the hell late night a server site changes. I’m just going to wait till everybody is asleep then I’m going to [00:45:55].
Jerome: If you’re lucky enough to be in an environment where you have an audience that’s just [00:46:06] for example, that could work if you got an audience for all. Now India is hitting the server then it becomes a little bit more difficult to find those times on.
Jason: One of the things that we’ve been that I absolutely hated but it worked to my advantage in here was, instead of having these huge Rails that works for different countries we will different Rails app per country so we could do things like that depending on their timezones. I was like, “I want to try something.” Because this company was anti [00:46:40] and testing and stuff like that. When I wanted to try something out and see if it worked, I would just wait until Australia is asleep and then I would mess on around and do it on an Australian site because when I was [00:46:55] I could make the change, mess up, figure out what I did wrong and put it back together. It’d be 7:00am in Australia and people are just getting up.
It worked wonders for me like that type of environment. But it’s still isn’t my favorite thing to do and you’re right, the way they’ve done they’re Rails app is one Rails app, it just [00:47:31] from different countries. It’s really hard for them to do things like just make the server space larger. They have to use, I can’t remember what tool they’re using. I think I’ll come back that you guys know because [00:47:44] add on to the show.
Charles: I’m going to just talk quickly about some of the ways that we’ve done deploying. We talked about Heroku and some of the things there. Have any of you set things up so that you can just get pushed to your server and then have everything do its thing once you do it get pushed?
Brian: I’ve done that before but I’ve had to mimic Capistrano when I did that so I’ll do get push and then I’ll have something move it over into another folder then I’ll have script sitting in that new folder in place before I couldn’t otherwise I wouldn’t be able to roll back. If there’s a better to do that, I’d love to hear it.
Charles: No, that’s basically what I’ve done as well. You have this Git Hook. It does all the things. It copies it over and then [00:48:44] it and then runs all the build scripts and stuff. Because I don’t check in the built Rails asset precompile or [00:48:57] assets precompile. I don’t do that and then push it, I always push it and then have it done on the server.
Brian: That whole process of pushing the Git repository that’s actually why I’m still using Capistrano 2.0 because a long time ago, the person who showed me how to do Rails deployment a long time ago suggested that I should use the copy strategy for Capistrano and that is in Capistrano 2.0. I could just set my repository to the local one so I don’t have to find laptop works, I could deploy it from the laptop, I don’t have to go out to an external GitHub repository where [00:49:36] do my deployments.
When the internet is down or when Github is down or something like that, I can still deploy things that I needed to deploy. That’s actually been one of those things where, oh that’s never going to happen but it’s actually happen to me a few times where I’ve needed to push deployment out and the place where our master repository was at wasn’t accessible. I’ve always been a little bit shy at moving to Capistrano 3.0 because I know that that’s not there anymore. I know that there are plugins to put in back. But like you said before, if it ain’t broke don’t fix it.
Charles: I’m curious, how big are teams you’re working on? Because that’s the downside I see with that particular strategy is that you have to pull and basically merge everyone else’s changes and then push it to the server instead of having it merged one way or the other upon the master server and then just have it fold it.
Brian: Well that’s the thing is, it’s the ability to do it when you can’t, when you need to, not that you would always do it that way. But it’s we need to do this so we have an outlet to do it. Normally, we’ll just go and get it and we’ll pull it down. Still, everybody can still push to GitHub, I can pull down the laptop or someone else can pull down the laptop and do the deployment. It isn’t that you need [00:50:53] everybody [00:50:54] patches around when things are down. It’s all on GitHub, I just pull master down to my laptop and do deploy. The deploy happens from my laptop. In that way, I’m also not putting keys on all the servers so the servers can go talk to GitHub.
Charles: Yeah, I’ve just set things up with deploy keys and I feel like the case where, oh GitHub is down or GitLab, if I’m using GitLab or something else is down, then I’ll just wait and deploy later. If you have mission critical that you have to deploy, then I could see where that would be a concern. I think it really boils down to what you’re situation is and how important that is to you because it’s just not to me. I’m sorry.
Brian: Yeah, sure.
Charles: But the Git strategy, you basically have a huge deploy script in you GitHook on your, I forget which hook it is, commit hook or something. But the flip side is, is using Capistrano which actually logs in via SSH and does the thing. In my case, I’m actually having it pull it from the GitHub using a deploy key and then it just runs all the built scripts and stuff like that. The thing that I usually like about Capistrano is that I get the feedback as it’s doing the deploy whereas with the Git commit hook, I don’t always get good information back if something goes wrong.
Brian: I don’t know how many of you all have to deploy the multiple server simultaneously but that’s a little harder with the Git strategy. It’s a little bit easier to do that with Capistrano to deploy, to push this app to the two servers.
Charles: You can also set roles in Capistrano so that you’re only pushing to the web servers and not to the database server or to the other piece of this app that runs in its own kind of thing or server or whatever. Capistrano is definitely my default way to go. Are there any other angles on deployment that we should dig into or dive into?
Jerome: I was going to ask the question that I get asked a lot is, when you have your application that’s deployed out there, how do you know when something’s wrong? Do you set up monitoring tools and what monitoring tools would you set up?
Charles: That’s a good question.
Jason: I use Code Climate. I absolutely love t. It’s one of the best tools out there that I’ve ever used. Actually, I run it before I even push [00:53:55] how to optimize the app and then afterwards it gives me feedback [00:54:01] what if there’s any problem or something is going on with the server. Their pro plan is pretty awesome. I absolutely love it. That’s what I use.
Charles: Does it do error reporting? If there’s an error on the server, does it report into Code Climate or is Code Climate just telekinesis tools?
Jason: It does. The pro plan has error reporting.
Jerome: I usually put some kind of error reporting tool on production too. Usually, in the past, I’ve used like Honey badger and [00:54:39], those are the main two I think.
Brian: I’m really interested in Honey badger. I’ve been hearing a lot of cool things about it and I just wanted to ask you how do like it the simplicity of it when it comes to use.
Jason: We have a couple of guys on the show like last week, you missed it. Although we didn’t talk about that.
Charles: Having used Honey badger myself, it’s really simple. It sends an email to the right people and it gives you pretty all the information that you would need to start tracking down the bug. I only run into one or two bugs so I didn’t have everything I needed to figure out what was going on and just quickly and easily fix the problem. It’s really, really simple interface.
Jason: Yeah. I’m a little bias because I’ve known the honey badger for quite a while but I really like it.
Brian: That’s the most important for me to have in my production environment is a good baseline. What does normal look like. I had for years used [00:55:56], I’m looking at using Zabbix instead because I want to be able to monitor the CPU usage over time and the memory usage over time. I want to know when something is going wrong, I want to know that that’s wrong and rather than just being reactive. I want to know what my baselines are and things like that in production environments.
The error tracking and things like that are great but I also want to know about those servers. Is this memory utilization? Is this abnormal or this just normal Rails doing it’s thing? I want to know those things. Those are the things that are really important to me when I’m monitoring my production environments.
Charles: The only other instrumentation that I’ve used that has been mention is [00:56:29]. I usually use [00:56:34], [00:56:35] does all of that monitoring that you’re talking about, Brian. You just install the server and it does its thing. But it also gives you information about how Rails itself is performing and typically what I’ve used that for is when I have a client or customer or employer that says, “This isn’t performing as I’d like it to.” In other words, it’s not necessarily that the server is maxing out or that Ruby or Rails is doing something in particular that’s bad. It’s just that, for whatever reason, one particular action on the server is taking a long time to come back or they just set the sites slow. That’s when I plug-in [00:57:17].
It actually breaks down, it’s spending a bunch of time on the database or spending a bunch of time in the controller or this particular method in the model, it’s spending a 150 milliseconds so then you could actually jump in and look and see, okay, what’s going on here? Oh, I need to do some Igor loading and looping over this in that way that it doesn’t perform and there’s an easier and more performant way to do this and start really looking at how the app is behaving overall.
Brian: If you collect that data, all the time, you can also compare, hey how did this perform before I push this new version of the app out, because you have to see what you got to change.
Charles: But I will say that [00:58:05] is a little bit pricey but it works. It does good job. I think they also have a front end monitoring tool that you can use. I’ve also used JS, anyway, there’s a front end monitoring system that does the same kind of thing. It gives you the error reports and things like that.
Brian: That’s TrackJS.
Charles: TrackJS, that’s what it is.
Brian: A friend of mine, Todd Gardner is one of the co founders of that.
Jason: I’m good with picks.
Charles: Alright let’s do it. Jason, do you want to start us with picks?
Jason: Sure thing. I don’t know why I always preface the fact that my picks never have anything to do with anything. But they once again, today don’t have anything to with anything at all that we’ve been talking about. My first pick, I know at least a couple of you guys have kids and you might have seen that show Dinosaur Train. Every once in awhile they have that guy Dr. Scott the paleontologist. He comes on the screen and says some stuff. Dr. Scott wrote some books and I bought one of those books. It’s actually really good. It’s called Dinosaur Odyssey. If your kids are in the Dinosaurs at all, they might ask you questions about Dinosaurs, that’s like, well, I don’t know. I wanted to educate myself a little bit so then I could teach that stuff to my kids who are both really into Dinosaurs. I like this book because there’s all those decay educational books that are just like a bunch of pictures and random little snippets. But this book is more like comprehensive and linear. If you want to give yourself a serious paleontological education, you can with this book. I really like that one. Again, it’s called Dinosaur Odyssey. One little fact that I learned from that, Mark Knopfler, the singer for Dire Straits, he has a Dinosaur named after him. I learned that from this book which is cool because I like Dire Straits.
The other pick of mine is also a book. It’s a called by Carl Sagan called Shadows of Forgotten Ancestors. That’s kind of about like the origin of humans and our relationship to other non human primates and stuff like that. Super interesting book, full of a lot stuff that I had no idea about before. Those are my two picks.
Charles: Awesome. Jerome, what are your picks?
Jerome: Roger that. My two picks are two Ruby books that I started reading this month. One is Rails Solutions by Justin Williams. It’s a really great book. Just opening up Rails to something that’s easier, not easier, best way to say it is open up Rails and make it more plain and easier to teach. That’s one great Book. Rails Solutions, Ruby on Rails by Justin Williams go check it out if you are looking for a good book to read.
Another book I’m going to pick up is, [01:02:15]. I’ve been really interested in learning how to use JRuby for some time so I decided to get a book and take the time to actually start learning its technology. Those are my two picks.
Charles: Awesome. Brian, what are your picks?
Brian: I got two picks, both books. One of them is book called Take My Money: Accepting Payments on the Web by Noel Rappin. It’s just a really great book talking about doing e commerce, taking money on the web. It’s got some great examples. I really want to encourage people, if it’s your first time out and you landed in e commerce project, this is a great book to read to get started. If you’ve been doing it for a while, this is a great book to read because you might get some different insights into how to go about taking those payments.
The other picks is, since we’re talking about deployment, the other pick is another pragprog book called Deploying with JRuby. Deploying with JRuby will show you how to set up your production environment, pushed Docker container to Heroku and take some other thing. It’s a fantastic book by Joe Kutner who works over at Heroku. He’s got some just fantastic advice on getting your deployment environment working on JRuby.
One of the more fascinating things about JRuby is I’ve been working with it to deploy some Rails applications on the Windows platform. There are certain organizations that they just don’t have Linux but they want to take advantage some of the power that Rails development can bring them. J Ruby tends to be a great solution for that. If you find yourself in a situation where you’re in a Windows environment but gosh you really like doing stuff with Ruby. JRuby is kind of a cool alternative that way.
Charles: Awesome. I actually interviewed Noel Rappin while we we’re rebuilding the panel for the show about his book. If you want a preview on what the book’s about, you can go check that out. We’ll put a link to that in the show notes, I think it’s episode 280 or 279. Joe Kutner has also been on the show. I think he was talking about health and being a healthy programmer.
I’m going to jump in with a few picks here. The first one is the Echo Dot. I bought my wife an Echo for mother’s day or something or for birthday, I don’t remember. But anyway, we use it all the time. When I found out that they had these Echo Dots for $30, I was like, “Oh, that’s cool. It’s cheaper than the regular Echo and it does everything that the Echo does.” I bought three more of them. I just put one in my office. I’m going to put the other two in the master bedroom, master bathroom. But it’s nice because you can just tell it to play music or do whatever. You can also hook it into home automation stuff through, if this and that and stuff like that. I’m really getting into it and figuring out how to make it do what I wanted to do.
The second pick I have and this is something that I’ve been doing over the last few weeks. My office got two record levels of disaster. I’ve been cleaning it up, just spending, I spend a couple hours yesterday, I’ll probably spend another hour or so today. I’ve made a pretty fair dent in it but it’s still a mess. But just the cleanliness it gives me a little bit of a better feeling and things like that, coming to work and getting to work. I’m definitely going to pick that as well.
Jason: Have you read the book The Life-Changing Magic of Tidying Up?
Charles: No, I haven’t. I’ve had a few people recommend it but I haven’t.
Jason: It’s a great one. I’ll just toss it there as a little extra pick.
Charles: Finally, I have one more book pick and that is Getting Things Done by David Allen. I think I’ve picked it on the show before but I saw an interview that he did with Jaime Masters from Eventual Millionaire. It was terrific. The way he talks about Getting Things Done because this is whole system for organizing your stuff that you got to do. It seems so complicated and then when he talked about it, it was like, “Oh, oh I get the principles behind this now and it’s not as complicated as I thought.” Yes, there are systems for a lot of this stuff and it takes some stick-to-itiveness in order to make it happen on a regular basis. But the flip side of that, is that, if you really understand the principles behind it, then the practices become a whole lot easier. I’m going to pick that book as well. I’ll have to check that book out, Jason. Thanks for the recommendation.
I guess we’ll go ahead and wrap up. Before we do, Jerome, I know that Vets Who Code is doing a kick-starter. Do you want to just mention that real quick?
Jerome: Roger that. Actually, we just decided to end it because it was going to be closing up within the next week. We didn’t reach out goal but we got some really great things that came out of it. For instance, one other main reasons of reaching looking for our goal is because there’s API of an app that we use called [01:07:52]. If you guys [01:07:55] teaching people how to code, it’s a really great app. It would cost us about $1,300 a month if we had more than 100[01:07:55] the API. They decided that they were going to give the API. Thankyou [01:08:12] that [01:08:14] for giving us access to your really expensive API. That’s one of the big plans this month. We’re hoping that getting a couple more things lined up [01:08:31] can. We’re trying to get to the numbers of helping about 3,000 to 4,000 veterans a year versus helping almost 100 veterans a year.
Charles: That would be really cool. We’ll go ahead and wrap up the show. I’m glad that all worked out for you Jerome. We’ll catch everyone next week.