03:06 - Guests Introduction

• Florian Motlik
  • twitter
  • CodeShip
  • @codeship
• Badrinath Janakiraman
  • twitter
  • github
  • SnapCI
  • [GitHub] snap-ci 05:09 - Extreme Deployment - Definition and Opposite 09:20 - Tools for Deployment/Process
• Writing Tests 10:28 - Application Design
• Ross Snyder: Continuous Deployment at Etsy
  • Etsy 11:38 - Feature Flags
• Agile
• Extreme Programming
• Schoolhouse Rock!
  • I’m Just A Bill - Schoolhouse Rock!
• Staging 18:13 - First Step to Deployment
• Setting up Workflow
• Testing
• Continuous Integration 22:36 - Service-Oriented Architecture (SOA) 24:29 - APIs
• Designing Hypermedia APIs by Steve Klabnik 27:48 - Migrating Databases
• Naming 34:07 - Microservices
• Martin Fowler: Microservices
• Martin Fowler: MicroservicePrerequisites 36:08 - Transitioning to SOA 41:44 - Coupling and Cohesion 43:07 - Deployment with QA Teams
• DevOps
• Agile vs. Waterfall by Rutul Dave 48:11 - Testing Software
• AJAX 52:51 - Irreversible Migrations 56:25 - Tools That Help Deployment
• Amazon
• PaaS
• Heroku
• Packer
• Docker
• Chef
• Puppet
• Terraform
• Vagrant

See Also

Mike Brittain: Continuous Deployment: The Dirty Details

Book Club

Refactoring: Ruby Edition: Ruby Edition (Addison-Wesley Professional Ruby Series) by Jay Fields, Shane Harvie, Martin Fowler, and Kent Beck

Refactoring in Ruby by William C. Wake and Kevin Rutherford

Next Week

ActiveRecord with Eileen Uchitelle

Transcript

BADRI:  Yeah, unfortunately I’m from a, I guess I’m from a place where we didn’t have these games. So, I only knew of this code at a way to get websites to do funny things like make a dinosaur wear a hat. [Laughter] JAMES:  Not that that’s not equally important.[This episode is sponsored by Rackspace. Are you looking for a place to host your latest creation? Want terrific support, high performance all backed by the largest open source cloud? What if you could try it for free? Try out Rackspace at RubyRogues.com/Rackspace and get a $300 credit over six months. That’s $50 per month at RubyRogues.com/Rackspace.]**[This episode is sponsored by Codeship.io. Don’t you wish you could simply deploy your code every time your tests pass? Wouldn’t it be nice if it were tied into a nice continuous integration system? That’s Codeship. They run your code. If all your tests pass, they deploy your code automatically. For fuss-free continuous delivery, check them out at Codeship.io, continuous delivery made simple.]**[This episode is sponsored by Hired.com. Every week on Hired, they run an auction where over a thousand tech companies in San Francisco, New York, and L.A. bid on Ruby developers, providing them with salary and equity upfront. The average Ruby developer gets an average of 5 to 15 introductory offers and an average salary offer of $130,000 a year. Users can either accept an offer and go right into interviewing with the company or deny them without any continuing obligations. It’s totally free for users. And when you’re hired, they also give you a $2,000 signing bonus as a thank you for using them. But if you use the Ruby Rogues link, you’ll get a $4,000 bonus instead. Finally, if you’re not looking for a job and know someone who is, you can refer them to Hired and get a $1,337 bonus if they accept a job. Go sign up at Hired.com/RubyRoguesPodcast.]**[Snap is a hosted CI and continuous delivery that is simple and intuitive. Snap’s deployment pipelines deliver fast feedback and can push healthy builds to multiple environments automatically or on demand. Snap integrates deeply with GitHub and has great support for different languages, data stores, and testing frameworks. Snap deploys your application to cloud services like Heroku, Digital Ocean, AWS, and many more. Try Snap for free. Sign up at SnapCI.com/RubyRogues.] **CHUCK:  Hey everybody and welcome to episode 172 of the Ruby Rogues Podcast. This week on our panel, we have Avdi Grimm. AVDI:  Hello from Pennsylvania. CHUCK:  Saron Yitbarek. SARON:  Hey from New York. CHUCK:  James Edward Gray. JAMES:  Up, up, down, down, left, right, left, right, B, A, selector. [Laughter] CHUCK: I was so going to do that. I’m Charles Max Wood from DevChat.TV. And this week, we have a couple of guests. We have Florian Motlik. FLORIAN:  Hey guys. CHUCK:  And Badri Janakira- [laughs]. BADRI:  [Laughs] CHUCK:  He told me how to say it and I still can’t say it. Janakiraman. How’s that? BADRI:  Hey guys. That was a reasonable approximation. [Laughs] CHUCK:  And David Brady just joined the show. DAVID:  Hello. [Chuckles] CHUCK:  So, Florian and Badri, do you want to introduce yourselves really quickly? FLORIAN:  Sure. Do you want to go first, Badri? BADRI:  Why don’t you go first? FLORIAN:  Okay. So, I’m the CTO and one of the cofounders of Codeship. We do continuous delivery as a service. And yeah, since that’s what you guys wanted to talk about, I’m really excited. BADRI:  And I am Badri Janakiraman. I am the product owner for a product called Snap CI which does hosted continuous delivery as well. And I think the folks at Codeship and the folks at Snap are quite familiar with each other’s products. And I’m interested in the world of continuous delivery for a while. And I’m really interested to talk to you folks today. CHUCK:  Awesome. DAVID:  Cool. CHUCK:  Now, just as a disclaimer, both Snap and Codeship are sponsors of the show. I’m just going to put that out there. These guys are experts. JAMES:  [Chuckles] CHUCK:  But they are both sponsors of the show. JAMES:  So, yes. We are… oh no, never mind. [Laughter] JAMES:  We don’t have to discuss that on the air. FLORIAN:  Oops. [Laughs] CHUCK:  Anyway… JAMES:  I’m super excited that you guys came to talk to us. We’ve been having an email thread this week about what we’re going to talk about and the topics have been flying back and forth. And I couldn’t wait to get to this call. So, let’s do it. I’m excited. [Chuckles] AVDI:  So, I understand that you two can tell us how to deploy our software more quickly than once every six months? BADRI:  Once every six months… AVDI:  Possibly as fast as once every three? BADRI:  Five probably. For most products, who needs more deployments… AVDI:  [Laughs] BADRI:  Than once every six months? CHUCK:  No, no, no. You got it wrong. BADRI:  [Laughs] CHUCK:  So, you have to use the Microsoft method. So, we’re going to deploy. We’re going to release in two months. And then a year and a half from now, we’re actually going to give you the product. [Laughter] SARON:  There you go. BADRI:  Poor Microsoft, poor Microsoft. [Laughter] BADRI:  They’re really, really trying to change these days, trying so hard. It’s like a schoolboy. [Laughter] JAMES:  They’re doing a pretty good job, I got to say. I think they’ve made good progress. [Laughter] BADRI:  Yeah. But yeah, no, I’m not sure if we can tell you how to deploy your software more frequently than once in six months. But we can certainly discuss things that we are doing and see if any of that applies to your case. [Chuckles] SARON:  So, before we get too far ahead, I wanted to lay a little bit of context. What exactly is extreme deployment and what is the opposite of extreme deployment? JAMES:  That’s deploying when you’re tied to a bungee cord. [Laughter] SARON:  That’s what I thought. BADRI:  This is exactly what people used to say about extreme programming. Is it like when you are writing code when you’re skiing down a double black diamond or what? [Laughs] SARON:  That’s how I’d do it. BADRI:  [Laughs] FLORIAN:  I think the term extreme is, so I want to get deployment to a point where deployment is nothing you really think about anymore. It shouldn’t be extreme. It shouldn’t be out of anything. It should just happen. Deployment is something that is just necessary. It’s not necessarily, the act of deployment needs to happen but it’s not necessarily what we as developers even want to do the most, or it’s repetitive. So, we want to have something that is out there and doing it. And we just don’t want to focus on it. We want to focus on delivering our product, working on the code, just shipping new cool stuff. So, I think the act of getting to a point where it seems extreme is where deployment doesn’t seem like anything anymore, where you will lose fear of the deployment. You lose fear of the process. You really don’t have to think about or focus on it anymore. And then it just naturally happens all the time, because with every process, once you lose fear of doing it and you do it constantly, you’re going to do it more in the future. Because you just see how easy it is and how much more fun and how much better it is. So, I think that is taking out, and I think today for me, the extreme in deployment is when I have to get five people together sitting in front of a monitor making sure everything works and spending days or a weekend on deployment. That sounds super extreme to me in comparison to shipping small stuff all the time, small changes that cannot impact that much of one thing. That seems so much more sane than extreme to me. Although I understand the extreme with the amount. So, that’s something where we really want to get people and that’s where we wanted to get our team, where it’s just nothing that you think about anymore. It just happens. It’s natural, just goes with the rest of the process in the team. I think that’s where teams really need to go. CHUCK:  I totally love that idea. And since I never write bugs, I know it’s totally going to work for me, too. FLORIAN:  Absolutely. BADRI:  [Chuckles] So actually, I had never heard the phrase extreme deployment prior to the show, interestingly. But I think when I heard it for the first time… AVDI:  Let me just disclaimer on that real quick. BADRI:  Yeah. AVDI: That was just a phrase that I think I used as we were discussing the concept for this show. So, I don’t want to claim that this is a movement or anything. It’s just I was thinking about all of the practices around constantly shipping a product. CHUCK:  Yeah, so everybody tweet… JAMES:  Avdi, you lied to us? AVDI:  [Chuckles] CHUCK:  Everybody tweet at Avdi with hashtag extreme deployment. AVDI:  [Laughs] JAMES:  It’s a thing now, Avdi. [Laughter] AVDI:  I will be offering certifications for a [reasonable]… [Laughter] DAVID:  Yes. BADRI:  Wikipedia entry is probably already available at this point. CHUCK:  [Laughs] JAMES:  Right. DAVID:  Typing it now. BADRI:  Right. And when I heard [inaudible], when I first heard the term was how the phrase extreme programming came about is it was a concept of taking things that were understood to be good from a development perspective, and then turning them all the way up to 11 saying if they are good in small doses, why not just do them in large quantities? So, if code reviews are good for example, why don’t we have constant code reviews by doing pairing? If testing your code is good, then why don’t we do it all the time by doing automated unit testing? So, that’s how I took this to be. If the ceremony of deployment is something that is important, why not remove that importance away from it and make it something simple, make it a part of everyday life such that it becomes a non-event as Florian was alluding to? And when it becomes a non-event, it just becomes a regular part of development flow. And then all the benefits that you get from that come as secondary consequences. FLORIAN:  Yeah, I think the process of deployment just needs to be boring. I think that’s where we need to go to. It just needs to be boring and it needs to work all the time and just nothing fancy. I think that’s the really important part. BADRI:  Right. FLORIAN:  And that’s really, really hard, to get to that point. DAVID:  Yeah, I was going to say, it’s really hard to get there. I like the idea of having deployment be boring. But it’s exciting and terrifying to get it to be boring. CHUCK:  So, how much to getting it to be boring is your server infrastructure and how much of it is the tool or tools you’re using to get it deployed? FLORIAN:  Most important part of it is the process in the team. I think that as long as you don’t have full buy in, that that’s something that everyone in the team wants. And also, because it’s something that the business needs to agree as well and everybody in the company needs to on as well. So, if you don’t get to that point first, and not everybody really wants to push all the time, I think that step by far, I think technology is relatively easy. It’s still a hard problem, but easier to solve and we’re way more equipped to solve it than the process challenges it has. How do we set that up? Which kind of process do we put in place when we ship stuff that’s actually good? And get everybody to agree that yeah, actually we should spend quite a bit of time on, writing tests or doing code reviews. So, all those processes I think are the really first and important hard step. And once you get buy in from everybody, we can build pretty much anything as a community. So, as soon as that buy in is there, it’s still a hard problem, but then a lot more stuff can happen. But sorry, I think I got into your way Badri, there. BADRI:  No, well I was just going to say that yes, the organizational challenges are the biggest thing. The cultural challenges behind a movement like this are often the biggest hurdles to get over. And then after that, just before the processes, I find that application design is something that’s really, really important as well. You need to design an application that is resilient to this nature of upgrades. Because people are quite familiar with building applications that need to be released as monoliths and performing upgrades, both data and user functionality in large increments, and having to deal with data that’s just at one predictable state and then moving it one delta forward. And to build applications that are designed such that they are resilient, that they work when you deploy all the time, and users do not see any service interruptions, to be able to get to that point requires you to think about your application design quite a bit more as well. And then following from that, in order to support that sort of application design, you need a lot of these technical practices such as testing at multiple levels, the ability to automate your infrastructure provisioning, configuration management, and all of that stuff comes to support your application design. JAMES:  I think that’s a really interesting point and it’s one of the things I’ve enjoyed the most about reading the links that have been provided and stuff like that. I think it was Badri who gave us some links to Etsy’s deployment strategy, which we’ll throw in the show notes. And they talk a lot about the use of feature flags and stuff like that so that they can, deploying something doesn’t mean it kicks in right away. It means that it’s up there, right? And then they use these feature flags and they’re like, “Okay. Let’s turn it on.” And that doesn’t even need to mean that we turned it on for the entire set of users. Maybe we’re turning it on for 10% of the requests or something like that. Then they can watch that live, see how it’s going, are things going okay? Is it tossing errors? Whatever. Then just, if they need to, they can flip the flag back or whatever. So, they don’t even need to go through a rollback process potentially, if there’s a problem. They can just flip the flag back. And some of that’s… BADRI:  Yeah, that’s all in the process of reducing fear. JAMES:  Yeah, that’s a great point. BADRI:  That’s just all, if the fear is reduced, if shipping something that throws an error is just one click away of just being unshipped and not available anymore, then whatever. Let’s experiment a little bit with stuff and just see where it goes. It’s that’s a major issue, then you just won’t do it. It’s just not going to happen because everybody’s going to say no and stop everyone. DAVID:  I like this notion and we’re calling it extreme deployment, but in agile and extreme, we have this notion of holding off on a decision until the last possible moment which is the moment when you have the most possible information. And there’s this pattern that we see in development where you build this feature and the feature is ready to go. And now it has to, it’s like the, oh, nobody will remember Schoolhouse Rock, the old ‘I’m Just a Bill’ song, the progress of the bill going through congress. Anybody remember that song? SARON:  Yes. BADRI:  Yes. DAVID:  Okay, a few of you were alive. A few of were on it. [Laughter] DAVID:  Right? So, it’s like you’ve got this feature and it’s ready to go but it’s in your story branch on your server and then you push it up to the server. And then somebody merges it in and then six months later when you do the deploy, then it goes in. And now you’ve got all these features that you have to worry about. And I like this idea of Etsy with using feature flags where now what we’re saying is, you build this feature, you get it ready to go, you deploy it. Well, there is no get it ready to go. Once it’s testing and it’s green, you put it up on the server and it’s out in the wild. But you’re still holding onto it. You haven’t really forced it on everybody and maybe broken everything. You’ve held onto this feature until the last possible moment when you have the maximum amount of knowledge. Then you flip the bit and it goes live and it’s either party time or panic time. But it’s not panic time, right? It’s either party time or turn it back off time. BADRI:  Right. DAVID:  And that’s fantastic. BADRI: Yeah. And I think you can even take this concept of turning on a feature one step further. Sometimes features may be in a state where they may be turned on but your users may not necessarily have the information to activate it. They may not know where to go on your system in order to find this new feature. So, you even have one additional step which is until the point you actually publish the release notes and announce this feature is actually live, you’ve still got one more step that you can take. [Chuckles] BADRI:  Saying, “Oh, we released it and nobody’s using it.” So, we can see what the system behaves like due to the introduction of this feature when nobody’s using it. And then you know that after you announce it when people start using it, the only thing that could have changed is the consequences of people actually using that feature. And then you have even more information. So, there are just many different levels at which you can release features. And separating release from deployment is one of the core themes that have been coming out over the past few years. People use to conflate those two things saying, “Oh my god. If I deploy every six hours, am I going to be forcing half-built features on my users every time?” And by segregating this notion of deployment and releasing, you remove that fear and you don’t have to deal with that anymore. JAMES:  I think GitHub uses a strategy along those lines where when they initially put features up, they make them available only to GitHub employees. So, as GitHub employees are going around and working, doing their work on GitHub, they’re using the next generation of GitHub, basically. And so, they get an internal testing from them and tweak there. And then eventually, the feature gets released to the general public. FLORIAN:  Yeah, and I think that’s something that’s really important for, just to always use the latest version of your own code, especially with where you can dogfood basically. Because as the GitHub’s people say, if they break their stuff at any point, GitHub is not going to be able to work on GitHub anymore, so they’re going to fix it really, really quickly because it’s really important at that point. So, I think that’s something where when you move to something where silent releases, you just push it out in feature flags or just don’t tell anyone and where release is basically more a marketing issue and not really a technical issue anymore, where just release means, “Okay, now I tell people,” but technically it’s already available for a while. And we’ve certainly done that a lot of times with just pushing stuff out there, trying it ourselves, telling a few people maybe. But it is available for everyone. But then, if you really dogfood and use your own stuff all the time and your product is a big part of your own workflow and your use the latest stuff all the time, it becomes so critical that anything you break you will fix immediately. And so, that’s a good pattern that we found to get into. BADRI:  Yeah, and we do that as well. We do it slightly differently. We don’t do, the Snap production gate, the production pipeline goes through a staging environment. So, the staging environment always contains the next generation of Snap. And that is what builds Snap itself. So, we use our staging environment to build and deploy Snap. So in other words, we can’t promote our software into production if staging breaks for some reason. And staging has all the latest new features. So, it behooves us in order to fix the software to get staging up and running as soon as possible so all bugs are caught, or a good number of bugs that would have otherwise only been caught in production get caught. And we, the Snap team, are the only ones affected. And when we push it out there, there is less risk over there. Once again, this is the sort of thing that’s made possible if you’ve got the ability to bring up environments on demand, push whatever code you want there. And when that sort of stuff becomes really simple and easy to do, you have the flexibility to promote code into any environment you want, see the consequences, and then push it out into production to a wider audience, once again reducing risk. SARON:  So, it sounds like there are two parts to this puzzle. There’s the people and the culture and the fear and the human part of it. And then there’s the technology that allows it to even be possible and done on a broad scale. And I’m wondering, to get more people to adopt the extreme deployment workflow, which one is more important or which one comes first? Changing people’s minds, or making the technology available? FLORIAN:  I think they basically go hand in hand. You have to show people, for example what we always tell people when they start with us is set up the workflow first. So, set something up that you can use easily. Set up a first test. Don’t try to write all of the tests. Just set up the base workflow and do one simple thing so you can show the rest of the team and convince them on the people side that it’s valuable and it helps a lot. And then take it from there. I think that starting with one small application on the side or one small part of your infrastructure, deploying that continuously, and then just growing that workflow into the rest of the company is a lot easier than the big bang release, or big bang change. Because otherwise, you’re just running up against people or people’s opinions, and that’s really hard if you don’t have the workflow and the experience there in place. So, it definitely goes hand in hand. BADRI:  Yeah. And I found that starting with continuous integration and just testing is just a really, really good place. And once teams become familiar with just writing self-testing code and running continuous integration all the time and making sure that the code always get green and the tests are always passing, the rest of it seems to arise as a natural question saying, “Okay, now we seem to be getting our code such that it does not have any regression defects on a regular basis. What’s the next step we can take?” And by introducing the sense of continuous improvement and starting small to demonstrate progress, you stand to do this over a longer period of time. The fact is it takes a pretty long time, even for teams that have done this, to start out from a brand new codebase and get to the point where you’re deploying your software 100% of the time. It could take a fair bit of time. And for teams that are starting out, I’ve heard of teams that take well over two to three years in order to complete this journey. And if you try to abbreviate that entire journey into something that is all going to happen at the same time, you’re most likely going to run into failure. So, starting off with a really small part of the process on a single project, like Florian said, the other aspect of it is that nothing really succeeds like success. So, if you can show success on a small part of the codebase, other people want a part of that action. They want to be the team that manages to deploy frequent, low bug, really high quality, low bug releases. And that spreads a cycle, a virtuous cycle through the organization. And it’s good to be able to tap into people’s desire to be better rather than trying to mandate a process, because that comes down as heavy-handed. Rather, if you can show people what tomorrow could look like and they have a really good example to look up to, that may set into place a virtuous cycle that means everybody wants to follow that. And that’s something we’ve seen in ThoughtWorks as well during our consulting gigs, is you get a pilot project going and that then sets into motion a bigger change inside the organization. FLORIAN:  Yeah, I totally agree. I think that in addition to that, just the important part there as well is for people, I think a lot of people don’t really understand or really have the experience or the knowledge about how to build the technology in a way that is really easy to change. Because it’s just, for a lot of teams just a new process or a new problem. So, I think that a lot is happening around that now, a lot of tools coming out, a lot of blog posts or just content in general written about that, on how to build immutable infrastructure for example where you just place everything. You have those processes that go hand in hand with continuous integration, continuous delivery, but how you build your infrastructure and your technology in a way. And so, these just need to spread a lot more, so people really are aware of what’s out there. How can I do this? How are others doing it? How can I build a standardized way that makes it super easy to get started with small services, service-oriented architecture, and just immutability throughout my whole infrastructure? Each one of those might seem hard to implement. But when starting early with your application or starting small, they’re not that hard to get into your infrastructure and grow it in there. And then it’s just a matter of time until you have the technology in place and the processes before that in place that just make it super easy to just exchange anything, deploy anything new, just make deployment boring in that sense. CHUCK:  So, I want to ask about this. If you have a service-oriented architecture set up, a lot of times one service depends on another. So, you need to deploy them in tandem or you need to deploy maybe the entire system. How do you manage that? Because I think most people, if they’re familiar with deployment, they’re used to deploying one monolithic app or maybe one tiny app that does one tiny thing well. So, how do you manage deployment over an entire system or ecosystem? FLORIAN:  Yeah, I think that in terms of services, so when you have service-oriented architecture, it really needs to be separate services and dealt with as a separate service. So, if you need to deploy one of them, it needs to be able to deploy separately without deploying another one at the same time. So for example, that could be, if it’s an API, you just need to version the API. So, the old system at least for a while, when one of them is on the old one, one of them is on the new one, can coexist and call the other one at the same time. I think that’s something that, especially if you move to a workflow where you have small changes and you don’t do these huge releases so it’s rarely the point that you have to completely change your API and the interface between services, it becomes less of a problem in our experience. And also, it just needs to be seen, like separate services really need to be as if you’d call a third-party service, as if you’d call the GitHub API and want to move from the GitHub API v2 to v3. That’s a little more overhead definitely in how you can roll stuff out. But then it makes it way more resilient and much easier to change. So, I think that those silos really need to be silos. And you need to be able to depend on those services and old versions and new versions to coexist for a while, while deployment happens. But it also puts some work onto the client side in the service, just to really make sure that while you’re upgrading you have a, and with zero downtime. You just have a window where you might need both services or you might want to work with the old one and then the new one. BADRI:  Right. And actually, that is something that I’ve come across more often, is there seems to be this recurring meme I’m coming across of not versioning your APIs. And that does not mean wait all the time. JAMES:  Yeah. I was going to say that. BADRI:  Yeah. You want to go ahead, James? JAMES:  All I was going to say is that we actually do this at work. We have many services that are interdependent on each other and we manage it primarily with the idea of hypermedia APIs. But you could apply these principles in other ways if you weren’t communicating via an HTTP API or something. But most of my ideas come from Steve Klabnik’s book, ‘Designing Hypermedia APIs’ I think. I’ll put a link in the show notes. But the main idea is there are certain rules that an API should follow. And so, just to give an example one of those rules is ignore anything you don’t know. And this is how browsers work. If you have a new tag in the webpage and the browser has no idea what to do with it, it just throws it away. And that means that on the browsers where they understand that tag, then it becomes what it becomes. And in the browsers where they don’t understand that tag, it just gets thrown away. So, if you just use that one idea in your system of interdependent services, as long as they’re trained to ignore what they don’t know, then you can deploy one that has the new thing and the rest of the system should ignore that new thing for now. Then you deploy the other thing that uses the new thing that you deployed previously and that starts to kick things into motion. BADRI:  Exactly. There’s even a parallel to this in how people do database migrations for these sorts of systems, is it should always be permissible to find extra columns or extra data in the database without your application needing to break. So, from the client side, if you follow the protocol of ignoring anything that you don’t understand and from the server side, if you always started out with the aim of widening APIs first before narrowing them down, so then what you get to do is first you get to widen your API. Clients start to use the widened API but they don’t break. And then you can get to see, and you still get to monitor to see if somebody’s using the older version of the API. And at a point where it has minimal usage or very little usage, you can just have a communication with the teams that are using that API, put it on an end-of-life, and then turn it off. In the same way the original Flicker blog post about how to do database migrations in this kind of an environment actually set off this idea in me that you can widen things. You can widen interfaces first before narrowing them down. And widening should not break any client contracts. So, if you apply that general principle to services, then even without versioning, it can help in a lot of ways. FLORIAN:  Yeah, where release becomes more than just one deployment basically. So, release can be comprised of several deployments over steps, over several days or weeks even, until the feature is actually completely changed into a new version. DAVID:  Right. That actually was going to be my next question, is we talked about deploying two separate services. But I’ve done this in the past and I have an answer but we do this thing in the show format where we interview guests who know more than us. [Chuckles] So, I’m going to ask you guys. If you have a feature where let’s say in the customer’s table you’ve got a name field and it’s got first name and last name. And you’ve got a model that’s customer model and you want to migrate in the database to having a last name and a first name. And you want validations in Rails and you want validations in the database. The table, you want those columns to be marked as not null and then you want them indexed, et cetera. How do you release that so that’s in continuous and the site is never down? BADRI:  That is the big kahuna version of this problem is when you’ve got to do addition and removal at the database level, you have to do it in a staged manner. And so, what you might do for this problem specifically is your object, your Rails Active Record Object for example would have a method called name in its public interface. So, at first step, you could start by providing a separate method which returns that exact same data which can then point to a different field in the database. And then as a first step, you can add the first name and last name columns. And in that you can attempt to do a migration of the first name and last name. You can split that up. You can do the data population in there. But you don’t use those two fields to begin with. And this is the part of widening of the API so to speak, but this is widening of the data model before reducing its scope. So, once you widen the data model, you’ve got all the reads coming off of the old column and all the writes continuing to go into the old column. And with the next deployment of the application, you can deploy probably a background service or a background job which every time the name gets written, it copies the name over into the first name and last name column. At this point, you don’t need to validate that those columns are not null or the Rails models validation don’t need to kick in because at this point, Rails is still writing to the old column and all the old logic can exist. In the next version of the application, you can start reading from those new columns and continue writing to the old columns. And that way, your reads come out from the new data. And at this point, you still don’t need to do anything about the writes because the writes are still going into the old column. And then in the last stage, once all your data has, you know that you can read through the new columns, you can start writing the logic to start writing into the new columns as well and eventually remove that background job which copies the data over from the old column to the new columns. And over a sequence of migrations, which if you are deploying multiple times a day can be almost as quick as just building the feature, you can make sure that neither the app nor the database are ever in conflict with each other. FLORIAN:  Yeah. That’s exactly the process that we use as well. And so, what we typically do there then as well is prepare all the pull requests beforehand. So, when we roll out all the pull requests that we’d written, everything, there’s a clear timeline of when is which part getting released. And that makes it really easy to understand the whole rollout. And it takes a little more time than just directly rewriting it, but that’s the price to pay if you want to do zero downtime. JAMES:  It also has the advantage of letting you move in smaller chunks. So, you don’t have to build this great, huge change and then it’s like, “Ugh. We’ve changed how the beating heart of this thing works.” Not only is it difficult to deploy, there’s going to be issues along the way. Would you rather deal with them in the tiny paces or in the one big ball of mud, now we’re trying to make it all work kind of thing? BADRI:  Right, right. For example, you’re adding these columns and you want to index them. Indexing columns or creating columns that are indexed on the fly can take a really long time. DAVID:  Right. BADRI:  But initially, when you’re just widening the data model, you don’t have to create the index. It’s only at the point where you start reading it that you need to create the index. So, things like that can get staggered as well. And you have a lot more flexibility. And you get to pick and choose at what point you want to put in things like constraints and indexes and validations rather than feeling like they all need to go in at the same time and somehow magically the system needs to work in its next version. DAVID:  Yeah. The system that I worked on when we did this, it frequently had data migrations that could take six to eight hours. And when we came up with this method where you could actually implement, you widen your data model, do the data migration, and the service, the whole system is still running, the customers were happy. Because they didn’t realize that the database was just absolutely losing its mind for the current eight-hour, we did it overnight but still, the database was just completely spun around its own axel. And then when it was done, we flipped the data model over to have validations. But the database didn’t. So, you couldn’t save an object now. The objects would say no. You haven’t given me a last name. You can still read the name attribute, but if you want to save me you have to give me a first name and a last name. But there was still no constraint in the database. And when we went back at the very final last step, we added the index for the read back and then the very last step was adding in the actual, the database will throw an error if you don’t provide us with the data. And if you ever had to rollback, you just unwound each step. Take the constraint off the database, then take the validation out of the model, as separate deployments down the line. BADRI:  Yeah, and so interestingly what this then means is the entire team that’s working on this piece of software have all got to be aware that these changes are happening. And everybody stays in constant communication about exactly what is changing and at what point in time. But because the delta in each step is so small, people can still fit it in their heads. DAVID:  Yeah. BADRI:  If the same thing was happening over a three-month period, people can’t remember what everybody was working on for that three-month period. But if you just have to remember what the person sitting next to you was working on four hours ago, that’s probably much easier. And everybody can cross-check and verify each other. And the team reinforces their collective desire to move forward quickly. JAMES:  So, if I can take this in a slightly different direction, Florian you said something controversial in the emails that went back and forth planning for this episode. So, I’m going to try to get you in trouble now. FLORIAN:  Perfect. [Chuckles] JAMES:  Yeah. I’m glad you’re so willing. You said monolithic applications don’t scale in terms of their development complexity for the speed we want to release, so service-oriented architecture or microservices. Can you explain what you meant by that? Because I love it. FLORIAN:  Sure. We’ve all been there with the monolithic beasts that are just, we don’t understand them anymore. And we don’t understand all the models in there, all the classes in there, or how they really work together. And at some point, all the models, just everything, get entangled in that one big mess. So, we need to have more developers on there because it gets more complicated and more complicated. So, more developers come in, write more code that is even more entangled. And that just doesn’t scale in terms of the team. You might be able to make it scale on a technical level. But even that’s really hard if you have so much stuff in one thing, basically. But it just doesn’t scale on a team level. And especially not if you want to release quickly and all the time. If you want to release a lot of small changes to that big monolithic application, it just doesn’t work anymore. And I think that there seems to be more and more push generally but also in the Rails community for a while, just to be able to do those microservices and service-oriented architecture and [inaudible]. So, in my opinion, that’s something that’s really hard to do from the beginning. But starting out with small services that just care about one specific task or a specific problem in your application and combining those services up into higher layers that really fulfill the whole application, that will be the dream. That’s still really hard to do because in the beginning it’s just so much easier to write the monolithic app, have all your models in there, all your controllers, all your logic in that one specific monolith. So, we still need to evolve the technology and the processes that we have to be able to really do service-oriented stuff or microservices from the beginning. And I think that’s a big part where we need to learn a lot more. But then those large applications, you can’t use them if you want to deploy 50 times a day. That just… or you can use them but it’s really hard and really expensive and really painful. SARON:  So, if it’s hard to start off with SOA, at what point should you make that transition or start thinking about that. BADRI:  That’s a really interesting question. And I think it’s really hard to answer and I think this is where the monoliths get created, because very few people start out with a desire to build an unmaintainable monolith. But somehow along the way, the majority of people seem to get tripped up. And really, I personally do not have too much experience with it. But one thing I have found to be useful is a similar understanding to what we talked about when SOA was getting popular, is to break things along business boundaries. So, if there is a business unit that cares about a particular set of operations or a particular set of objects, then that is a good place to typically put in a service layer if only because typically business units within an organization tend to have their own vocabulary and their own understanding of what processes are. So, that is one rule of thumb that you could possibly use to carve up your services. If you’ve got an order department, if you’ve got an inventory department, if you’ve got a sales department, those might naturally fall out as services in your application. The other things that we try to do as services are slightly trickier. So, if you need to make search or reporting be a service, that then becomes slightly trickier. And I don’t think there are answers that fall in one direction or the other at this point. People are just trying to explore what it means to do that properly. I have heard of people implementing even things like search which actually spans multiple services as a service in itself. But I’ve heard of people just trying to build it as just an independent entity or just inside each of the individual services as well. So, there’s lots of back and forth on that. But along business units tends to be one easy way you can think of services to begin with. So, if you’re building a monolithic application that touches multiple departments and multiple people from different teams in your organization using it, that might be one way to split it up. FLORIAN:  Yeah, I like the thought of splitting it up as well into products. So, if you consider your big monolith, maybe you’re able split it off into internal products where one team or person is responsible for that product to really make it run well and really provide a good service, a good product, to some other consumer, any other kind of service. So, it’s really something where you try to build a service that around a specific cause and really then build them together into larger class. But it’s really vague and really hard to decide what exactly. So, I’ve recently been to the DevOpsDays here in Boston. And we had a long conversation with a lot of people and people running live service clusters. And it’s just a hard problem to decide when to split stuff off. And I think that’s where we need more, generally in software development, we need more people doing it, writing about it, their experience, how they split off specific parts. Because that’s been discussed I think in a more theoretical manner in a lot of ways than really practically, like how are people actually doing this? It has, there are blog posts and content about that, but not as much as we frankly need I think for small and medium-sized companies. Obviously large companies write about their stuff. But for small or medium-sized companies it’s still not been a lot of discussions around the topic, how they could do that. JAMES:  One of the things that I would add to that is just to remember that you don’t have to get it right on the very first time. You can change your mind. So, I do work at a small to medium-sized company that does an SOA architecture. We have a pretty large deployment of services. And we did do SOA from the start, sort of, with a couple of caveats. And that was interesting. I’ve learned a ton about it. I’ve definitely not written about it enough, because I’m still figuring a lot of this stuff out and trying to decide what I know. But the one thing I will say is that we do change our mind. At one point, we had this service and we thought we had very good reasons for putting these things in it. And then we tried it that way for a while and used it. And then we figured out more knowledge of what we were doing. And then we’re like, “Oh, this is actually some separate things. Let’s bust it up.” And so, we changed it and redeployed it. And the good thing is, in an SOA architecture, it’s usually not that… it’s less hard I think I should say, to bust the pieces up more or combine pieces, or shuffle them around. Because you’re generally working with smaller puzzle pieces than the overall picture. FLORIAN:  I just wanted to have one more quick one about it. All the decisions you talked about and how you split that up, that’s perfect topics I think for a blog post. Because I think a lot of people stumble exactly there on how to split their stuff. And having those blog posts, just seeing how these SOAs evolve and how people get it wrong and what’s the right [inaudible] or it might still be the wrong decision as it turns out later, but seeing that evolve is something that’s very helpful. It is of course a technical problem and there are large technical challenges with that. But I think it’s more of an understanding. How do we actually build that? And just a fear of getting started is a lot more holding people back than actually the technology. So, those blog posts or if more people out there want to write blog posts about how they decided about their SOA and their specific services, please send all of them to me. I want to read all of those. [Chuckles] BADRI:  James, can I ask you a quick question about what you just mentioned. So, you said you don’t have to get it right the first time and you can change all the time. One thing that used to be common when the concept of components was quite popular in the Java world was to talk about coupling and cohesion. So, things that went together or changed together but did not change along with other things. So in other words, there was high cohesion but low coupling with other things were probably good candidates for making into a component by themselves. Have you observed any such thing while breaking up services as well or did it just naturally happen to be that way? JAMES:  It’s hard to say, because I think it always depends on the nature of the problem a little bit. In our particular problem, we have these certain kinds of services that naturally show up and it makes sense to divide along those lines. And they all have differences internally but they can for the most part be made to conform to a similar interface. So then, that leads to us being able to extract the like elements of those and build on that. So yeah, I don’t really know in the general sense. Definitely in our problem there is, but then I am not sure how much that translates to the general sense. BADRI:  Right, cool. DAVID:  Okay, can I take us in an interesting different direction? JAMES:  Do it. SARON:  Go for it. CHUCK:  Left turn, signal on. DAVID:  At least I hope it’s interesting. So, a lot of companies, even those practicing agile especially a lot of the larger companies that have a lot of liability and therefore fear about releasing the wrong thing, they will have dedicated QA teams. And I’ve seen friction between trying to move towards more agile and having a QA team and that sort of thing. But the reality is there are still companies out there that consider it a best practice. And I agree with them for their environment. It’s still a best practice to have solid QA teams. Now, the problem with QA teams, or rather the advantage to extreme deployment is that we’re talking about making deployment very boring and low ceremony, it sounds like. Here is the problem. I’ve never met a QA tester who didn’t want to maximize boring. CHUCK:  [Laughs] DAVID:  But I’ve never met a QA manager who did not want to maximize ceremony around deploying. So, how does extreme deployment fit alongside a QA team? BADRI:  I think this has got parallels to a couple of different things. The DevOps movement for example, the Dev and Ops teams have similarly been separated by the sort of artificial constraint that was placed saying the development team is responsible for pushing changes out as fast as possible and they’re incentivized to do that, whereas the ops team is incentivized to resist any change in the interest of maintaining overall system stability. So, if you continue to set up your organization such that there is this weird antagonistic tension between departments, you’re just setting yourself up for a world of hurt. So, the easy way to solve, well not an easy way [chuckles], the easy way to talk about solving this problem is to break down those barriers. In reality, the cultural change required is enormous. And people typically have a very good reason for setting up these sorts of departments because there is definitely a lot more information exchange that can happen. There is career building and opportunities that can happen within these small groups. But there is no reason that these groups cannot work with a development team in order to release a product. And that comes from having a product mentality. And so, you can still have your departments in order to do all the good things that come from having a department, which is knowledge management, training, capability [to building]. All of that stuff comes from having that and at a large enough scale. But that does not mean that these teams have got to be made to work against each other organizationally. DAVID:  Nice. BADRI:  If at the organizational level, at the top level, you set people up and you incentivize entire product teams to deliver and make everybody point in the same direction, then there is much more collaboration between the teams. And the easiest way to make it happen is to show that developers care about releasing good quality software and QAs care about being the process through which they can feed that information back to the development teams. And you build quality in, in the same way that the DevOps movement tries to build reliability and resilience into the software. The QA teams and the development teams working together try to build quality into the software. And if you have that mentality, if you start with the mentality of building quality in and building resilience in rather than making them be externalized entities, then I think you have a much greater chance of success. DAVID:  I love that. I’ve never considered the endemic emotional sub-problem of developers trying to go as fast as possible and QA trying to maximize quality. Can I push you a little further though? Because I’ve worked with… BADRI:  Sure. DAVID:  Every QA team that I’ve worked with, some of the recent ones, especially as we move towards agile, we do get towards more of a product mindset where there’s just this wave of visible relief that comes off the QA team when I walk over into their department and say it’s my job to provide you with everything you need to get into every dark nook and cranny of this software to test anything you want. And they go, “Oh, good. You’re not here to fight with us and to try and hide your crimes before they get released into production.” And I don’t want to give away what we did, but I will say that prior to moving to agile, it was very much like a waterfall deployment. Development would write a feature, we would “release” it. And what that meant was QA could now take a run at it and they would put on a staging server and they would test it. And if they approved it, then it went into the next release. And on deployment day, this is serious old school, we had deployment day where everyone stops what they’re doing and they do the deployment. And all the developers sit around twiddling their thumbs waiting for QA to test everything while they just sit around. And QA comes back and goes, “No. There’s a missability bug where this branch and this branch, when they intermix they cause a bug.” And that’s bad. So, the developers jump on it and they fix it. And that hasn’t entirely existed on teams that have moved towards agile. But again, can I actually pin you down on the actual mechanics of if you’re a developer and I’m on QA and we have agreed to cooperate and we’re doing extreme deployment, how do I test your crap? BADRI:  Right. [Laughs] So, that’s an interesting problem. So, one of the solutions that we have adopted in places on different teams that I’ve worked on is to have a pull-based mechanism for the QA team. So, in the same way that you said the QA team visibly relaxes when you go in and say, “I’m here to make it easy for you to test my software,” one of the things you can do is if every QA person, if every person on the QA team who’s going to test the software, can pull whatever version of the software onto an environment, on demand, test it, then they are not relegated to testing the software right at the end when every story is deemed development complete by a developer. DAVID:  Right. BADRI:  They get to make their process be iterative as well. But in order for them to do this, a QA team’s not necessarily specialized at infrastructure automation or build automation and being able to promote builds into environments. They need help in order to do this. And this is where the collaboration between the development teams and the QA teams in this sort of an environment can really, really help. If you set up processes such that any version of the software can be deployed on demand, or can be pulled in by anybody on the testing team, they tend to trust you a lot more because now, you’re making their job easier. They don’t have to wait until the end. They don’t have to keep long lists of things they need to check right at the end and then throw that list back at you saying, “Here’s what you get for not telling me what you were doing for three months.” DAVID:  [Laughs] BADRI:  [Chuckles] You see? Because at the end of it, they are people too, and they are trying to do the best job they can. But if we work against each other then it doesn’t produce results. But if you make it easy for them to test then it works really well. The other thing we’ve done is pairing with QA teams. So, if for example you write software, I’ll take a pathological example. If you put an element on a webpage that doesn’t have an id or a distinct class on it, then a tester can’t locate that with a testing tool in order to assert its value or presence or absence. That is a problem. If you plan to change the interaction of a page from being a full page reload to being an Ajax reload and you don’t tell your QA team how to do a wait for loop in order for an element to appear or disappear on the same page, that’s going to make their test be flaky and then somebody’s going to point at the QA team and say, “You wrote a flaky test,” when really we changed the logic when the developers change the logic from a full page reload to an Ajax load. So, when you make these sorts of changes, pairing with the QA team in order to actually implement some of the things that they may not necessarily be experts at lends to an atmosphere of trust between the QAs and the developers. And that can go a long way towards building quality in into the software right from the start. DAVID:  I love how you’re blaming the developers for these things. It’s almost like you’re considering QA people to be actual people. [Laughter] BADRI:  I know, I know, I know, it’s really, really awkward. It’s really, really awkward and I’m pretty sure it’s not fashionable to do that. But I… [Laughs][laughter] BADRI:  I happen to see people that I work with, and they are nice people and they’re trying to do their job. And I would like to help them if possible. JAMES:  I think what I’m definitely hearing here is that in order to move to continuous deployment, what we need to do is we need to break down these walls like the DevOps culture has done where developers and ops people, they work together to create the environment. And then again, on the same side with QA and stuff like that. And the more of those walls we break down and work together, the better we can do, which I think is just awesome. And if moving to continuous deployment also makes us use these better processes in addition to building more testable software and better manage software, then it’s just moving to this seems to improve everything in the workflow which I really like. FLORIAN:  It just puts the right incentives for everyone on the team. And because you have to, as Badri said, you move so quickly, you [inaudible] over everything, you can’t have those handoffs anymore. And really, QA shouldn’t be expensive replacements for computers to run it automatically. We should write those tests, but they should really focus on the quality. And that’s not clicking through or just writing the tests themselves but really focusing on being [inaudible] the teams and making sure the quality of the stuff that gets shipped is actually as high as we expect it to be, and also the people responsible for that. CHUCK:  What, so you want quality assurance to be quality assurance? [Laughter] FLORIAN:  It’s crazy, I know, yeah. It’s really strange apparently. CHUCK:  Alright. Any more questions, guys? DAVID:  Irreversible migrations, question mark? [Laughter] BADRI:  Yes? FLORIAN:  Part of the game. DAVID:  Part of the game. How do you do that? So, we talked earlier about the customer name table. I’ve got a name in there like David A. Black. And I need to make first name be David, last name be Black. But at the reverse migration, I’ve lost that middle initial. How do we make that be part of the game? JAMES:  Don’t do it. CHUCK:  [Laughs] JAMES:  I’m serious. I think that’s my answer. That is definitely not a case where you can’t split it into, how many negatives does it have in there? Maybe I got lost. You can put that in two fields or you can separate it out or whatever. DAVID:  Okay. I can give a better example. The customer name field contains James Edward Gray II. [Laughter] JAMES:  [inaudible] DAVID:  I know you have said that you have had systems that think your last name is Edward Gray 2. JAMES:  Right. DAVID:  Right? JAMES:  Yeah. CHUCK:  It’s just 2. DAVID:  Now, that’s a different problem. That’s the problem of getting the name division wrong. But let’s say I do actually get it right and now I’ve got you stored in the system as James Gray, which technically is your dad, right? But aside from the fact that we never should have done that migration in the first place, [laughs] how do you handle that irreversible migration? Do you hang onto the data for six months and then decide to throw it away? Do you just suck it up and go with it and just realize that, “Hey guys, we can’t roll this one back. So, everyone, all that panic we haven’t had since we went to extreme deployment, it’s back.” JAMES:  Add a feature flag so you can shut the feature off. FLORIAN:  Yeah. Deploy a second migration a week later or a month later that just removes the column if necessary, but just keep as much in the data around until you’re super sure it’s not going to be used anymore. And there’s just move over, over a period of migrations. And then just wait until you’re sure that nobody’s going to have any problems with it. So, that’s what we’ve been, when we did that, we’ve been successful with it. Obviously, so something that we do as well is just take migrations, put a dump of our database everyday onto a staging system, and run the migrations over them before we deploy just to make sure nothing breaks. And typically we do small migrations there. Nothing really breaks or they’re pretty fast to run over that staging system as well. That’s really important. But then just don’t delete data until you’re really, really sure, until everybody asks you, “Why is that field still in there? Didn’t we remove that six months ago?” DAVID:  Right. FLORIAN:  When you’re at that point, then you can remove it. BADRI:  Yeah. DAVID:  I would also add that a step you have to add going forward, which is that you have to continue to populate that legacy field. So, when somebody comes in and they type in first name James, last name Gray, you have to save first name James, last name Gray, and you have to save customer name as James Gray. So that when you roll back, you’ve still got your legacy column. JAMES:  Right. Avdi? BADRI:  Or you could separate the reading and the writing and keep a background that breaks the name down. So, you could continue to write the one big name and you could have a job that populates the new columns. And that way, you separate the reading process and the writing process. And I’ve found that that separation really helps as well. DAVID:  Right. AVDI:  So, basically in summary, to quote Bane, “When Gotham is ashes, then you have my permission to delete the column.” [Laughter] AVDI:  I do have another question which I think leads us into the picks. And that question is, are there any particular tools that you find make all this stuff a lot easier? FLORIAN:  Yes, certainly. [Laughter] JAMES:  Excellent. FLORIAN:  You don’t need to say… BADRI:  I should [inaudible] our own product. [Laughter] FLORIAN:  No, I mean in terms of just building infrastructure. The way we build infrastructure generally is really to make every part of it replaceable at any point. So, for example we use for Amazon, so generally use platform as a service if possible because then other people have to deal with it. So, we use Heroku for a lot of stuff. Deployment to Heroku is simple. They follow the model of immutability so everything just gets replaced at any point. So, that works really well. On Amazon themselves, there’s Packer, there’s Docker, and tons of other tools coming out now that really make it very easy to build the infrastructure in a way where change is not really a problem because you just replace stuff. In our experience, any kind of Chef or any other, Chef, Puppet, or other tools where you just try to get convergent state at some point. And maybe some deployment fails on some servers, but then Chef tries to get that in a place where it is actually at the point where you expect it to be. That’s really painful. So, really building images and then starting from those images and the immutability thought, but basically it’s the same that we do in software with classes and instances. You just have that class and then you instantiate 1, 15, 50 or a thousand of those machines. And the same thought process can be used for infrastructure as well. You just need to start building it in a way. And there are services and microservices that help a lot where you can replace any part of your system. You have load balancers in between where it can switch between services and between servers then. And that’s something. So, those kinds of tools, Packer, Docker, Terraform, all of the, from the Vagrant guys, from HashiCorp, they are doing great tools for that. That’s really helpful. Obviously, a good continuous delivery system is as well. And I’m sure that we have all of our different opinions on which are the cool ones. But generally, yeah I think outsourcing as much as you can and focusing on the stuff that is really important and building everything in your infrastructure in a way that can be changed at any point without you noticing, without you knowing, that’s really the key. BADRI:  Yeah. And the interesting thing is we do immutable environments as well but we use Chef and Puppet to bring up the immutable environment. We use base AMIs which don’t contain too much. We follow Amazon’s mantra of just enough OS. And then we use Chef to bring up the environment, but the environment is completely brought up from scratch each time and then we deploy your application onto it. So, there aren’t any of those convergence issues that we typically deal with. So yeah, that is one thing that we follow as well. In addition to that, I think everything that Florian said is very relevant to all teams trying to do this. CHUCK:  Awesome. Alright, we need to get to the picks. David, what are your picks? DAVID:  I’ve got three picks, really quick. The first one, they’re all to help you get a little more control over Twitter and Facebook, over your social media, because you’re being gamed by people. And there are people now that are fighting back. And so, here’s some information that you need. The first one is a Twitter account called @savedyouaclick. And you need to follow this account because all this account does is goes and finds click-bait, which is a provocative headline and a link. They retweet the click-bait, they remove the link, and then they just give you the spoiler of what the answer to the headline is. JAMES:  That’s amazing. DAVID:  So, it gives you a spoiler. So for example, retweet, re-quote, “You’ll never guess what Facebook is cracking down on now.” And Saved You a Click retweets that and says, “click-bait.” It then retweet the BuzzFeed, “Did Tony really die at the end of Sopranos? David so and so reveals the answer.” And then they just put, “No.” [Laughter] DAVID:  It’s fantastic. It’s basically if you like spoilers, it’s the spoiler Twitter feed. It is absolutely fantastic, because they’re not spoiling good entertainment. They’re spoiling people that are trying to make you click to artificially inflate their search engine optimization. So, Saved You a Click on Twitter is a fantastic account. You absolutely need to follow it. The second one is a pair of articles that just absolutely blew my mind. So, on Facebook when you click on the like button, you are committing an economic transaction and Facebook tracks it. And they will immediately show you more stuff related to what you just clicked like on. And they are absolutely gaming you based on the things that you clicked like on. So, the first article is on Wired and it’s called ‘I Liked Everything I Saw on Facebook for Two Days. Here’s What It Did to Me’. CHUCK:  [Laughs] DAVID:  And in the spirit of Saved You a Click, I’ll tell you. His entire Facebook feed turned into nothing but bots and spam and extreme political left, extreme political right. It went absolutely crazy. And furthermore, everyone that he was friends with, their walls filled up with nothing but Matt liked this, Matt liked that. And it was like crazy… CHUCK:  Wow. DAVID:  Extreme political stuff. The second article is on Medium.com and it’s called ‘I Quit Liking Things on Facebook for Two Weeks. Here’s How it Changed My View of Humanity’. And again, in the spirit of Saved You a Click, it got rid of a lot of the ads. It got rid of a lot of the bots. And what this person did instead of liking, they commented instead. And this caused this person to begin to engage more humanly and more humanely with people on Facebook. So, Facebook probably isn’t going to like people abandoning the like button, but there are two great reasons why you want to consider it. So, those are my picks. CHUCK:  Alright. Avdi, what are your picks? AVDI:  As I do every few years, I decided to throw away my Emacs customizations and start again. And this time, I decided to do it based on the Prelude project from Bozhidar Batsov. CHUCK:  Yay! AVDI:  I’m sure I’m murdering his name. But yeah, I decided to just start with that this time. And the reason I decided to start with that is I’m a big fan of his blog, Emacs Redux, in which he always brings up really cool either Emacs extensions or Emacs customizations that make a lot of sense. And then the blog post, more often than not end with, “and of course this is configured by default in Prelude.” And having [chuckles] read so many of these over the years and thought, “Hey, that’s a great idea. Hey, that’s a great idea,” I’ve decided to finally just bite the bullet and get Prelude, because clearly he knows what he’s talking about. So, I’m going to try to rebuild my configuration based on that now. And already, I’m really liking it. There’s a lot of smart stuff in it. DAVID:  I will say, Avdi I will say if you’re using Xiki, they do not play well together yet. But I’ve been chatting with Craig and trying to get them to play. AVDI:  Yeah, I wouldn’t expect them to, but good to know. Another little tech pick is the fingerprint GUI project for Linux. If you have a Linux machine and it has a fingerprint scanner, I definitely recommend checking it out. This stuff isn’t configured or available in Ubuntu in default but you can add a PPA which will add it to Ubuntu and integrate it pretty nicely. And basically, got a fingerprint reader, it will just start popping up fingerprint dialogues anytime you get prompted for your password, which is great, because anytime that I want to do sudo apt-get something or gem install whatever, it will prompt me for my password as usual. But then it will also pop up a fingerprint GUI prompt. And I can just slide my finger instead and not type in my password. And since I seem to be incredibly prone to mistyping my own password, this has been a huge timesaver. And yeah, yeah, I know. Fingerprint readers aren’t quite the most secure things in the world, but I’m willing to make the compromise. CHUCK:  I want it to pop up if somebody else uses it and says, “You’re not Avdi. Please use your middle finger.” [Laughter] CHUCK:  Alright. Saron, what are your picks? SARON:  Sure. So, I had two but I’m going to add one more because David reminded me of it when he was talking about Facebook. So, there have been a lot of comparisons between Twitter and Facebook specifically about what happened in Ferguson about the shooting and about which social media network did a better job of spreading the news of what happened. And Facebook got a lot of slack for it just not being very good and Twitter being amazing and how it just rapidly spread. And there was a blogpost from a woman who was really frustrated because she just wasn’t seeing anything about Ferguson on her Facebook feed. And she suspected that it had to do with the algorithms just not prioritizing that. And so, what she did was she created a fake wedding post because she knew that Facebook would rank that very highly and would want to share that. And so, it was a fake wedding announcement and it says, “Hey guys. Not actually getting married. Just really wanted to make sure that you saw the news about Ferguson and wanted to make sure that you’re aware, because I don’t think Facebook is showing you this information.” JAMES:  So clever. SARON:  Yeah. It was really, really interesting. So, I’m going to add that and share that with you guys. The two that I originally had, one is about a thing called HeatSeek. And so, I think this is a New York City thing. I don’t know if this is entirely for the US. But landlords are legally required to keep the temperature over a certain number specifically in the winter, but they don’t always do that. And they don’t always respond when tenants complain about how cold their apartments are. So, to help tenants get heat, a few Rubyists, a few developers made a little 3D printed sensor and wrote a program that detects when the heat drops, send that information in and gives you a little alert so that tenants have a better and easier time recording that data and having proof that they’re not getting the heat that they legally need and deserve. And so, it’s called HeatSeek and they launched a Kickstarter to help them build these things. And the best part about the story is one, the program is written in Ruby, and two it’s written by a group of programmers that have been coding for about six months. So, it’s really awesome that people who are new to programming are taking a stand and doing civic engagement with our awesome language. And the third one is this beautiful 3D Wall Mural that I found that I’m really excited about. I love 3D drawings, the ones that look like they’re 3D but they’re not. And there’s one that’s done on a wall and it’s like a graffiti image but it’s so cool. And it took me a very long time to be convinced that it wasn’t 3D. So, I’m really excited for all of your guys to see that. And that’s what I got. CHUCK:  Awesome. James, what are your picks? JAMES:  Well, since three seems to be the magic number, I have three as well. This first one is a blog post called the ‘Introduction to A*’ which is a path-finding algorithm used in games. And a friend of mine showed me this link recently and he’s like, “Are you familiar with this article?” and I said yes because I thought I recognized it. And I learned A* from a really great article. And then actually, I got to looking at this one and realized it was brand new and I learned a long time ago. And so, this couldn’t be it. And this one is actually even better than the one I learned it from. So, sorry Nathan, I was wrong. Now you have it on recording, so there you go. Everyone should read this article, even if you do not care about games at all. You need to go read this article because you have no idea what blogposts can be until you go read this article. So, that’s all I’m going to say. Hopefully, that’s enough to get people to check it out. Also, I was whining to my role-playing friends recently that I don’t get enough role-playing time these days. I love a good role-playing game. And a buddy of mine, Luke, told me to check out Fiasco. And the cool thing about Fiasco is it’s a GM-less role-playing game. And you do the setup as part of a play. And so, you can play a full game in about two and a half, three hours or so, with setup and everything. And nobody has to spend a bunch of time setting up something beforehand. We played our first game a bit the other day and it was so much fun. So, if you love role-playing and you haven’t had time to do it in recent years or whatever, this is a great way to go. And my third pick, the way I found out Fiasco was good was TableTop. I can’t believe I haven’t brought up TableTop on the show before. That’s Wil Wheaton’s video series where he plays games with his friends, which are famous stars pretty much. And it’s great. And so, when my buddy told me about Fiasco, I went and watched the TableTop episodes. It’s broken into three parts, the one on Fiasco, about Fiasco. And they’re absolutely great. And that’s what talked me into getting the game and trying it out and realizing that it’s just great as it is. And TableTop does lots of other games, many of which I’ve purchased because of their coverage. So, if you don’t know about it, you should. I will link in particular to the Fiasco episodes because they’re just awesome. Those are my picks. CHUCK:  Alright. I just have one pick and it’s a comic that I thought was funny. It’s The Oatmeal. TheOatmeal.com/comics/kids. I just laugh because I see people do this kind of stuff all the time. So yeah, so I thought I’d share people giving great advice for, “Well, you should have kids.” SARON:  That was a great one. I love that comic. CHUCK:  [Laughs] It’s funny because it’s, well, it’s a little bit exaggerated but it’s somewhat true to life. Anyway, so yeah, that’s all I’m going to pick. Badri, what are your picks? BADRI:  I actually just had one book that I pick. It’s a book that I’ve picked up a really, really long time ago. It’s called ‘Generative Programming: Methods, Tools, and Applications’. It’s by two authors, Krysztof… I’m probably going to butcher their names, Krysztof Czarnecki and Ulrich Eisenecker. And what I really like about this book is it is an attempt to do domain-specific languages with C++. In other words, it’s an exploration of really, really good ideas in a really atrocious language with STL. It’s just a whole load of template magic. And it’s wonderful to be able to see how far you can actually make C++ bend, a language which seems to be famous for its rigidity. But I think it’s incredible what they managed to do with it. And I really like some of the ideas that they explore, ideas that have since become really common and popular, things like how do you implement aspects? How do you implement traits? How do you implement a full-fledged language workbench? And what is this intentional programming thing all about? And it’s all written from way before these technologies became common or became popular. And in a sense, I find it similar to ‘The Art of the Metaobject Protocol’ which is a much more readable book, which is about, how do you build something like Common Lisp and a lot of the Common Lisp features, starting from a base Lisp? And I find this book really fascinating for that. And this is what got me really interested in domain-specific languages a really long time ago. And I think it’s a good book for folks to try and sink their teeth into. If nothing, it will give you a good few hours of trying to remember what the 90s were like playing with C++. CHUCK:  Awesome. DAVID:  I used to have the pragma warning number memorized for when you have an STL vector of string, standard vector of standard string. And it would automatically raise a debugger warning that the debug symbol for that was more than 124 characters long. And so, you would do a pragma warning disable on the number. I can’t remember the number anymore. And I’m so happy. BADRI:  [Laughs] JAMES:  That’s awesome. CHUCK:  Alright. Florian, what are your picks? FLORIAN:  First pick is, I don’t know if you’ve seen that one, but Tim’s Vermeer. It’s a documentation by Teller from Penn and Teller. It starts out as about Vermeer, a painter, a Dutch painter, but then goes into how that Tim Jenison just redid a Vermeer by himself by hand with just technology. Painting it by hand, but with help by technology. It’s a great movie. You definitely need to see it. And he basically proves how Vermeer could have painted that thing in the past. But it’s really about him in the end and how, what an impressive engineer that guy is. So, for all, for me as an engineer, it was super interesting and super nice to watch that guy and how obsessed he is with painting his own copy of a Vermeer. Really great movie. My second pick is Wunderlist. I’ve tried pretty much every to-do list or thing under the sun. And I’ve stuck with Wunderlist now for quite a while. They have really great applications for mobile or on the Mac and works really, really well. So, if you guys are struggling with managing your tasks as much as I have in the past, Wunderlist is definitely a great pick. And so, the third thing for me is, so recently because I don’t get to code a lot anymore or as much as I would want to but I still want to do more CS stuff or get more stuff in that regard, I recently started reading some CS papers again. So, I just started reading one that Google released last year about Omega, how they schedule tasks across their whole infrastructure, which is super interesting. Complicated enough that it satisfies my engineering heart when I don’t get to code a lot, but still something that I can read relatively easily in the evening. So yeah, that’s my third pick. For people who don’t get to code as much as they want to and spend too much time in email, read some CS papers. It’s really fun. CHUCK:  Awesome. Alright, we are drawing closer to the book club book where we’re going to be reading ‘Refactoring: Ruby Edition’. So, check that out and thanks guys for coming. Really appreciated your expertise here. DAVID:  Yeah, this was awesome. CHUCK:  If people want to learn more about what you guys do or how to get better at this stuff, where do you recommend they get a hold of you? BADRI:  I can be reached at badri@thoughtworks.com, B-A-D-R-I at ThoughtWorks dot com. And I’m more than happy to answer any questions that come in from folks who listen to this podcast. FLORIAN:  Easiest place to reach me is on either Twitter, so FloMotlick, just my name, or just send me an email at Flo@codeship.io. And I still want all of those SOA blogposts. If you’ve got them, send them to me. I want to read those. CHUCK:  Awesome. Alright, well we’ll wrap up the show. Thanks for coming. BADRI:  Thank you. Bye-bye. FLORIAN:  Thanks for having [inaudible].[A special thanks to Honeybadger.io for sponsoring Ruby Rogues. They do exception monitoring, uptime, and performance metrics and are an active part of the Ruby community.]**[Working and learning from designers at Amazon and Quora, developers at SoundCloud and Heroku, and entrepreneurs like Patrick Ambron from BrandYourself, you can level up your design, dev, and promotion skills at Level Up Con taking place October 8 **^th and 9** ^th ** in downtown Saratoga Springs, New York. Only two hours by train from New York City, this is the perfect place to enjoy early fall and Oktoberfest while you mingle with industry pioneers in a resort town in upstate New York. Get your ticket today at LevelUpCon.com. Space is extremely limited for this premium conference experience. Don’t delay. Check out LevelUpCon.com now.]**[This episode is sponsored by MadGlory. You’ve been building software for a long time and sometimes it’s get a little overwhelming. Work piles up, hiring sucks, and it’s hard to get projects out the door. Check out MadGlory. They’re a small shop with experience shipping big products. They’re smart, dedicated, will augment your team and work as hard as you do. Find them online at MadGlory.com or on Twitter at MadGlory.]**[Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.] **[Bandwidth for this segment is provided by CacheFly, the world’s fastest CDN. Deliver your content fast with CacheFly. Visit CacheFly.com to learn more.]**[Would you like to join a conversation with the Rogues and their guests? Want to support the show? We have a forum that allows you to join the conversation and support the show at the same time. You can sign up at RubyRogues.com/Parley.]