165 RR Systems Programming Tricks with Julia Evans

The Rogues talk systems programming tricks and hacks with Julia Evans.

This episode is sponsored by

comments powered by Disqus


AVDI:  I’m drinking breakfast this morning.

DAVID:  Smoothie or, “Hi Avdi, welcome to meeting”?


[This episode is sponsored by Rackspace. Are you looking for a place to host your latest creation? Want terrific support, high performance all backed by the largest open source cloud? What if you could try it for free? Try out Rackspace at RubyRogues.com/Rackspace and get a $300 credit over six months. That’s $50 per month at RubyRogues.com/Rackspace.]

[This episode is sponsored by Codeship.io. Don’t you wish you could simply deploy your code every time your tests pass? Wouldn’t it be nice if it were tied into a nice continuous integration system? That’s Codeship. They run your code. If all your tests pass, they deploy your code automatically. For fuss-free continuous deliver, check them out at Codeship.io, continuous delivery made simple.]

[Snap is a hosted CI and continuous delivery services that goes far beyond letting you do continuous deployment. Snap’s first class support for deployment pipelines lets you push any healthy build to multiple environments automatically and on demand. This means with Snap, you can deploy your staging environment today. Verify it works and later deploy the exact same build to production. Snap deploys your application to cloud services like Heroku, Digital Ocean, AWS, and many, many more. You can also use Snap to push your gems to RubyGems. Best of all, setting up your build is simple and intuitive. Try Snap free for 30 days. Sign up at SnapCI.com/RubyRogues.]

CHUCK:  Hey everybody and welcome to episode 165 of the Ruby Rogues Podcast. This week on our panel, we have David Brady.

DAVID:  Six Rubyists walk into a bar and we apologize for the background noise.

CHUCK:  Avdi Grimm.

AVDI:  Hello from Pennsylvania.

CHUCK:  James Edward Gray.

JAMES:  Don’t start yet. I’m not done reading Julia’s blog.

CHUCK:  Saron Yitbarek.

SARON:  Hey everybody.

CHUCK:  I’m Charles Max Wood from WordPress hell. And this week we have a special guest and that’s Julia Evans.

JULIA:  Strace, strace.


AVDI:  Ra, ra, ra.

CHUCK:  Do you want to introduce yourself really quick?

JULIA:  Sure. I’m Julia. I get really excited on the internet about programming.

DAVID:  Excellent.

CHUCK:  I don’t know anyone else who does that.

JULIA:  [Chuckles] I live in Montreal. I do data science.

DAVID:  Wait, what?

JAMES:  Yeah. We’re going to need a definition on that.

DAVID:  Do you mean, do you formulate hypotheses and conduct experiments?

JULIA:  I try, yeah.

DAVID:  Cool.

JULIA:  [Chuckles]

DAVID:  Is there such a thing as data homeopathy?


JULIA:  Some people would claim that that’s the same as data science.


DAVID:  You take 32 gigs of RAM and you set it all to zeroes and you turn one bit on and it’s super effective.


AVDI:  Well, the weird thing is that bits on hard drives actually do have memory.

JAMES:  Oh. [Laughs]

DAVID:  Nice. You know, I tried putting the magnets on my hard drive and it does not make them run better.


JAMES:  Well, how many people have we not made mad in this opening, just to be [clear]?


DAVID:  So, data science. Julia, data science, can we get a definition? What is data science?

JULIA:  What is data science? [Chuckles] This is something I try very hard to avoid.

JAMES:  [Chuckles] Excellent. We went straight for the best question.

JULIA:  [Laughs] My actual job is some combination of [inaudible] thing to do. I’m finding out why the databases don’t work.

JAMES:  Nice.

JULIA:  And trying to remember how to do statistics.

JAMES:  It turns out Julia, that’s actually all of our jobs, no matter which part of programming you’re in.

DAVID:  Yup. I can refer to any of the stuff that I work on as science, because the program I’m working on is a sample size of one.

JAMES:  Hmm.

DAVID:  Wow, that joke was funnier in my head.


SARON:  It was a good attempt.

DAVID:  Yup. Mandy, can you leave that in the show just so people can hear how dumb I am when you don’t edit me?


DAVID:  Thank you.

JAMES:  Alright, so Julia, we’ve been reading your blog.

JULIA:  Yeah.

DAVID:  Yes.

JAMES:  Awesome.

AVDI:  The reason that I wanted to get you on this show is you have some of the most mind-blowing blog posts about systems level programming that I’ve seen in a long time.

JULIA:  Thank you.

AVDI:  You tried to write a kernel in Rust, didn’t you?

JULIA:  I did. It was an accident, in a sense.


JULIA:  I didn’t actually set out…

JAMES:  How do you accidentally write a kernel?

CHUCK:  Wow.

JULIA:  I didn’t set out trying to write a kernel in Rust. I was just going to write a kernel in C or something. And it was Thursday night and I was at Hacker School. And Lindsey Kuper who has worked on Rust came in and I was like, “Hey Lindsey. You do Rust? I don’t know any Rust and I had no reason to use it, because I don’t do systems programming. Also, I want to write an operating system today.”


JULIA:  And she was like, “Have you considered that operating systems may be systems programming, Julia?”


JULIA:  And so, I did it in Rust.

JAMES:  Wow.

AVDI:  Awesome.

JAMES:  This makes me worry that how I plan out my day is entirely not hardcore enough.


AVDI:  Well, we finally out-hardcored James Edward Gray.

JAMES:  That’s right.

AVDI:  Okay, so I don’t even know. I’m not sure where to start with that. But I don’t know. Can we start with talking a little bit about Rust and what that is?

JULIA:  Sure. So, Rust is a programming language from Mozilla, a research programming language right now, which means that it’s fairly heavy development. And its goal is to make things like memory access harder to screw up. [Chuckles] So, if you’re writing C++ you can shoot yourself in the foot three billion different ways, because you’re managing your own memory. But you also have a lot of power that you don’t have otherwise, like in Python. So, Rust tries to give you the same amount of power that you would have in C++ while making it harder to screw things up. And it does that through the type system.

AVDI:  Right, it’s got a lot of different types of safe [inaudible] and stuff like that, right?

JULIA:  Yeah, exactly. So, it gives you compile time guarantees about the memory safety of your program.

JAMES:  And it doesn’t have nil, right? Didn’t they throw out nil?

JULIA:  So, you can always write unsafe code in Rust. So, you can have null pointers and everything terrible. But you need to explicitly wrap it in an unsafe block.

JAMES:  Gotcha.

AVDI:  In other words, you have to say, “I do solemnly swear that I am up to no good.”

JULIA:  Exactly. And then you try to do that as little as possible. Most of the code in my kernel was unsafe.

JAMES:  I have to say, that’s one of my favorite parts, is when you start talking about what you’re doing, you’re like, “Okay. In order to write a kernel, you basically can’t rely on anything. So therefore, you put in this special pragma, no standard stuff.” And you’re like, “And most of Rust language goes away at this point. You had a list of everything you can’t use. It was scary.

JULIA:  [Laughs] Yeah. That was super terrifying. But it really helps you understand how an operating system works when you realize that you’ve lost your entire world.

JAMES:  Right. And now you just have to build it back, one piece at a time, right?

JULIA:  Yeah. And you can build it wrong. I built a malloc that always returns the same pointer.


DAVID:  That’s efficient.


DAVID:  [Inaudible] that.

JULIA:  A equals two. B equals three. And then now A is three, because there’s only one place memory ever gets allocated.


JAMES:  That’s awesome.

JULIA:  [Chuckles] So it’s…

DAVID:  The entire heap sits in one register. It’s awesome.

JULIA:  Yeah. It’s the best.

JAMES:  I guess I’ll go to the obvious question. Why do this? [Chuckles]

JULIA:  Well, so I got to Hacker School. And the point of Hacker School is to become a better programmer, right? You have three months. And they’re like, “Do whatever you want and try to come out of it a better programmer than you came in.” And I had all these things I wanted to work on. And the kind of metric I decided on is I wanted to work on the scariest things.

DAVID:  Nice.

JULIA:  And building an operating system is pretty scary, because how do you even do that? It’s impossible. You’re never going to finish.

JAMES:  Right.

JULIA:  So, that was why I did it. And then it ended up working out. I learned a billion things.

CHUCK:  So, besides this being an interesting exercise, is anyone actually using your operating system for anything?

JULIA:  Oh, no. it doesn’t compile. Nobody can compile it but me.

DAVID:  [Chuckles]

JULIA:  And the reason for that is that Rust changes every day, or it used to. It’s more stable now. But there would be changes to the syntax of the language every week. So, I would try to use some documentation. Ad it would be wrong, because the language would have changed in the meantime.

DAVID:  Wow.

JAMES:  I have to say, I’ve tried to look at Rust a couple of times. And each time I have, I’ll go grab some tutorial and then on line one, that doesn’t work anymore. It’s rough.

JULIA:  Yeah. And Steve Klabnik is working on that right now. He has a contract to improve the Rust documentation, which I’m really excited about. But yeah, so I would just go into the IRC channel, complain for hours. And they were super nice.


JULIA:  I would be like, “I don’t understand strings.” I still don’t understand strings. I actually still don’t understand strings, sorry. [Chuckles] Because Rust strings are really weird. They’re all Unicode strings, which is a bit strange to think about. And also, all of Rust types are terrifying.

JAMES:  [Chuckles]

JULIA:  But the people on the IRC channel are delightful and will actually answer your questions.

DAVID:  So basically, it’s that you’re all uniting against a common enemy, which is the syntax?

JULIA:  Yeah. [Chuckles] And the syntax has changed. It’s almost unrecognizable I think, since when I did Rust six months ago. So, my Rust operating system compiles with probably the nightly Rust from December 18th or something.


DAVID:  So, in build instructions, step one: This is the SHA hash of the version of Rust that you need. [Chuckles]

JULIA:  Basically.

DAVID:  You need it from 6pm to 7:43pm on December 18th. That’s awesome.

JULIA:  Yeah. And I actually still have that version of Rust on my computer so I can do demos.


JAMES:  Nice.

AVDI:  Yeah, you’re locked in now.

JULIA:  [Chuckles]

JAMES:  Right?

DAVID:  This is the best works on my machine story ever.

AVDI:  [Laughs]

JULIA:  But I don’t think anyone else has ever compiled it. And I don’t expect anyone to. And the code is also not interesting, because if you read my blog, that’s much more interesting.

AVDI:  It really is very interesting. I recommend these articles on writing the kernel to anyone.

JULIA:  And it was super frustrating. And also, not a lot of people at Hacker School had a lot of operating system development experience. So, I’d have these terrible bugs. Like, I had a string and it would disappear. It would just be replaced with zeroes and I didn’t know why. And I didn’t figure this out for a week. And I just had no idea what to do or where to start. And so, I read this 20-part article about linkers in an attempt to figure it out. I just spent an entire day reading about linkers in the hopes that I have a linker problem. And then I [inaudible] linkers.

AVDI:  What did you find out?

JULIA:  So, what did I find out about linkers? That’s a really good question. It didn’t solve my problem. That was the main thing that I found out.


JULIA:  It’s really upsetting, because I thought that I have all this new knowledge and it was totally going to help me. But one concern is that there are different ways to, when you call a function… no, I can’t explain this.


SARON:  So, was it just three months of being really frustrated?

JULIA:  Well, so I worked on the operating system for three weeks actually.

SARON:  Okay.

JULIA:  I did other things for the other nine weeks. I tried to write a TCP stack in Python.

AVDI:  Wow.

JULIA:  And I more or less succeeded. I communicated with Google. I got google.com with my Python TCP stack.

AVDI:  After the three weeks of operating system, did you feel like whenever you worked in a high-level language, everything was on easy mode now?

JULIA:  Yeah. It makes the debugging easier when you can do things like print.


JAMES: It turns out, that’s really handy.

JULIA:  I wrote all these really terrible printing functions. Though I eventually learned how to use a debugger, which helped a lot. I could actually run it in an emulator and attach a debugger to my emulator, which was a huge advance that I only learned after two weeks.

JAMES:  Wow.

SARON:  So, what was the value of doing these kinds of, in your words, impossible projects in this scenario, like the Hacker School versus doing it on your own?

JULIA:  So, Hacker School is magical. So, one thing is that you have to actually go. Normally, when you’re doing things on your own, you don’t necessarily do anything. Like, you’ll work on a project and then you’ll stop because you’ll go eat popcorn or something. I don’t know.

SARON:  Very true.

JULIA:  But Hacker School, you have to go every day and come in at 10:30 and tell people what you’re going to do. There’s a lot of accountability. And another thing is that you actually get a ton of help from other people. I arrived at Hacker School and I didn’t know. The first thing I did was I wanted to learn about the Linux kernel. And normally, I would Google something and get confused. But what I did instead was I ran a workshop on the Linux kernel and I was like, “Hey everyone. We’re going to learn about the Linux kernel. Actually, I don’t know anything.”

SARON:  [Laughs] That’s awesome.

JAMES:  That’s the best way to start a workshop ever.

JULIA:  “Do you know anything?” [Laughs]

SARON:  And nobody left. Everyone stayed. It was like, “We’re going to do this together?”

JULIA:  Yeah. And people stayed and people actually knew stuff. So, people in the audience were like, “I’ve actually done kernel programming.” And I was like, “Okay. What is a kernel?”

JAMES:  [Chuckles] Let’s start at the beginning.


JULIA:  And then they would explain and I would write it on the board. And then I learned what a kernel was. And then, we talked about things like scheduling and how might a kernel do scheduling, process scheduling, or how to know when to swap processes in and out. And we could just talk about whatever we wanted. And there are people who actually knew things in the room, which helped. But to run the workshop, I didn’t need to know anything.

JAMES:  You just had to be a facilitator, right?

JULIA:  Yeah. So, one really big thing about Hacker School is that there are people who know a ton of stuff who can help you learn whatever you want.

SARON:  I love that. I wish that worked in job interviews. Like, you could just go to a company and say, I don’t know what the CEO does. Let’s just work on it together and I’ll write on the board. And then I’ll learn and then I’ll do it. That’d be great.

JAMES:  Very true. I think job interviews would be better if they were like that.

JULIA:  Mmhmm.

JAMES:  I know that when I went through, I guess you’d call it a fork phase where I was obsessed with the function fork, and learned basically everything about how operating systems fork off processes and stuff, I cannot believe how much that taught me about computers, just figuring that out.

JULIA:  Right.

JAMES:  How do they inherit streams? How do you change the environment so that the underlying child sees something different or redirect this so that when you fork off that process, it thinks it’s writing to this but I’ve actually changed that? Or just all kinds of things like that, or even just now, you can open your shell and you can be like, “What’s the parent process? What’s the parent process?” And you can keep going all the way back up until you find the one that its job is to launch processes. It’s just amazing.

JULIA:  Yeah.

JAMES:  I can’t believe how much I began to understand about computers when I did that.

JULIA:  And then understanding computers makes it easier for you to understand, to do regular programming work. You think that spending a month learning about fork or forever learning about fork might be an academic exercise. But then I found learning stuff like this really useful to my normal work, which was a really big surprise to me.

AVDI:  Can you give any examples of that?

JULIA:  Yeah. So, this is basically my whole thing about strace, is I started looking at strace as a way to just understand how, the interface between kernel space and user space, because that’s what system calls are. And strace lets you see all the system calls a process is calling. So, I was on the train. I was leaving Hacker School and I was on the train from New York to Montreal. And it took 13 hours. So, I got really bored.


JULIA:  And the internet was a bit, because it was on Amtrak and the internet wasn’t the best. So, I started just stracing killall. And I was like, “I wonder if I can find out about how killall works by using strace?” And I totally could. I looked at the system calls that it called. And what it would do is it would at /proc, list everything, and then find the name of each of the processes and if it matched the thing that you tried to kill. So, if you’re killing Firefox, it’d be like, “Is this Firefox? Yes. Let’s kill it. Is this Firefox? No. Let’s not kill it.” And you could see it doing that with strace.

JAMES:  That is amazing.

AVDI:  We should probably define strace real quick. What is strace?

JULIA:  Right. And I guess we should maybe also define a system call.


JULIA:  System calls are, like when you’re going to google.com on your computer, you might type in things on your computer. When you type in the URL, you press keys and those are keyboard interrupts. And then you’re interfacing with the kernel there, right? I think that’s actually not a system call. That was a bad example. But Chrome will need to open files, right? Opening files is a system call.

JAMES:  Right.

JULIA:  And it needs to communicate over the network and there are system calls to communicate over the network. So, everything you do with your computer needs to go through system calls, especially things like I/O.

JAMES:  Right, launching processes.

JULIA:  Yeah, launching processes is a really big one.

AVDI:  All those pesky things that change the state of the world.

JAMES:  Right.

JULIA:  Though not things like CPU instructions, which do change the state of the world. Things like changing registers go through system calls. So, strace lets you see all the system calls your program is calling. So, if you want to know what files it’s opening and if it’s spying on you, you can spy on it with strace and see everything that it’s doing without having to know anything about the process. And it can be a close source program written in, I don’t know, Prolog or something.

AVDI:  Which is actually kind of amazing.

JULIA:  Right. And it’s just there on your computer if you’re using Linux. And I found myself using strace all the time to debug things. One my first day, at the job I have now, we were trying to set something up in Ruby. And I really don’t know Ruby. So, we were trying to setup my computer and there’s this Ruby script. And we were looking at the Ruby script to see why it was running the wrong ssh command. And then we were like, “We could strace it and see what ssh command it’s running without knowing what it does and without reading Ruby code,” which made me happy because I was still like, “Ruby? How does it even?”


JAMES:  That’s really awesome though. So, what you did was you straced for the, it’s like exec vs or something.

JULIA:  Yeah.

JAMES:  And that shows you anytime a process spawns another process.

JULIA:  Yeah.

JAMES:  So, you could see, oh it’s trying to spawn this process. And as part of that, it gave you the arguments to the command, which is what it was trying to spawn.

JULIA:  Exactly.

DAVID:  The thing I love about that blog post is you made the comment about, “Well, let’s use strace, because the only tool you have is a hammer.” But you also made the comment of, “I’ve got a program. It’s making an ssh call. It’s making the wrong ssh call, or it’s doing something I don’t expect. I’ve got two options. One is I can read the Ruby source code or I can just look at it in strace.” And so many people, the only hammer they have is reading the Ruby source code. And so, your argument of when your only tool is the hammer, when your only tool is two hammers, pick the better one of the two. And that’s what you do there, which is so great. It’s like, “Who cares what the Ruby code, who cares that the program’s even in Ruby. We know it’s an ssh error.” That’s brilliant.

JULIA:  I really like it. And I started trying to think more explicitly about the right approach to debugging now, because sometimes the right approach really is to read the code and to understand the logic.

DAVID:  Yeah.

JAMES:  Right.

JULIA:  And sometimes, the right approach is, assume that the person writing the code was malevolent.


JULIA:  And just go [inaudible].

JAMES:  I pretty much start from there, especially if it was me writing the code.

JULIA:  A past version of yourself might as well be an adversary.

JAMES:  Right, right.

SARON:  Very true. So, how do you know what approach to take?

JULIA:  That’s a good question. I don’t think about this in as principled of a way as I would like. I think I like the observing my program approach better because in some sense, you have to think less. It’s less cognitive load, because you’re like, “What’s it going to do? Okay, that’s what it did.” And you don’t need to keep all the state in your head about the program.

SARON:  Yup.

JULIA:  So overall, I like it better. But it doesn’t always help you. There are a lot of things that strace can’t tell you.

JAMES:  Sometimes, it’s surprising how often it does. I know at one point I was writing a really complex networking code and I was trying to figure it out. And I can’t remember what I used. I probably threw Wireshark in there or something to watch the packets go back and forth.

JULIA:  Yeah.

JAMES:  And there were no packets. None. And I was like, “Oh, I guess I’m not actually talking over a network like I thought I was.

JULIA:  Mmhmm.

JAMES:  It’s funny how that can teach you something.

JULIA:  Yeah. It’s this very incisive tool. It can’t lie to you in the way that your brain can when you’re trying to reason about your code. If it says it’s opening a file, it’s definitely opening that file.

JAMES:  Right. When you go into debugging mode and you start putting in print statements somewhere because you’re trying to prove, “This is what I think happens here,” but when you look at those system calls, that’s what really happened there.

JULIA:  Yeah, yeah. So, that’s my favorite. And I love that you can look at network, things going over the network. I was trying to understand Hadoop, the HDFS, and how the HDFS distributed file system works. And then I started stracing the client and I could see the information going across the wire.

JAMES:  I loved your examples on that. There’s actually chunks of files you saw like, “Oh look, there’s that chunk of that file that it pulled up.”

JULIA:  Yeah. And I think some people are maybe, think more abstractly about programs. I used to think more abstractly about programs, but I’ve become very concrete. And I’m like, “I want to see the data with my eyes.”

JAMES: That’s an interesting point. What makes you want to understand how the Hadoop Distributed File System works?

JULIA:  Mostly because it’s part of my job. I work with HDFS every day. And there are things like, I use this thing called Impala which is a database on top of HDFS. And it crashes sometimes or hangs. And so…

JAMES:  Did you rely on [inaudible]?

JULIA:  I was hoping that if I understood HDFS better, I might be able to work towards understanding Impala better and debugging why it crashes. It helped. But it’s hard. I guess it’s hard to understand big systems that you start out not knowing anything about.

DAVID:  Yeah.

JULIA:  Because there are a lot of moving parts. And then it’s like…

AVDI:  Well, and you don’t know which part, you don’t understand which part you don’t understand.

DAVID:  Yeah.

JULIA:  Exactly. So, now I at least understand the basics of how HDFS works, which is good.

AVDI:  I think that’s a great point about thinking more concretely because I have a really bad habit of wanting, I want my programs to live in a perfect platonic, ideal world. And I want to be able to think through the problems with them. And I think I waste a lot of time sometimes trying to think through problems that way, instead of just saying, “You know what? Let’s get our hands dirty. Let’s get concrete and rule some stuff out real quick.”

JULIA:  Mmhmm, and it can help you. You have this huge branching decision tree process and it can help you chop things off quickly.

AVDI:  Yeah. You can remove huge areas of problem search space.

JULIA:  Yeah, which is really amazing. Another thing which is fun is /proc and procfs because you can find which things are open, like which files a process has open right now, and then follow them around and find out where they’re writing their log files. Even if they’re not writing them right now, you can see what they have open. And then if it’s a pipe, you can find out what other thing has that pipe open and then follow the pipe.

AVDI:  That’s an interesting question. Are there any other utilities that you find really useful for this stuff, tracking this stuff down?

JULIA:  The main other thing I’ve used is /proc. I want to use perf. There’s this tool called perf which will tell you things like how many CPU instructions your program has executed.

AVDI:  Oh, that’s really cool.

JULIA:  And it’s really magical. And it’ll be like, “You have this many L1 cache misses.”

DAVID:  Wow.

JULIA:  Which is amazing, and I’ve played with it a bit. I wrote this really tiny C program and I tried to optimize it a lot, and during Hacker School reunion week.


JAMES:  We’re all back together. Let’s go back to the crazy?

JULIA:  Yeah.


JULIA:  It was the plan. And it was like a typical Hacker School thing where I was like, “I’m going to write a database,” and then I totally didn’t write a database. I just wrote a 100-line C program and analyzed it for days and tried to understand how to make it faster.

JAMES:  I want a crack at Avdi’s question there, other tools that are handy. And the reason I want one is because, and I’m pretty sure David and I had exactly the same experience here, we were discussing before the call. But I read Julia’s stracer article and was like, “That’s amazing. I have to do that right now.”

DAVID:  [Chuckles] Uhuh.

JAMES:  And then I went to strace. Don’t have it on my computer. Can I get it from Homebrew? No. Why? Because strace is a Linux thing and I’m on FreeBSD which is what OS X is derived from. So, I was like, “Ah, what do you do?” And it turns out there is an answer, which is good news. OS X has dtrace instead of strace and it works slightly differently. I would say maybe in some ways it’s more powerful, but it’s also complicated, which was kind of a downside. But it turns out there’s some tools that sit on top of it that remove the complication and take it back to the ease of use that Julia shows off in her blog post. So, I’ll put a link in the show notes to an article that explains those really well.

Another one for spying on open files and network traffic and pipes like Julia’s just been talking about that I’ve used for many years is lsof, L-S-O-F. It’s List of Open Files or something like that. And that one is great. You can just pick a process and it will show you everything it’s talking to, because on Unix basically everything is a file. Then there you go.

JULIA:  I also want to learn how to use tcpdump one day. But I find it intimidating and I don’t know how to use it yet.

DAVID:  So, when’s the next Hacker School reunion?

JAMES:  [Laughs]

JULIA:  In a year.


JULIA:  What will I do?

DAVID:  Just walking with a soldering iron and just see how many people scream. [Laughs]

JAMES:  Circuit board.

JULIA:  People do hardware stuff at Hacker School sometimes. Someone hacked the door recently because you had to buzz people in. So, they made a Twilio bot so you could text message it to have it let you in.

DAVID:  Oh, cool. Don’t tell the IT people that you did that.

CHUCK:  [Laughs]

JULIA:  Oh, they know. [Laughs] It’s their system. It’s secure.

DAVID:  For anybody who has the phone number though, right? [Laughs]

JULIA:  No, your phone number needs to be registered.

DAVID:  Oh, okay.

JULIA:  It only accepts opens from people who are registered.

DAVID:  Okay.

JULIA:  Yeah. It’s actually kind of secure. [Chuckles] But yeah, it was just like a regular phone buzz thing that you buzz someone in with on a phone.

DAVID:  Yeah.

JULIA:  So, now there’s an Arduino attached to it.

DAVID:  Nice.

JULIA:  And the person who made it, Robert Lord, has this amazing blog post about it. And he was just like, “I opened it and I didn’t really know what I was doing. But you know.”


DAVID:  The beautiful thing is when you boil it all down, he spent a couple of weeks figuring out how to build a computer and program it to push a button.

JULIA:  [Laughs]

DAVID:  And yet, if you reduce it to that ridiculous level, you lose the fact of how much triumph you get out of pushing a button. [Laughs]

JULIA:  Yeah. And writing programs to interact with a real world thing is just so satisfying in a way that…

DAVID:  Yeah.

JULIA:  Sending stuff over the network isn’t always.

DAVID:  Yeah.

JAMES:  It’s not without its caveats though. I have to tell this funny story. When I was in Denver recently, I dropped by Boulder rb because I was there on the night they held it. And I hung out with them and they had this system much like you’re describing where the door is locked and they could buzz people in using an app they wrote on the iPad. And they had a camera on it that pointed so you could and see who was there. And I guess the camera froze at one point. So, there were tons of people locked out there trying to get in. And we’re like, “No, there’s nothing.”


JAMES:  Oops.

DAVID:  Oh, it’s beautiful. You mentioned a while back earlier in the call that you learn some of these things and you’re like, “I’ll never ever need this bizarre weird skill again.” And then all of a sudden, the next thing you know, you need that skill. I could cite dozens of examples. But are there other things that you can share that you’re like, “I’ll never need this but it’s interesting to me,” and then you turn around and you find out you’re needing it again?

JULIA:  This is a really trivial example, but when I started using Linux when I was 15, I did not anticipate that it would be a huge part of my job. [Chuckles] Ten years later.

DAVID:  Yeah.

JULIA:  I was just like, this seems fun.

DAVID:  We were talking earlier about systems, like not knowing where the error is coming from. And one of my picks on the call, or on the show a few months ago was for an ErgoDox keyboard. And it’s an open source keyboard. And one of the keys didn’t work. And I tore into it. I started trying to decompile the kernel on it. And I finally realized from years and years ago of programming video games, when you would let up off the button on the controller, sometimes the mechanical switch would bounce.

And so, instead of getting voltage on, voltage off, you’d get voltage on, off, off, on, off, on, off, on, off, off, as it would ring. And I spent probably an hour trying to figure out the software and finally I just flipped the keyboard over and there was a diode missing that was to keep it from bouncing. And so, just completely bizarre, I would never predict that I would need to know about switch diodes. And I went out to my garage and I still had a box of diodes from when I was doing electronics stuff.

And anyway, I said I wasn’t going to tell one of my stories. I lied. The point of this though is that I have no picks for the end of the show today. I just want everyone to breathe a huge collective sigh of relief anticipating this. But if I was going to pick something, it would be a book I’m reading right now about how to be creative all the time. And the answer is to be learning stuff all the time, because then you walk into a weird situation and you go, “Wait. I know this really bizarre, obscure thing that totally applies.”

JAMES:  I was talking to a buddy last night about Inform, the text adventure system. And we were talking about that because he’s going to take a workshop on it at Strange Loop. I thought, “Wow. That’s unusual.” And then I was like, you know, Inform I think, in playing around with it and TADS, which is similar, I think maybe where I gained an understanding for what DSLs are. Like how you just define these things and then the system takes over and makes things work so you can move room to room or pick up objects or whatever. And it’s funny how nowadays I look back and I think, “Oh yeah, that’s like Inform.”

JULIA:  That’s amazing. My friend, Lea Albaugh from my Hacker School batch is actually teaching that workshop at Strange Loop.

JAMES:  Oh, wow.

JULIA:  I think one of my favorite things about Hacker School was people like her who, she doesn’t identify as a programmer. She’s an artist and a seamstress and a magician. Not actually a magician, but I think of her as a magician. And she was like, I can do better art if know more about programming. So, I’m going to go learn more about programming.

DAVID:  Cool.

JULIA:  So, I can make amazing interactive cocktail dresses that [chuckles]

JAMES:  That’s amazing.

AVDI:  Sort of in that vein, do you ever find that any non-programming disciplines spark your programming creativity?

JAMES:  Anger management?


JAMES:  Oh, I’m sorry. You didn’t ask me that.

JULIA:  The interaction between writing and programming has been interesting to me, because I find that I learn programming better by writing about it, which surprised me. I didn’t know that I wanted to write about programming until I did it by accident, every day for 12 weeks.

AVDI:  So, was that something that you explicitly set out to do, I’m going to write about every day of this?

JULIA:  Yeah. But I didn’t have any clear concept of what it would turn into, or what it would look like. I was like, “I should just write about this so that I remember what I did and so that my friends at home can know.”

JAMES:  That always turns out to be the best blog post though. Whenever I write a blog post that’s like, “I’m doing this so that when I google it six months from now, I‘ll actually hit this.” And then those are the articles I return to again and again, because it’s like I wrote that down to get it out of my head so that I don’t have to worry about it forever and ever. And now I just have a bookmark in my head that’s like, “Oh yeah, I’ve got a blog post about that.” And I can go check that out when I need it.

JULIA:  Yeah. And it helps you not worry about the audience for your blog post, if the audience is literally just you.

JAMES:  Right, yeah. If other people are getting stuff out of this, great. But who cares? I am.

JULIA:  Yeah. And then one of my friends would be like, “Nice blog post, Julia,” and I was like, okay, that’s fine. It is acceptable. This is on the internet. I won’t die. I have a problem where I don’t do enough things that aren’t programming right now.


JULIA:  I go biking but that’s not related.

JAMES:  It gives you time to think about programming.

JULIA:  Okay.


JAMES:  David Heinemeier Hansson recently drew connections between programming and writing as well, saying that he feels like in writing, we’re trying to be clear for our audience or whatever and that he feels that’s how we should treat our programming as well, that when we’re coding, we’re trying to be clear for our audience.

JULIA:  So, this is something that I find interesting. Because when you writer a program, you start out trying to write it in a very clear way. And then it acquires bugs. And then you fix the bugs and it gets [inaudible]…

JAMES:  Wait, this never happens.

JULIA:  Over time. And it’s not clear to me if there’s a way to fix that.

DAVID:  Yes, [inaudible] to program.

JULIA:  Because at first you think the logic that you’re trying to communicate is clear and elegant. But then it turns out that it’s complicated and terrible. And you’re like, “I want to do this except in this case and also this other case,” and then there’s this other edge case that I didn’t think about.

JAMES:  There’s this terrific line in, I think it’s ‘Smalltalk Best Practice Patterns’ where it says something about programming is having this idea of what you want to do and then starting to do it. That changes your idea of what you want to do, because of what’s possible and how things work. Like you say, bugs crop up and edge cases you have to handle and stuff. And then basically rinse, repeat. So, now you have this new idea of what you want to do and you try to do that. And that changed the idea about what you want to do. But yeah, I definitely hear what you’re saying.

AVDI:  No planet survives first contact with the enemy.

JULIA:  [Chuckles] Yeah. And I guess like right now, when I write programs for other people to use, if they’re small enough then I’ll just write something and be like, “Okay. What if I wrote this? Would this be useful?” And try not to make it nice, but try to make it something that I can change.

JAMES:  A lot of times, especially when I’m out of my depth, I have to write it a few times before I even know how I can reasonably structure it. So, I’ll get halfway through it and be like, “Oh yeah. That’s why you can’t think about this problem like this.”


DAVID:  Because you’re wrong. I think it was Alan Perlis that said you cannot proceed from the informal to the formal via formal means.

JAMES:  [Chuckles] I like that.

DAVID:  And so, sometimes there are things that, we don’t understand them. And there’s no formal way to solve this problem. All you can do, it’s [chuckles] okay, I have to tell a story from my childhood. I had a half-sister who lived with us for a year. And we explained to her that one of the ways that you tell if spaghetti is done is the wall test. You flick it against the wall. You pick out a noodle, you flick it against the wall, and if it sticks, the spaghetti is done. So, she goes, “Oh, okay.” And she got spaghetti boiling on the stove. Ten minutes later, I swear to god this actually happened, ten minutes later we hear this [whap!] and then, “Oh, crap!” and we walk in and she had flung the entire pot of spaghetti against the wall.


DAVID:  And that’s the metaphor for how I solve some problems. It’s like you walk in and you go, “There’s 10,000 variables here. There’s moving parts. There are systems within subsystems within subsystems. You know what? I’m just going to fling the whole pot of spaghetti against the wall and whatever sticks, we keep. And whatever doesn’t stick isn’t ready to eat yet.”

JAMES:  It turns out you have to be super explicit with those food testing instructions.

DAVID:  Yes, yes.

JAMES:  So, what’s next on your list of crazy things to figure out, Julia? I know you said tcpdump which is intimidating currently. You ought to be about out of intimidating things, by the way.

JULIA:  This is not true. I think the next thing, we’ll see. The next thing I want to stop being intimidating is all of statistics.


JULIA:  Which is pretty different than systems programming I guess. But I did my undergrad in math and then I forgot. I decided that statistics was dumb.

JAMES:  [Laughs]

JULIA:  Andthat I wouldn’t take those classes. And when it turned out that that was what I want to do all the time, that I guess was a mistake. So, I want to learn a lot of things about statistics.

DAVID:  I have to agree with you there, that when I first started looking at statistics…

JAMES:  The math is dumb?

DAVID:  Yeah. Well, it’s like you get into a statistics class and they start talking about Chi squared and R squared regression and you’re like, “This is dumb. Why?” And then you get into the real world and all of a sudden you need to go, you’ve got 10 billion database samples and you want to know if there’s any correlation between this variable and that variable. And oh, turns out statistics can tell you that. And more importantly, if you knew statistics, you would know that of course X and Y are going to be correlated because Y is actually derived from X.

And so, believing that you have found a correlation between X and Y means you’re stupid. And if you had, [chuckles] if you knew your statistics you would know that that’s a false positive. You need to be looking at Z, or you need to analyze the entire dataset, because it turns out that there’s W and Q and M and they’re all interrelated in interesting and fascinating ways. And the math of statistics is dumb. But the questions of statistics are awesome.

JULIA:  Yeah. There’s a lot of math in statistics. But the questions are amazing, yeah. And knowing when you can reasonably make a conclusion based on your data is really tough. And there are a lot of approaches to it. I’ve done more Bayesian statistics in the past. But usually, when you take a stats class you learn frequentist statistics which is P values. So, I’m trying to actually learn basic frequentist statistics now. And I’m like, “Oh wow. This math is, I can do it in two seconds.” But remembering how to think about it the right way is hard, because I’d be like, “So, the probability of the null hypothesis,” and my friend would be like, “No. You’re not allowed to say that.”


JULIA:  This is a frequentist blog post you’re writing, Julia. There are rules.


DAVID:  Yes. Somebody asked, I made the claim a few weeks ago to a friend that this was a once in a lifetime thing. And he was like, “Can you prove that?” and I said, “Well, disproving the null hypothesis requires that I die. So, this is why I have an anti-science agenda.”

JULIA:  Yeah.

DAVID:  And right now, there’s three statisticians listening to the podcast laughing their butts off. Hi guys.

JAMES:  Julia, you talked about how in part of learning statistics, you figured out the value of asking questions. You want to talk about that a little bit? Because it was a great article.

JULIA:  Oh, do you mean asking questions that sound dumb?

JAMES:  Yes.

JULIA:  Yeah. Oh, this is my favorite thing to do. This is how I learned everything. So, this is actually what I learned at Hacker School, which was not anything about Rust but actually how to ask questions. So, if you ask a question that’s really basic, like for example, what’s a P value? It’s a sort of embarrassing question to ask as someone with a math degree. [Laughs]

JAMES:  That’s awesome.

JULIA:  But you could still ask it, if actually you don’t know because you never took a statistics class.


JULIA:  Or what is the Linux kernel, is an embarrassing question if you’ve been using Linux for the last ten years. What does it do at all?

DAVID:  So, what I’m hearing here is that asking stupid questions is a superpower.

JULIA:  Yeah. Well, asking smart questions is easy, because if you ask a really smart question then people think you’re smart.

DAVID:  Right.

JULIA:  But if it turns out, I think that if you ask questions that are dumb, like what is the Linux kernel, they very quickly lead to questions that are smart. Because you’re like, what’s the Linux kernel? Okay. What about this particular part? What about scheduling? How does scheduling work? And then you’ve gone into some specific detail about the Linux kernel scheduling algorithm, which is definitely not a stupid question because, who knows that? So, I think that’s one thing that I like. And then people start not knowing the answers. You’ll ask a question that seemed dumb and people will be like, “Oh well, it’s obviously this.” And then you’ll ask a follow up question and they’ll be like, “Ah, well I think this. But I’m not sure.” And then you’ll continue asking follow up questions and then you end up in the realm of there’s a research paper on that.

DAVID:  Yeah.

JAMES:  Even the question we dealt with earlier, what is a system call? I learned more about that just maybe six months ago or something. And it actually gets kind of mind-mending how far you go into that one question, because your code runs in user space, and system calls live in kernel space. And those two are not allowed to interact. So, how exactly does that work? And the answer is something like your code throws an abort or something.

DAVID:  [Chuckles]

JAMES:  That the kernel is watching for.

JULIA:  Yeah, [inaudible].

JAMES:  Yeah. And then it kicks in, takes over, and it’s crazy. It’s really interesting to read about.

AVDI:  I always think of it as something like you take an offering and you put the offering in a dumb waiter.


DAVID:  You may.

AVDI:  And you press the button and you don’t know where the dumb waiter goes and you don’t know where it comes back from. But eventually, it comes back. And if you put the right offering in, it comes back with the thing that you wanted.

DAVID:  Oh my gosh. You just described DMA.

JAMES:  That’s amazing.

AVDI:  [Chuckles]

DAVID:  Right? Direct Memory Access?

AVDI:  Yeah.

DAVID:  You shovel all the… and the thing that blew me away writing graphics drivers, we would do this. We’d shovel all these polygons down to the 3D pipeline. And you make a system call and you basically say, “I want the offering plate.” And the kernel changes the voltage somewhere and literally switches control of a set of memory transistors in the RAM to be controlled by user space. And you can now set the bits in those little memory sticks. And then yeah, you say send of the offering plate and it sends an interrupt down to, and by send an interrupt what I mean is you put voltage on a wire. You’re actually moving electrons now. This is no longer an intelligent hyper-fast computing device. This is a physical machine. Well, physical. I say physical. But there’s particle movement.

And the offering plate gets triggered and it flips the voltage on that DMA bugger and the top and says, “User isn’t controlling it anymore. Now kernel owns it.” And the offering plate has been delivered. And it then sends it down the pipe. So cool. The cool part for me is to realize that this amounts to, especially on the older processors like 386 or 480 or even the Pentiums, you could get out a diagram of the pins on the CPU and you could actually figure out which pin was getting 5 volts or 3.3 volts applied to it in order to get the CPUs attention to tell it, “Hey. You need to switch this [inaudible] and go in this direction. So cool.

JAMES:  Does that knowledge ever turn out useful or is it just fun for scaring people at parties? I don’t know.

DAVID:  Can’t it be both?

JULIA:  [Chuckles]

JAMES:  That’s a good point.

JULIA:  One thing that surprised me is that I learned about operating system programming for six weeks. And I wrote all these blog posts. And then I got an email from someone being like, “Would you like to interview for this kernel developer job?”

AVDI:  [Laughs]

JULIA:  I was like, “Is that a joke?”

JAMES:  If you’ve been reading my blog posts, I just learned to turn the screen blue, okay?

JULIA:  [Laughs]

DAVID:  Yeah, you’re not two orders of magnitude above everybody else in that space. [Laughs]

JAMES:  Exactly, right? How many people know how to do that?

JULIA:  So, I ended up interviewing to work on the iOS kernel, which was hilarious and I didn’t think that that would have been a thing. So, if you ask questions that are really dumb, then eventually you actually know things.

JAMES:  Right, that’s exactly right.

DAVID:  So Julia, have you read ‘So Good They Can’t Ignore You’ or are you just the accidental poster child for it?

JULIA:  I don’t know what that is.

DAVID:  It’s a book that I think Katrina or James picked on the show

JAMES:  It’s Katrina.

DAVID:  Katrina picked on the show six or seven months ago. And it basically talks about the concept of career capital. And basically, you invest in yourself. And this builds up the amount of capital that you have in your own career. And you are describing to a tee how to invest in yourself and then have it pay off at a much higher rate and a much higher, a much sooner rate of return, date of return, than you every thought possible. You wrote a couple of, you hacked on some stuff, you wrote a couple of blog posts, and the next thing you know, you’re working on it fulltime.

JULIA:  Yeah. And one thing that was hard is I realized that I didn’t actually want to work on it fulltime. I was like, this is really fun. However, I’m going to go do machine learning now.

DAVID:  Yeah.

JULIA:  But I was like, I could. It was really surprising. If I worked more I could actually get a job doing this, and this could be my life.

JAMES:  That’s actually a good thing, though, that you learned enough to know, “Eh, not that one.”

DAVID: Yeah. Yeah. Get enough of a taste of it to know that it tastes bad.

JULIA:  [Chuckles] Or at least not right now.

DAVID:  Yeah. There are spaces that you get into that you’re like, okay the science is interesting. The programming is cool. The engineering is awesome. And all the people that are doing it are jerks.


DAVID:  I’m just going to keep this as a hobby. I’ll come back in ten years and see if these guys are a little more enlightened.

JULIA:  This is what’s scariest about the Linux kernel to me. There are a lot of really amazing people who work on it. And there’s the GNOME Outreach Program for Women, which is this Linux kernel internship that you can do if you’re a woman. And I think all the people who work on that are really wonderful and supportive and are beginners. We could have beginners and then they can learn things and then they will no longer be beginners. And they’ll mentor you and not yell at you. But I think the community as a whole seems kind of scary.

DAVID:  And it depends on the type of community, right? It used to be a running joke that the Gentoo Linux crowds were very hostile to newcomers, so much so that the joke arose that people would say it’s really easy to get tech support in Gentoo. You just go into their IRC channel and scream, “Gentoo is a piece of crap. It can’t even drive a Brother XL4E Laser printer.” And you’ll immediately get 6 people who will go, “Yes it can. You just,” and then you will get six answers on how to.

JAMES:  That’s amazing.

JULIA:  [Chuckles]

DAVID:  Right? But if you go in there and go, “Hey, I’ve got this Brother printer, how do I?” you’ll get, “Well, why don’t you learn how to program?” And okay, that was ten years ago that that joke was relevant. I don’t know if it still is anymore. But you talked earlier about the Rust community. You went into the IRC channel and you asked what you thought was a stupid question. I don’t understand strings. And then you ask the same stupid question, which is usually career death if you ask the same question twice. But you weren’t asking the same question. You were saying, “Okay. I’ve listened to what you’ve said. I’ve learned a bunch and I still; don’t get it. And these people who were probably coming back and saying, “It’s okay. We changed the way strings work.”


JULIA:  Yeah.

DAVID:  It’s not your fault. The language is changing. And god bless those people for sitting down with a new programmer and saying, “This programming language is a table saw with all the guards and safeties taken off. It wants to take your thumb off. And we’re going to take a new programmer and we’re going to try. Basically, where we want Rust to be in a year or two is usable by everybody. And what better way to do that than to take this person who’s going, “I still don’t get strings,” and make it so that she does get strings. That’s a victory for Rust and for you, and for the people in that. And that’s a very different community than a tired, curmudgeonly, user-hostile or newbie-hostile community. That’s wonderful.

JULIA:  And the Rust community is very explicit about this. The Rust community has a code of conduct that says you’re not allowed to be a jerk.


DAVID:  That’s cool. I’m switching languages.

JULIA:  [Chuckles]

JAMES:  [inaudible] really [inaudible].

DAVID:  Chuck, can we start a Rust Rogues Podcast?


JAMES:  Rust. It still works. Rust Rogues.

CHUCK:  Yeah, you can host it.

DAVID:  Never mind.


JAMES:  When m17n was added to Ruby and then shortly after that they came to me and set, can you replace csv with faster csv? Which meant I had to rewrite the inner guys of faster csv and to be aware of m17n. And I asked, I’m sure everyone on the core team was ready to just ban me by the end of that week, because I can’t believe how many stupid questions I asked. What’s this do? What’s this do? What’s this do? What’s this do? And most of the time, they had pretty good answers.

And that worked out to their benefit because then I was one of the few people in the world that almost understood what m17n did. So, I turned around and wrote it all down on my blog so other people might figure it out someday. So, that helped, one. And then two, I did actually find legitimate edge cases that they hadn’t thought of. It’s like, “Well, what do I do when I’m here?” And they’re like, “Oh. Hang on.” New commit to Ruby. “Try this now.” “Okay, thanks.”

JULIA:  This thing that you mentioned about going to the Gentoo IRC channel and saying that your Brother printer doesn’t work, I think I do this at work sometimes, is I won’t have documentation for something and I want documentation. So, what I’ll do is I’ll write down the documentation to the best of my knowledge and then put it up somewhere in public and be like, “Is this right?” And two people will be like, “No!”


JULIA:  And then go fix it.

JAMES:  That’s awesome. I wrote this. It’s probably terribly wrong. If you would like other people to see the right thing, you could correct it.

DAVID:  This is [inaudible] hits by Google right now.

JULIA:  And then you say, [inaudible] pretty quickly. And then everyone is happy.

DAVID:  That’s a brilliant twist on a creative hack that I will use where I’ll ask people for ideas, creative ideas, and everyone’s like, “Well, I don’t have an idea.” So, I will start throwing out bad ideas, deliberately horrible ideas. And you’d be surprised how people, how quickly people come up with good ideas when they want to get away from one of my bad ideas.

JULIA:  Yeah. [Chuckles] That’s my favorite.

JAMES:  That’s awesome.

JULIA:  [Inaudible] like this, like, “Nooo!”

DAVID:  Yeah, exactly.

JULIA:  Response in people.

JAMES:  Yup. Nothing like doing it wrong to make others want to do it right, right?

DAVID:  Yes.

CHUCK:  [Chuckles]

JULIA:  But this is something you can hack with yourself too. Sometimes when I write something, I’ll be like, “I’ll just write something terrible.” I’ll write a terrible blog post and maybe then I’ll be able to fix it after.

JAMES:  Great point. Or, write the terrible blog post and wait for all the commenters to show up and tell you how to do it better.

JULIA:  Yeah, which is scary. It’s scary to publish things that you think are bad.

JAMES:  Right.

DAVID:  Yeah.

JAMES:  Because commenters, internet.

JULIA:  Yeah. I have a good comment policy, which is that I moderate all my comments.

DAVID:  Yup.

JULIA:  So, nothing bad can happen.

JAMES:  That’s the best policy. Well, Julia, that was awesome. Our stroll through your blog was a week well spent in my case.

DAVID:  Yes, yes.

JULIA:  Thank you. This was really fun.

JAMES:  Absolutely.

CHUCK:  Yup. Alright, should we get to the picks?

JAMES:  Let’s do it.

CHUCK:  Alright. David, do you want to start us with the picks?

DAVID:  So, I said during the show that I didn’t have any picks. And during the show, one came to me really quickly. So, I’m going to do this in 15 seconds. Candice Pacheco is a musician and she was fiddling around with programming her midi controller in fourth. And one day, she accidentally played the source code in to the midi controller. And it came out sounding bizarre and strange and musical and she liked it. And she went on to build an entire album based on some of the sounds and noises that her midi controller made when she fed it source code instead of the actual compiled code. So, the album is called, ‘If Then…Else’. It’s world music mashed up with techno and electronica. And I kind of like it. So, that’s my picks.

JAMES:  Dave, have you ever heard Meat Loaf song ‘45 Seconds Of Ecstasy’?

DAVID:  No, I have not.

JAMES:  I think you need to check that one out.

DAVID:  Okay.

JAMES:  But most importantly, you need to look at the runtime on it.

DAVID:  Okay.

CHUCK:  My source code plays the Mario theme when you play it through midi.

JAMES:  [Chuckles] Mario. There was actually, I don’t think it works anymore, or maybe it’s been fixed, I can’t remember. I know I checked it at one point and it was broken. But there used to be a command, a gem, RubyGems plugin called rubygems-sing. And then you could sing and then you could give it the name of a gem. And it would sing the “source code” by basically, every time there was a new scope started like a class or a method or something, and then every time it hit an end, it had to use different midi sounds. And so, different code had different rhythm. It was interesting.

CHUCK:  Awesome. I see it in RubyGems.org, but the homepage is, what is it, RubyForge. So, it’s not there anymore, because RubyForge is gone. Alright, James what are your picks?

JAMES:  I believe RubyForge is gone when I stop getting spam from them.


CHUCK:  Alright James, what are your picks?

JAMES:  Okay. Here’s a couple. I was involved in a fun discussion on Parley this week about service objects in Rails and moving things out of controllers. And somebody brought this article to my attention, ‘Gourmet Service Objects’. And I really like it. I read about 10 million of these kinds of articles. And I’m sure probably most of us by now are aware of all the problems of adding another layer of objects and stuff. But this one seemed really practical. It didn’t try to do a bunch of things like, now you got to handle flash messages and redirects and all of that, inside these objects and crazy stuff that I see people doing all the time. And it was really straightforward and simple. I don’t think this is the kind of thing you want to do everywhere. But when you do, I think you want to be aiming towards something way closer to this. So, I’ll throw this out there just because I thought it was pretty good.

The other one is this question on Quora that I saw that was, “What are some lifestyle changes that save money?” And this is one of those great questions, because I looked at it and I’m like, “Yeah. I’m pretty good with money. I know all that.” And then I went and read the first answer and was like, “Wow. I never thought of that. Wow, never thought of that. Wow, I never thought of that.” So yeah, lots of really great, ridiculously practical advice on the very first answer to this question. So, I think it’s worth checking out, even if you think you’re good at this kind of thing. Those are my picks.

CHUCK:  Someday you’re going to be so rich you’re going to be paying our listeners to listen.

JAMES:  Yeah. Wait for it.

CHUCK:  [Laughs] Avdi, what are your picks?

AVDI:  Oh, I got a few. Let’s see. I’ve been pushing up some more apps these days. And with apps comes the need to monitor them and make sure that they’re not falling over and dying too often. And I’ve been using Honeybadger to monitor the errors coming out of my apps, not that app errors would ever come out of my apps. But on the off chance that one might, I’ve been using Honeybadger. And I like it a lot. There are a lot of apps in the space for monitoring errors and reporting them. And I can’t say that Honeybadger is the very best because I haven’t tried all of them. But it’s got a nice UI. It’s got nice integrations with other things, feels very snappy, and all in all has been working pretty well. Oh, and it works well with my non-Rails, just pure Sinatra, Rack and Sinatra apps that I tend to put up. So yeah, Honeybadger.

Another thing that I’ve been using is Papertrail. The other day I pushed up an app that I wrote in a caffeine-fueled 12 hour frenzy. And it had no tests and it was very small, did one thing. And part of my piece of mind in pushing it up without any tests was that I made sure it did copious logging and then I made sure the logs’ dumped into Papertrail. And Papertrail, all it does is it gives you a place to dump all your logs. And it makes them very easy and fast to search. And you can set up notifications if certain events happen or if too many events happen in a short period of time or something like that. So, it gave me just enough piece of mind that I could be like, “Okay, if something breaks. I’ll at least see what happened to the logs and I’ll be able to fix it.”

Here’s a gem that I use all the time. And it’s one of those things that are so basic that it would only be seen weird if it didn’t exist. But the fact is, there was a long time when it didn’t exist. And that gem is Tilt. Tilt is this intermediary between programs and all of the many, many languages that exist in the Ruby ecosystem. So, whether it’s Markdown or SASS or ERB or Haml or Slim or et cetera, et cetera, et cetera, Tilt knows how to take a file in one format and then expand it using whatever appropriate templating language gem into a different format. And yeah, one of those things, of course it has to exist, except there was a time when it didn’t and things sucked more than they do now. So, Tilt is cool.

I will also pick deadlines. I know that Douglas Adams famously said that he liked the whooshing sound as they go flying past.

DAVID:  [Chuckles]

AVDI:  And that’s a nice bonus. But I don’t have as many deadlines as I used to. I deliberately structured my life to not have as many deadlines as I used to. And mostly, that’s a good thing. But I have to say every now and then, when I do set myself a deadline, it’s remarkable how much I get done, on projects that I’ve been leaving slide for months and months. It’s surprising how much I can get done when I do set a deadline. So, I will pick occasionally setting that deadline, because it seems to work out well.

I wasn’t going to pick anything else, but here’s one that’s topical for the episode. We’re talking about tools that let you spy on things and let you see what’s really going on. And one that I used just yesterday to do that is man-in-the-middle proxy, mitmproxy. And basically, it sits in between web clients, HTTP clients and HTTP servers. And it tells you everything that’s going on between them. And it lays it out in a nice UI where you can dig into the requests and see what’s really going on in there and figure out why the JavaScript on that page isn’t loading or whatever.

JAMES:  I’m drooling over here. That’s amazing

AVDI:  Yeah, it’s a really neat tool. And I’ve just started to dig into it. It’ll hex dump the actual data that went over the wire and everything. You might look at it and think, “Okay. Well, that’s what Wireshark does.” But it’s a bit more specialized for HTTP than Wireshark is. And so, it gives you nice abilities to do things like replay HTTP requests without actually forcing the app to redo the request. Or, exit the headers for a request and then replay it. So, it’s a little bit higher-level than Wireshark would be and lets you do some more HTTP specific stuff. Okay, I’m finally done.

CHUCK:  Awesome. I’m going to pick a productivity tool that I’ve been trying out for a while. It’s called Redbooth. I might have picked it in the past. It also has a Gmail plugin for Chrome. So, I’ve been using that. And what that does is it allows me to add email or email threads to Redbooth and create a task. And then when I get up in the morning I just go through and look at the pending tasks. I usually start with the urgent ones and then the not urgent ones and move a few into here’s what I’m going to get done today. And it’s really nice. I’m really liking it I guess it used to be called Teambox. So, if you’re familiar with that, then it’s the same thing. But anyway, super-duper handy. And I’m just really digging that.

I also, I hadn’t been feeling well for a few weeks. And it was off and on with different bowel related problems and stomach related problems and things like that. And I actually, I have a brother and a sister that both have issues with eating wheat. And so, I decided to go off of wheat. And so, I’ve been off wheat for about as week and I’ve been feeling better. I’m not completely sure if it’s because I’m off wheat or because since I’m off wheat, I’m not eating other things or not eating out as often. And it’s hard, but at the same time I’ve been feeling a lot better. So, I’m just going to throw that out there and say, “Give it a shot.”

And there’s a book out there that explains some of the science behind it that I picked up after I went off wheat. And it also gives you pointers on how to eat wheat-free. It’s called ‘Wheat Belly’ and I’ll put a link to that in the show notes as well. But anyway, I thought I’d throw that out there. So, if you’re having health problems of one type or another, you may want to just try this. Julia, what are your picks?

JULIA:  A few months ago, I read this article about how to become a senior engineer, which really amazed me because I was like, “Oh, I’m going to read about how to code better and become better at typing on my computer.” And then it was like, “Mentor people and be kind to people and help other people become better, and be ego-less about your code and try to become detached from it.” And it was all of these things that I think about a lot but didn’t think of being part of maturing as a programmer. And that’s this post by John Allspaw from Etsy.

Another thing about strace is that Brendan Gregg has this fantastic site about tools like perf where he explains how to use them and what they can do and gives examples. And he has a tool for generating beautiful flame graphs about your code.

And my last pick is what I talk about all the time, which is Hacker School. And you can always apply to Hacker School if you want to become a better programmer for 12 weeks. Applications are always open. In particular they’re open today. That’s all.

CHUCK:  Yeah, definitely go check that out. It’s HackerSchool.com isn’t it?

JULIA:  It is HackerSchool.com.

CHUCK:  Well, thanks for coming, Julia. Before we wrap up, I just want to remind everybody. We are doing our book club book. We’re going to be talking about ‘Refactoring: Ruby Edition’. So, go pick up the book. Start reading it. There’s great stuff in there. And yeah, we’ll wrap up the show and we’ll catch you all next week.

[A special thanks to Honeybadger.io for sponsoring Ruby Rogues. They do exception monitoring, uptime, and performance metrics and are an active part of the Ruby community.]

[Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.] 

[Bandwidth for this segment is provided by CacheFly, the world’s fastest CDN. Deliver your content fast with CacheFly. Visit CacheFly.com to learn more.]

[Would you like to join a conversation with the Rogues and their guests? Want to support the show? We have a forum that allows you to join the conversation and support the show at the same time. You can sign up at RubyRogues.com/Parley.]