This episode was recorded live at Lone Star Ruby Conference by Opus Video Productions

Transcript

JAMES: Okay. So if you are not familiar with it, the Ruby Rogues is a new podcast. I've been working on with a bunch of other guys, including Avdi, over here to my left. He's another regular. And each week, we get together, and discus some topic relevant to Ruby; sometimes it's a hot topic in our community, sometimes it's something we care about, sometimes it's a random idea we get off from Twitter – you never know. But we tend to discuss it for a while, and give our opinions on it. And it's a panel discussion, so we usually uncover some interesting ideas -- I hope. This is our first ever live show, so we'll do that here -- which is cool. We have guests, which are normal for us. We often have guests on the show for various reasons. We're missing some of our normal panelists obviously, because they can't be at Lonestar. We miss them. And that’s why I'm hosting -- which I don’t usually do. But we're here, and we are going to get started. This time, we are going to talk about science and engineering in our field – Computer Science, basically. And kind of some of our opinions on that, to kind of tie in with going on talk this morning. We'll do this a little bit differently this time. I’d have this talk for a bit, and then we'll open it up to questions a. And we'll answer anything you guys wanna ask us. So, that’s our plan. So I'm James Gray, and I´ll be the host today of Ruby Rogues. I've been in the Ruby community a long time and worked with a lot of projects in the community, and obviously at conferences and stuff like that. So that’s me. And I going to have the other panelists introduce themselves. Avdi, go ahead. AVDI: I am Avdi Grimm, and you are not. JAMES: [Chuckles] AVDI: Anything else you wanna know is at Avdi.org. JAMES: Glenn? GLENN: I'm Glenn Vanderburg. I'm a Tech Lead at Living Social. JAMES: And Obie? OBIE: The one and only Obie Fernandez. I wrote the Rails Way, The Rails 3 Way. And I've been around this community for a while. JAMES: All right, so we got lots of great backgrounds here and we should get a pretty varied opinion. I´ll try to get us started with a couple of questions, but hopefully the discussion will take over from there. So this morning, I said I would like to see a lot more science in the things we do as computer scientists. I would like to see us… A lot of science in the things we do and test… the way we test hypotheses and things like that. But I've seen lots of opinions here, right? Even Glenn a little bit talks about what to do when we feel like we can't do that. And then I've seen other talks here today which says, “Actually, computers is more of a design discipline, it's not so much a Science.” I think we all see difference of opinion like that of what exactly it is that we do. So I think it's clear that I think it's at least large part science, I’d like to hear what others think it is, so we'll just go around the table. Obie? OBIE : Nowadays, what I'm hammering the most of it, it needs to be much more of a science. But maybe not in the way that people… not in terms of like lab coats and things like that; science in terms of the scientific method, which means you basically have a certain… you identify an assumption and based on that assumption, you expect a certain outcome. So what do you do? You have an experiment which should prove that assumption -- to some degree. And we do that to some degree when we test for some implementation. So we assume that a particular piece of code is going to make a certain test pass. That we've kind of reversed it if we write the test first. But anyway, and I'm saying… actually we need to do that for everything. We need to do that for the particular design of the software, we need to do that for features, we need to do that for refactoring of a code base, so refactorings should change the velocity of the team. That should be something that could be lodged in to a process system, in such a way that if the technical debt increases to a level where the velocity drops, that causes your build to “break”. “Holy crap, the build has broken. We have a red bar, we need to fix the build.” How do we fix the build? We refactor the code base to eliminate some technical debt, so that we can actually get back to a reasonable velocity. Now the build is not broken anymore. It may seem like I'm conflating terms here that are not usually put together, but that’s what I'm saying is they need to be put together. We need to get to a point where non-functional requirements are expressed equally in a test suite as something that can be built. JAMES: So Glenn, I thought your talk kind of said the same thing this morning. Like I'm thinking currently of your spork example, and the kind of problems it leads to. Do you wanna add to that? GLENN: Science is a word that is often misunderstood, and people have idealized, or in some cases, overly grandiose or in some cases, not grand enough ideas of the word “science” means. I think that we all ought to do science all the time in our jobs. I think that software development should have more science in the small, not necessarily science in the large. And by that, I mean some people think of science in the large; they think of big projects that try to discover universal truths and laws. There are a lot of people who at various times in the history of our fields, have said that we need to use that kind of approach to figure out how we ought to be building software. And I think that’s kind of a losing proposition because there's so much contextual in projects and what we do. And quite frankly, our field is kind of a bunch of babes in the woods about how software really ought to be built. But science in the small, being empirical about decision making and measuring the results of those decisions and feeding that back in to our process, that kind of science, we ought to be doing more of everyday. JAMES: So Avdi, you've been kind of the spectator in this exchange this morning. What do you think about science in Computer Science? AVDI: Well, I kind of wanna push back on Glenn’s point a little bit and say that I think that we are at a very young stage in software, but there are a few things that we do have some data on, and I've seen some people say things lately that are kind of along the lines of, “We don’t know anything about software,” and we do know a few things. I mean, one of my favorite books – on of the books I push a lot – is Code Complete by Steve McConnell. And one of the things that’s really special about that is unlike most of the other code construction books, it really focuses on, “Here are some things that people have done research on.” “Here’s some things that people have run numbers on very large projects and seen correlation.” And we don’t have that on a lot of things yet. We have that on a few basic things. We have pretty good correlations on the idea that more pass through a particular piece of code, for a particular function is correlated with more bugs. It's a pretty basic thing, but it's pretty well proven at this point. There's been a lot of data collected on that. And so, at kind of that level, I think it's important to recognize that there is some science. There is some prior occurrence, some research that you can do. And so when somebody says, “Well, I see that refactoring and that looks really elegant, but that’s just like your opinion, man.” That it's actually not just… in some cases, it's not just your opinion man. In certain cases, where it reduces the number pass to the code or something like that, we know empirically, that it's likely to have fewer bugs. JAMES: But there can definitely be exceptions to the cases, I think. Like even just how our compiler work in line methods, where it's great to extract things out, but then it turns out when we actually need the raw speed, the easiest way to get it is to just to shove it back to it. So I think there can definitely be exceptions and stuff. I thought Glenn’s point on science in the small was really great. And Obie talks about this lot, that everyone thinks science is in a lab. I really try to get this across in my talk this morning, that the example used was my one year old daughter was doing science every day. She's observing the world, she's making hypothesis, she's trying them out, she’s getting peer review from me -- hopefully -- and that's how she's forming her ideas. And actually, I watched Avdi’s talk today, which is basically a brain explosion of everything you can possibly think about exceptions in Ruby. And when I look at that, what I think is Avdi is doing science. He's made all these observations, he's got a lot of hypothesis about this leads to better code. And he’s passed a point where he's trying  to prove them wrong himself and his ideas are getting really strong. And now, he's putting them out there for peer review with things like his Exceptional Ruby book and stuff like that. What do you guys think about that? Do you think that really is science? OBIE: I think that’s a soft kind of science, but very much so, it's being empirical about the kinds of decision we make day to day in software. And thinking them through, developing hypotheses, basing those on your personal experiences, which is kind of a weak, narrow kind of evidence. But then you reach some conclusions, preliminary, and throw them out there, and see if other people have the same kind of experience to back that up. GLENN: The blog post essentially function in the way that journal and white papers would function. I wonder if there's an opportunity for someone to come along and kind of catalog these in the manner of scientific journal. You know, in a quarterly basis or something. It will be different than whatever is the hottest thing at the moment, it actually be taking the approach of what is the most significant “work” that’s been published in blog format, that is scientific; that shows some empirical research and established conclusion. Maybe that is something for someone to pursue. JAMES: Avdi? AVDI: That’s a neat idea. I think it's something that we need. Like it kind of reminds me of my pick last time which was the Hacker Monthly, which sort of takes a few really good articles out of Hacker News and puts them together in a nicely formatted way. I think it would be good to have a journal along those lines that is more focused on engineering and computer science. OBIE: There are a lot of blog posts that I've read over the years that I keep referring people to, but if they don’t have somebody to point people to them, they are otherwise just kind of lost out there among all the others because they are not the hot new thing. GLENN : Well the main if you've ever done research on white paper and stuff like that, when you discover other white papers through the citations. So maybe even take it a step further and say, “Okay, while we have semantic markup or other maybe lighter weight types of constructs that we could use to establish formal citations from one blog post to another,” or maybe when you actually prepare them for publication in this journal, you would actually list the links to the concepts that were… labor of love for someone probably -- it's not going to be profitable. [Chuckles] AVDI: So I have kind of a question just popped into my mind. I've been thinking a lot about… what I guess is effectively science in the small lately, in just my personal process of building software, I would like to do with one experiment. I would like to discover which things make me faster, which things make me more effective. And not just faster, but also mean a light, more effective code. I've kind of run into a block, as far as how to go about that. Like, what is really a good measure? I can think of thousands little measures, but I've been trying to think, what is one or maybe one, two or three good measures of personal productivity -- personal process -- that I can apply to my work and say, “Yes, this definitely improved me.” So really for both of you, I guess starting with Glenn. GLENN: I was afraid you were going to say that. AVDI: [Chuckles] Okay, jump in there. GLENN: Lean software development has the concept of minimizing work in progress. And I think we measure the times that it takes to go from one stage to another. And if you had a system for keeping in count in an empirical way, how long it takes for you, from the time you come up with an idea, to the time that you specify it in some way, to the time that you implement it and are able to make some use or decide that it's garbage and throw it out. And you have a system that actually capture that passively for you… that information passively. I think what you'd end up with is a repository of data that you could use to determine the score for yourself. So are you improving? Are you getting faster, more effective at minimizing your work in progress at getting through the stages faster. Because obviously, the faster you get to decide whether something is good, that better, right? That’s the one way I would go for it more because it has applicability in the small and the large, applicability to someone personally, and for a whole team, and for a whole company. JAMES: So I thought I’d just jump in here real quick. Zed Shaw just had an idea that I really like about his metrics tracking. He used to be pretty religious about that; built a lot of shell scripts and stuff to track when he was adding code, when he was making changes, check his build, how many errors, how many warnings. And then, he would turn that into graphs so he could see, “When I was doing this particular thing, I’d go this far.” And I think that’s kind of Obie what he was talking about, maybe on a small scale and it's just kind of one person. OBIE: We were roommates when he was first kind of getting into that kind of stuff, so that might be some of what is bumping around in my brain. But yeah, he was very, very retentive about keeping logs of those things, probably to the extent that none of us would be able to do on natural basis, and that’s why I said passively. So if we build it in to our tools, and to our editor, and to our shell… and there's so many out-of-the-box things that you can think of, if you go after that goal of picking up that information. Think of almost everything that you do on your computer leaves an artifact of some sort, whether it's system log entries… JAMES: So kind of expanding on that, Zed did one of the Peepcode screencast… I can't remember what they are called now, the series where Jeffrey sits down with the developer and they just do some quick experiment. Play by Play, maybe that’s what it's called. And he did one with Zed, and Zed goes through his metric thing in there and he's using his version control tool and that one, which in his case is Fossil.  But he was using it as a way of tracking his changes, and getting his metrics and stuff. So it's like what you are talking about. That information was already available, he just put it into use. OBIE: You were talking about sort of getting through feature back log and things. I thought of a related kind of inverse metric that I can definitely need to start tracking, which is [unintelligible] per day. And half joking, half seriously, I’d really should keep track of how many times I wonder off into something that isn't the highest priority right then. And I always have that moment when I realize, “Oh, okay I need to go redirect myself.” AVDI: It's hard dealing with that sometimes. OBIE: Related to something I was talking about my talk, fundamentally, you ask the question about productivity; and in software, any measure of productivity is hampered by the fact that we can't define what a feature is -- in any measurable kind of way. Certainly not so that we can correlate them base on size, anyway. And what that means is inherently, all these metrics are going to be proxy metrics -- for what we really want to know. And that leads us into one of the traps of Gilbs Law, which is this is tremendously useful to do, but you need to be aware that these are measuring second or third order effects related to what we really wish we can be measuring, and treat those numbers with the rarity that they deserve, which is perhaps not very much or be aware that they might lead us towards optimizing things that don’t actually optimize the productivity. GLENN: Well, this is where using a tool like KISSmetrics has been eye opening for me, because maybe we are not meant to be measuring features. Maybe that’s just wrong. It’s barking up the wrong tree. However, you can measure the time that it took to influence a key metric. And a key metric by definition is something that affects your business. The reason you are doing it is for business. So, I think as more people come around to using systems like KISSmetrics, you are going to necessarily see orientation of the way that we define productivity towards those key metrics. And I think that will be the best thing that’s happened to our industry. But one of the reasons I'm so bullish in saying it’s a green field opportunity for people, is that KISSmetrics is only one way to do this, right? And it's not free. [Chuckles] It's cheap, but it's not free. There can be open source solutions that could be created, that basically do very similar kinds of jobs and tie it into an infrastructure that is extensible and has plugins, and gives us all the goodness that we've come to associate with testing tool, except that this higher business level. I don’t really see a lot of innovation there right now. You've seen little stabs at it, you've seen vanity for split testing  and some of these little libraries being created that kind of starts there. But like, what is the next Rails? Next Rails is when someone comes along and pulls us all together into some sort of scheme that holistically gives you that next big leap in productivity. JAMES: A/B testing definitely is gaining popularity, I think as people begin to use it to determine things like that. I've seen more and more people using it. I've seen people posting on Twitter and things I wouldn’t even thing to use A/B testing on. Like Peter Cooper, a few days ago, posted that he had done some A/B testing on whether or not he gave the number of subscribers to Ruby Weekly on the sign up form, something that turns out it's a big difference if it does or doesn’t on how many people actually sign up for Ruby Weekly. It's basically what I think you are saying is that we have to stop trying to track these artificial things like lines of code, or things like that, and start trying to track the metrics that… OBIE: In other kinds of static analysis, metric foo I think it's pretty much worthless as something to track on a regular basis, to use those tools in a focused way to drive a particular refactoring episode, sure, yeah. But to use it as a metric that you are going to track some as some sort of reflection of the team’s productivity or value, no, I just don’t see it. AVDI: So here’s a sort of related question: I'm an independent consultant. How do you track client happiness? OBIE: Survey. JAMES: Yeah, maybe. [Chuckles] GLENN: You just got to do it on a regular basis, right? Like an everyday basis or something, so that it's not… just poll them, yeah. JAMES: That’s an interesting thought. I guess one of the reasons I often think my clients are happy is they send new people to me, you know? Somebody else has some project they are working on they go, “Oh, you should go talk to the guy who did this for me, because he did well and he'll kook you up.” So, I guess sometimes I do that as a measure of happiness. GLENN: I think a lot of these things we are talking about will finally come about when we all have heads up displays and off neural net processing agents and things that just kind of know to keep a tick mark as to the fact that somebody gave you a reference, be able to pull it out. For now, since we are kind of limited in what we can do, we only have these kinds of approximations and what ifs. OBIE: See that sounds like the next Rails. JAMES: Yeah. Let’s get to work on that. [Laughter] I wanna go to one other thing that I thought was really interesting. Kind of axled in correlation between Glenn and I’s talks this morning. I was talking about how in science, we always go leap forward with just whatever our best ideas are, currently. And I use the Theory of Gravity as an example, which Newton got close and we used that for a time, and then Einstein refines it into General Relativity. But Glenn also had a section in his talk about being able to sit down and do an experiment and figure something out that's great. But also when I'm at work, some days I just have to solve this problem right now. And I don’t have time to experiment or whatever. How do I go forward when I don’t have the data that I need or whatever. And I would love to hear everybody’s opinion on that. How do we go forward when we don’t have ideal data? Avdi, you wanna start us off? AVDI: Oh, that’s a good one. Pillars of Authority. [Laughter] I'm half serious. So I got thinking about that particular logical fallacy after listening to James talk, and thinking about how sometimes appeal to authority can be really misplaced. Sometimes there are people out there… I mean, there are people in my life, in my programming career, that they had solved the exact, same problems before. And this has sort of driven home to me recently when I was researching about a post I did about the Law of Demeter. And I was reading the original Demeter paper -- which is a great read -- and it's a lot more sort of nuance than you are thinking of, when you think about of the Law of Demeter, if you are familiar with it. And I was reading through and I was thinking, “This is from the 80s. These were the exact same problems that we're focusing on now. The same sized program, same kind of dynamic object oriented system.” And so, we like to think that we we're breaking new ground all the time. But the truth be told, that even with such a young industry as we have, there's a ton of art out there. There's somebody out there that solved a similar problem. So you know, I think the place that I first turn to is just to try and somebody that has solved a similar problem in the past. And start with their solution, assume that they put some thought into this and then iterate on that. GLENN: In the context of Meritocracy, appeal to authority is not necessarily a bad thing. JAMES: [Chuckles] GLENN: So I have two things to say about this; one because I need to sweep up something that I forgot about in my talk. When you are making decisions with weak evidence or poor data or just too complex a situation to really be amenable to analysis, you do have to reason in the context where you are, but you also need to be doubly skeptical of your reasoning. And so I forgot to put on the reading list in my talk, the book Predictably Irrational by Dan Ariely -- or whatever his name is -- that talks about biases in our mind towards irrational behavior  and where those come from. Second thing I want to say is, if you have experience as a software developer, sometimes your hunches are really good -- and you should trust them. Years ago, I was on a panel at a Java conference and there's a whole bunch of guys, and Jason Hunter was one of them. He's the guy who wrote the O'Reilly Servlets book, which almost every Java programmer had on their desk for a long time. And somebody in the audience went to the microphone and said, “This question is specifically for Jason.” He said, “You learned about servlets and started using them and became an expert on them when they first  came out. And then, these EJB things came out, and all of us got distracted and ran over there – and you didn’t. You just stuck with the servlet and you continued to develop that way, and never went near EJB. How did you know?” [Chuckles] And basically, he was saying, “How did you know not to get sidetracked down this rabbit hole that turned out to be so terrible?” And Jason said, “Well, I have the servlet spec printed out, sitting on my desk. And it was about that thick. And then, the EJB spec came out, and I’d never printed it, but I can tell from looking at the PDF file that it would be about that thick. And I kind of went, ‘Eh, I don’t wanna learn this.’” [Chuckles] And the guy got mad. He's like, “That’s not a real answer.” Hunches about complexity: bad, simplicity: good. You know, sometimes that’s wrong, but if it's all you got, it's all you got. You should trust your hunch. OBIE: I wanna riff off your first answer. Because it's established by nature, you can't be aware of your own biases in decision making. So one of the only ways that I've seen documented that you can get over your biases is to have a process of running a decision making process by other people, who don’t have the same biases -- and then they can help you identify the biases. That's one of the reasons I'm still a big fan of pair programming, because it means you always have someone at your disposal, who is passively questioning your biases and your decision process. I totally agree with you that a lot of problems are solved problems, but I would add to kind of steer us back to the scientific process, you know, as a one disciplined way, like you have a problem, you need to fix right now, replicate the problem in some automated way. So you know, write it's spec or test or something that causes the problem to occur. And then I think what you'd end up with is a repository of data you could use to determine for yourself. So are you improving? Are you getting faster, more effective at minimizing your work in progress at getting through the stages faster. Because obviously, the faster you decide whether something is good or not, the better. That's one way I would work because it has in a  small, it has someone personally and for a whole team and for a whole company. Write it's spec or something that causes the problem to occur. And then fix it in a way that it's provable that you fixed it. And you kind of are writing an experiment that approves it, that you fixed the problem, instead of just patching it or throwing a Band-Aid on it. And maintaining that discipline is actually one of the best ways that I've experienced, personally in terms of running a fairly lose process, so it's not testing everything, kind of throwing a lot of stuff at the log. And every once in a while, you are going to have regression. Go back, put some coverage in, cover the regression and you are good to go. JAMES: So I was going to add what Glenn said about our instincts. And I think it's in Pragmatic Thinking and Leaning -- which is a great Andy Hunt book if you haven’t read it -- about how we think and how our brain works. And he talks about how that’s what separates the experts from everybody else is that they have their instincts -- which is a blessing and a curse. When your instincts are right, they are really right and they let you skip a lot of steps, but sometimes, they are wrong -- and they are really wrong. And they can get you stuck. And so he says that the best thing to do is to listen to your instincts, so when you run into problem, you have that instinct of, “Oh, that’s over here.” But then immediately verify the instinct, like measure  that it really is over there. And if it's over there, you should be able to prove to prove it is over there. GLENN: You know actually, I've told this to a couple of people that I've worked with recently, right along the lines of what you are saying. It’s just so, so basic and obvious that it's almost stupid, right? A lot of times, something goes wrong, you get exception; something goes up, you have an instinct of what it was and you don’t read the exceptions. JAMES: I actually mentioned that in… GLENN: [Chuckles] And you jump in to the code and you go fix it. And it's like, “Wait. Read the exception.” That’s actually, I think one of the things that separates a junior programmer from someone that is not junior is forcing yourself to always read the exceptions. OBIE: You know, I have noticed that so many times. And years ago, I said, “No, I'm going to force myself to always read the exceptions.” But it still bites me when I'm trying to fix something, and I make a change that I think will fix it and it still fails. And I don’t verify that it's still the same exception. JAMES: The same exception, yes. AVDI: We need some sort of output filter that notices [inaudible] and highlights it. [Chuckles] JAMES: It should say, “By the way, you fixed that problem, and now you have another one.” [Chuckles] Something like that. OBIE: Or maybe not. Maybe just introduce a different problem. AVDI: For some reason, it's a little bit on a tangent, but this conversation just reminds me a little bit of the Michael Feather’s book, The Legacy Code book. And I think it was in that book that I read about discipline of when you are trying to understand legacy code and modify it, you start by littering it with assertions. As you read down through it, you start inserting assertions that say something about your assumptions, about what the code is doing at that point. And what you are doing is you are just putting on all these little experiments -- all these little hypotheses throughout the code -- and then you run the code, and you see which ones are approved and which ones are disproved. JAMES: Yeah, I've used the anecdote a bunch of times, about how I tend to look at code when I'm code reading and I think, “Oh, this is so easy. I can rewrite this in ten lines.” And I do it, and I gain a great appreciation for what that code is really doing. [Chuckles] That it's much more complicated than what it looked to me at first glance. I'm sure we can keep talking, but I wanna give the audience an opportunity to ask some questions. We really do have live audience here. Go ahead and make some noise, audience so they know you are here. [Audience cheers] They are here. OBIE: Sounds like they like us. JAMES: Or they are very hungry. [Laughter] OBIE: Or they were just told to make some noise. JAMES: So let’s give them a chance to ask questions, and we'll repeat the question for the purpose of the podcast. That’s a great question. So the question was that we've mentioned several studies or reference, things that we know about that already solve problems, that programmers continue to struggle with. And how is a person to know what resources to study the resources. Avdi, do you have an answer for that? AVDI: I wanna go back to back to what I said earlier, which was read Code Complete because that’s kind of the starting point. And I´ll give you a bunch of good references and some of the older research. You know, an odd source -- maybe an odd source -- that is actually pretty good is Microsoft. Microsoft is a big enough organization, that they have a sub-organization that’s dedicated to basically this sort of sticking needles and vials into various processes, and going like, “Okay, this team is using TDD. And this team is not using TDD. And now, we are going to see how productive they are in reference to each other.” And so they’ve actually issued a number of papers, where they talk about the results of studies like that. JAMES: And the cliff notes of those are Joel Spolsky’s blog or something like that -- which I don’t agree with everything on. AVDI: He's kind of moved out of there… JAMES: But he was in there in one of their super interesting periods, I think. OBIE: Some of the pop science books that have appeared on the best seller list are actually good places to start. Like for instance, Drive by Dan Pink, was really instrumental to my understanding of intrinsic versus extrinsic motivation. And that’s something that comes into play in terms of motivating your team, and the folks that you are working with, and motivating yourself. And I can say the same thing for probably on twenty other books that have appeared on the best seller list. The great thing is that most of those books usually have citations to studies that have been done, so if you got to do a deep dive in to it, then you just go to your local university or someone that has access to journals, and you can actually get the white papers. A lot of times, you can find them online. Sometimes you even get lucky and find a well of material that you wanna build a business on. That's essentially what happened with me with … is that I start researching an for my business, and got into science for it and decided, “Hey, this is actually appealing to do as a startup.” There's a researcher named Larry Williams who fairly early on in the days of agile started trying to fun ways to do real studies of programmer behavior what the cost and of your programming in TDD and things like that and now she's worked with some of the Microsoft guys and they collaborated on and as Obie said, well this work is you can find one paper that is about the topic you are interested in even tangentially and look at the bibliography and go from there. GLENN : There's a researcher named Larry Williams, who fairly early on the days of Agile, started trying to find ways to do real studies of programmer behavior, and what the cost and benefits of pair programming and TDD and things like that were. Now he's working with some of the Microsoft guys and collaborated on papers. As Obie said, where it works is you find one paper that is about the topic you are interested in – even tangentially -- and then look at the bibliography, and go from there. I remember one time in college I had to get I was trying to research something I couldn’t find anything on it. I finally found this one paper and our library didn’t have it. I have to send off to have it copied and sent. So I went to go pick up the photocopy, or pick up the photocopy that had been sent from another library. And it included a note that said, “I left out the last page, because it was just the bibliography.” [Laughter] JAMES: I think I pretty much agree with everybody else’s answers, except that I'm just going to add that I learned at least as much reading non programmer material, as I do reading programmer material. Like Glenn mentioned the book Predictably Irrational. Predictably Irrational is a very famous book in skepticism circles. It's a great book about how our brain doesn’t work like we expect it to, sometimes. GLENN: There's a sequel to it by the way. JAMES: And those books, especially I will be reading through them and they'll mention, “Like the such and such experiments,” and I'm like, “Ooh, that sounds interesting.” And I´ll Google that and then it leads me some place new. And I think I learn at least as much useful information that I end up applying to my programming. OBIE: Yeah, I would say some of those are even more valuable, because like for instance Lori Williams I think that’s her name. Those were super controversial because of the team, she was working with students and all those stuff. But when it comes down to it… GLENN: So that’s a good point too. I mentioned in my talk today about just how hard it is to reliably do control studies in software -- especially when teams are involved. So I would say certainly put more faith in smaller scale studies… not smaller scale studies, but studies about smaller scale things. So the example that Avdi mentioned in particular from Code Complete about the correlation between number of code paths through a piece of code and a number of bugs, that’s really strong. And it's kind of a small, constrained problem, and it's easy to -- as a skeptic -- think about how you would design a study to determine this. And I think that’s probably a really reliable finding. Some of the studies that go back to the 70s about the reduction in bugs just from having code inspections, right? Today, we would implement that in a different way -- usually with peer review or pair programming or something like that. But it's a really reliable finding, that more than one pair of eyes on that piece of code, really drastically reduces the number of bugs. Those things I think are probably really reliable. Once you get up into the kinds of things that like Lori Williams was doing with measuring different teams against each other and things like that, you think of all kinds of different variables that might be in there to mess that up. So be a little less trustful of those things, unless they are just really a huge swing. JAMES: I think we have a microphone now, so we’ll it pass around. [Hi. I was wondering if you can talk about the relationship that people have towards science, versus the relationship that people have towards craftsmanship. And the conversations I hear at blogs and polls over the last year was there's a certain… people are… with the craftsmanship that there's the mythological thinking about it. Maybe that’s my bias towards it, but… so it's not science versus craftsmanship, but maybe science and craftsmanship. Or the foundation of science being craftsmanship. Can you speak about that a little bit?] OBIE: I think craftsmanship has a lot of potential pitfalls, as something that we are going to use as a metaphor. Those pitfalls being that you can have software that's crafted really well in terms of software quality, in terms of extensibility and various nonfunctional requirements that make that code beautiful and elegant, and well-crafted. That don’t really speak towards the usefulness of that code -- for some business purpose. So specially, if you apply craftsmanship in business scenarios versus like open source or more artistically-oriented scenarios. It's really dangerous. The people practicing craftsmanship could end up with a black eye, for having put too much emphasis on doing things in an elegant, well-crafted kind of way, and their project fails because it didn’t actually meet the needs on the business. JAMES: I was going to jump in and say that craftsmanship makes me think of what I'm building – the software that I'm constructing. And to me, the software is just a means to an end. It's the knowledge that I've gained when I construct a piece of software that’s important to me. And some people even talk about… I believe Zed on that Peepcode video I mentioned before, talks about how you'll get down a certain line, realize that you are just fixing the wrong thing. And that though most of us freak out when they see this, he does like a reset at that point -- it was version control --  and completely throws away his work. And he says, “That doesn’t matter. It's what it taught me that matters.” And I feel that way about what we do, because no matter how invested I am in crafting this application I'm working on right now, that won't be what I'm working on in the future. And I still need to improve my mind and my skills on that. So I'm in it for what I can learn, and my best understandings that I have at any given time; as opposed to obviously, I want my software to be good, but my focus is on my gained knowledge more than the thing I'm building. OBIE: The ideals of software craftsmanship really resonate with me. I do understand some of those dangers. And certainly, like nearly all the terms were use to describe our field, the word “craftsmanship”  can be easily misunderstood. If you are calling yourself a craftsman, and you are not focused on building what your customers need, you've got a reset to do. That's inherently what you wanna be thinking about. To me, the importance of software craftsmanship as a movement or whatever you wanna call it, is actually to establish an emphasis that counters one of our biases. -- one of our cognitive biases -- which is that kinds of bad internal quality in software that craftsmanship is concerned with -- this is oversimplified in some degree -- poor factoring, poor designed, poor testing practices, those deficiencies in software that craftsmanship worries about are deficiencies that are not obvious – certainly not from the outside. And have a real cost that’s off in the future somewhere. So things that are not obvious and tangible and are not now, but are in the future, we have a tendency to just not think about. And to me, the way I think of the craftsmanship movement is recognize the value of good design and good testing, and clean factoring of your code. And even though the cost of not doing that are deferred, they are very real; and you need to factor that into your decision making. Now, in some circumstances, it's like, “Okay, that’s a cost we are willing to pay if it comes to that, to get something right this instant.” But I think an emphasis on those things and keeping them in focus is a really good thing. GLENN: Would you agree it's an orthogonal issue to taking an empirical approach to how do you decide what to build and how you build it? OBIE : Yeah. I think they are two different concerns. You can have the biggest scientific approach to deciding what you’re going to build, and how you are going to structure the team that’s going to build, and the assumption in how you document the assumptions and the experiments, and still write really poorly crafted software or not. GLENN: I’d like to use an example from world outside software. And it's a world we're still all familiar with, and that is Apple Industrial design. So for a long time… it hasn’t happened in a while, but for a long time, at Apple keynote events and things, Steve jobs would have have Johnny I've take the stage and talk about their industrial design process and the things they were doing. And he would get up there and talk about materials, and the beautiful simplicity of glass and the characteristics of aluminum. JAMES: [inaudible] GLENN: [Chuckles] Exactly. And you know, I'm sure that 99% of their competitors, and a pretty significant percentage of their customers would go, “Oh, give me a break.” [Chuckles] But that level of craftsmanship has had real positive consequences for them. It turns out that because they started really trying to work with the best materials they could, well it turns out that aluminum has some characteristics that make it really good for machine milling with very minimum waste of materials, and that allowed them to build unibody machines that are lighter and all that kind of stuff. And you've probably heard this, so I don’t need to go on. But, I doubt very seriously they were thinking about that characteristic of aluminum when they started down that path. And this has had a profound effect on their business. And now, lo and behold, the competitor can't match them on price, even though they have the largest margins in the industry. And I think a lot of that was a sort of beneficial side effect for their focus on craftsmanship. Note, at every step of the way, they were clearly concerned with building a business and being profitable, and meeting customer’s needs. And it didn’t stop them from building machines that now they probably look back and on and say, “Wow, those were crappy.” But that was the best thing they knew how to do at the time. And while they were figuring out how to do it better, they built how they knew to build it back then. So craftsmanship has real benefits long term, it shouldn’t be allowed to get in the way of meeting the needs right now. And I don’t think there's any real conflict there. OBIE: There's a caveat to that, which is that a lot of teams can't afford to have a master craftsman or just because of the realities of the labor market don’t have one available, that actually knows what good craftsmanship is. And that’s where a lot of that kind of falls apart until we get to the point where there's more. GLENN: I think that’s right, but I think the point is that there should be something we teach and is a part of how we bring up young programmers to value those things. Not to value them inordinately,  but to see the value of good design and those kinds of things. AVDI: When I think of craftsmanship, I think a lot of people, when look at craftsmanship, especially in the context of software, I think they imagine somebody sort of pouring over a piece of work very carefully and taking little chips at it and very meticulous and taking a long time about it. When I think of craftsmanship, I think of my grandfather, who was many things; he was an engineer, he was a lot of different things, but one of the things he was a master wood worker. And watching him, he made beautiful things, but he wasn’t meticulously, slowly chipping away at them. What he did is he made swift shortcuts. He had  practiced so much, that he the right instincts. Certainly, he measured carefully and then he cut quickly, and then he made the right cut and didn’t have to do it again. And so I think that there's an idea that craftsmanship is always equivalent to going more slowly. And I think that if that is your idea of craftsmanship, you’re selling yourself short, you are selling your ability short. I think craftsmanship to me, where it meets science for me it's about internalizing lessons that you've learned, to the point that they are intuitive. To the point that you could, if you thought about it long enough, realize the chain of deductions you've made intuitively come to the decision that you made, that you made it so fast, that it was an intuitive deduction, and you've decided how to move forward with this design – and you did it in a split second. And that to me is craftsmanship. It's not a slow process; it's a process of you internalizing things until they are fast. JAMES: I think I pretty much agree with Avdi about how that goes. The only other thing I wanted to add is that, in both systems, we are talking about people they are measuring what they are doing, observing effects of that and trying to improve their skill. No matter what your system is for that, you've already won. You are doing the right thing. If you are checking your process, if you are trying to improve it, if you are making it better, that’s what is important, in my opinion. Next question? [So I think you guys are spot on, and something really tickled my mind when you guys were talking earlier which is that we are biased as programmers to think of metrics that we care about, like lines of code and test coverage, and crap like that that our customers don’t find value in. And I think it was Obie that said it that you can't measure features.] GLENN: I said we can't measure features, and Obie reminded me that that's not what we care about measuring. [Exactly. So my real question that I think we should apply some science to is, what should we measure and what is it that adds value that we can really do something about that, we should be focused on instead of the mental masturbation that is all the software development stuff that we focus on. I mean, I'm not trying to denigrate any of that, because I really like test driven development and all that, but it's self-serving. It makes me feel happy.] GLENN: I mean, what's the software that any of us is getting paid to write is meant to facilitate something else in the non-software world or in some other software system. So the answer to your question is kind of necessarily has to do with what that software is intended to do. So for any given function of the software, you look at what it's supposed to do. You see what doing that better would entail on and then you derive a function from it, whether it's some mean that’s changing, or some level or something like that. One of the reasons I really like the tool KISSmetrics is because it gives you a framework for firing events off from your software to a repository, and then gives you a web-based interface that gives you some wizard and forms for deriving metrics. Because unless you've studied the process of creating metrics and stuff like that, you don’t really know where to start. And that’s why I'm saying that more tools like that are necessary. OBIE : Kathy Sierra, and I think I'm going to get this quote wrong but Kath Sierra likes to say, “Don’t worry about making yourself awesome; worry about helping your customers be awesome in what they wanna do.” And that kind of metric, if you can find metrics that measure how awesome your customers can be, using system or whatever, and improve those, and then measure how rapidly you improve those and improve that… JAMES: I agree with what that. The only thing I thought about it is that in the beginning when I don’t know, I cast a wider net and try to look at more things. And then as I'm zeroing in on what is the most important to me, I try to restrict that back down into the things… Sometimes it's easier for to see if I'm in the 10,000 ft. view first, and then work my way in to what really matters. The points that seem to matter the most. OBIE: There's the concept of minimum viable product or MVP, so when you are starting on a new project, you are in a rush to get to your MVP, so you can release. And one of the best ways lately that I've heard MVP described, is the earliest stage of which you can test assumptions, which clarifies things for me quite a bit, because otherwise, if you don’t have that kind of clarification, well what constitutes viable? Viable to what? Viable for them to do something. “Okay, how do I test it to do something?” Then it all kind of becomes clear there. JAMES: I think it's surprising too, how far you can take that. Like I'm working on launching a product of my own at the moment, and I'm really  trying to do that and say, “What is the core chunk of this?” And you can't believe how many things you can throw out when you do that. You know, “Oh, it would be great to have comments here.” “Eh, comments is a side thing that we'll get some day.” Doesn’t have anything to do with what I'm trying to do here. OBIE: Well to riff on that, basically, comments on some typical web kind of product, what are they meant to do? They are probably meant to increase engagements or visits or something like that, right? So you wanna write a test for that. And you have to get the system to the state, right before you add the comments and then start tracking that metric, that you know whether it was worth adding comments or not. Maybe you add comments, and that metric goes down. JAMES: It’s is the internet response effect. I don’t wanna go too far over. We usually try to keep our shows right at an hour, but we'll take one more question. Who’s brave? [Hey guys, so as a programmer, my brain is always pattern matching. So I'm trying to think, “Okay, what is the situation were this has happened before?” And it sounds to me a lot like the reinvention of tailorism and scientific management and that started to be introduced to change the way that management worked. And so, I can't off the top of my head, think a lot about what a lot of the criticisms of what tailorism are, but a common thing I hear about metrics is that you have to make sure that it's not leading you into a local optimization as opposed to a global one.  And so I think that metrics are great and that we should follow them, but they are also not to make sure that we do make those changes when they actually need it, in order to advance the metric even further. It's really easy to get trapped into those kinds of like, “If I'm changing the text of my headlines and it's converting a little better, that’s great.” The way that the page was designed is poor and we could actually advance even more by focusing on changing that as opposed to changing the headlines. So sort of a question/comment is I guess is there an easy way to know when you are making a hyper local optimization as opposed to like measuring something that’s bigger or that will be larger change, I guess.] JAMES: So I just wanted to apply a tiny correction. He said that as a programmer, his brain is always pattern matching. What he meant to say was as a human, his brain is always pattern matching. But other than that, the question is great. GLENN: Let’s just use an example from what we were talking about. Putting comments on webpage. And Obie said, you get the measurement in place, and then you launch that feature. And then you may find that the level of engagement goes down, or maybe goes up at first and then it goes down. Who knows? But it turns out to be a net negative. Well, so then who do you draw conclusions from this? That the kind of narrow conclusion would be, “Oh, comments are clearly a bad idea for purposes of engagement on the site.” But it might be the way you implemented it, or that un-moderated comments are net negative because people are turned off by negativity or something there and go away But if you had moderated comments that raised the level of discourse, they will be a really good thing. There's all kinds of ways that you can go from there. You have to interpret the metrics in terms of a model of what you are trying to achieve. And in that particular case, that model will incorporate who your customers are, and how they think. You know, I've been talking a lot recently about engineering, and a really good book about engineering about Fred Brooks is his recent book called The Design of Design. And Brooks was educated as an engineer, and then found himself in the software world and the computer world. And when he was in the engineering school, in the foreword or preface of that book, he talks about being in the engineering school at MIT, and the dean of the engineering school at that time at MIT was this Nobel prize winning physicist. I can't remember what his name was. And he was very concerned that engineering should be more of a science, and should be grounded in the findings of science to a much greater degree than it was. And he lead from this position as dean of engineering at MIT, lead a big shift in engineering education in his country, towards being more of a scientific discipline. And there has been a backlash against that by engineers, because of exactly those concerns; it leads to a very narrow kind of thinking, and demotes the idea of design. And engineering is fundamentally a design discipline, that employs scientific findings rather than a science itself. And so I think we need to have good models of what we are trying to achieve, and try to think beyond neuro metrics, and maybe take bold experiments sometimes that might be counter indicated by what we've measured, but might help us break out of a narrow view point. OBIE : So that the more tools and infrastructure that we have for taking those experiments, the more they will happen. Because one of the reasons they don’t happen now is because the tools and the frameworks for doing it don’t exist. Quick point to your question, in the kind of grand vision for this experiment or goal oriented development, the designer/developer programmer team or whoever is implementing features, I think actually gains a hell of a lot more freedom to be an artist and a craftsman and to do the implementation the way that they want to, if the business is trained to look at the experiment results. “Okay, we are looking to prove retention by 8% by adding commenting. Go.” And literally, that would be the requirement that’s communicated to the team. And the team would be comprised of experts that would know how to build the commenting system in maybe three different ways; un-moderated, moderated, threaded, not threaded. And they go and they do it. And they find the one that satisfies the metrics that have been provided in the best form. And it actually gets rid of business interference in what should be the domain of the program or designer, and letting them implement stuff in way that they know it needs to be implemented. And why should a product manager, someone that is responsible for meeting financial goals of the company be discussing the proportionality of a particular text field to the another text field? That’s bullshit. We have folks that are experts in doing that. Let them do their job without interference, and that it saves a lot of time spent in meetings, it saves a lot of documentation, a lot of grief and just speeds everything up. JAMES: I think the only thing I really thought to add on those points was that if I'm making little  improvements and things are still getting better, then I'm happy. As that begins to slow down, which inevitably it will, then I guess that leads me to assume I've gone about as far on that wagon as I can go, and it’s time to start looking at it on different angles, and expand my view, my scope, I think is one thing. Another thing is not to be afraid when to admit when you make a mistake. Right, that’s the thing you see over and over again. There was a big gap logo that turned out to be huge fiasco, or what about the new Skype? I mean, can we call that a mistake? It's a mistake, okay. It's absolutely the worst UI design I've ever seen. I feel like at some point we can admit that and say, “You know what, we didn’t do the right thing,” or whatever. GLENN: But it's nice that they took a bold step in both of those cases, but the trouble was that they weren’t ready to acknowledge it as an experiment that they can pull back from. JAMES: Right. Which is like Google’s way, right? Think about it. Google, they mercilessly kill everything. GLENN: They spent a billion dollars on this Wave thing. [Chuckles] JAMES: I know. They put everything in “lab in beta”. And everything is like eternally in beta. And then they just try something for a while, and if it catches on, it's awesome and that’s great and we'll keep that. And if it doesn’t, they just kill it. It doesn’t matter because it was beta anyway. It was nothing. Everything at Google is an experiment, right? Ready to fail. OBIE: Yeah. Has easy flags or has easy is one of my favorite Rails plugins now for a while. It basically lets you add arbitrary … to a class, which means you can very easily add flags or preferences to say, user class; which means it's very to give the user the option to turn on things that you are not sure about. You know how they are going to work instead of rolling them out to everybody, you just sneak a checkbox onto their setting screen, and  you know that some percentage of them are going to click on it and turn it on, and you just see what happens for those that do. I thought I’d throw that in there. [Chuckles] JAMES: That’s a good idea. I guess we'll wrap it up here, and we'll now go around the panelists one more time and thank them. And they may want to make a little announcement. So like Avdi, thank you for joining us again. And where can people see you at the Lonestar tomorrow? AVDI: Wait, what? [Chuckles] JAMES: At Lonestar tomorrow, were you doing anything at Lonestar tomorrow? I thought you said you were working on a talk for Lonestar. AVDI: Oh, yeah. This has been fun. And I think I will be doing a talk tomorrow. And again, more stuff on avdi.org. Happy hacking! JAMES: Glenn? GLENN: Yeah, it's been fun. I'm at vanderbug.org, which isn’t my first name. Couldn’t get glenn.org. So and I´ll be at SCNA later this year. That’s about it. JAMES: Obie, thanks for joining us. OBIE: Thank you for having me. ObieFernandez.com. I'm actually getting into the lean startup circles with Eric Ries and some of the other folks in that community, and mentoring a series of workshop conferences called Lean Startup Machine. So the next two that we have are coming up in September in London, Edinburgh, and in Berlin. So Lean Startup Machine, Google it or check my blog, and just talk about the interception of Agile and Lean, and all that stuff. A lot of potential for innovation there. JAMES: And I've been your host, James Edward Gray II. We'll see you next week! [Applause]