Charles: Hey everybody and welcome to Episode 275 of the Ruby Rogues Podcast. This week on our panel, we have Sam Livingston-Gray.
Sam: We can’t stop here, this is bat country.
Charles: Jessica Kerr.
Jessica: Good morning!
Charles: I’m Charles Max Wood from Devchat.TV. A quick shout out for railsremoteconf.com. We have a special guest this week and that is James Shore.
James: Hi everyone. I’m a programmer, I’ve been focusing on Agile methods since 1999. I got involved with extreme programming, how to got a fund with it and haven’t looked back. These days, I work primarily as a consultant helping the more entrepreneurial companies figure out how to do this stuff well.
Charles: Now, consultant is code for you get paid a lot and you don’t work a lot.
James: I put the con in consultant. Well no, [laughter].
Jessica: Con like negative, right?
James: What it means is that my dad tells a joke, he’s really nice that way. A consultant is somebody who you ask them the time, they take your watch, tell you what time it is, and then keep the watch. I guess I’m not selling myself well here but that’s the best I got.
Jessica: Totally. That is like when you go to a customer and you help them but you also learn a ton from them and then you take that knowledge and do even better at the next place and make even more money because you gathered all the experience from your customers.
James: You should totally do all my marketing.
Sam: That reminds me of something my brother said about one of our grand fathers which was that he’s sort of this person who if you ask him what time it is, he’ll show you how the watch works.
Charles: I have a whole show about freelancing and consulting, so maybe we’ll get you on there so you can show us how the con actually works.
James: It’s good fun. The nice thing about it is I really get to set my own hours, the bad thing about it is I get to set my own hours and I have to do all my marketing and sales and all of that stuff that as a programmer I don’t really enjoy. At its best, it’s a chance to really help people in a lot of different ways and places.
Jessica: Do you think that doing your own marketing and sales helps you empathize with the business as a whole better than you could when you were just a programmer?
James: Yeah, absolutely. As a consultant running a solo shop, I’m a very, very small business. It still gives me a lot of appreciation for skills that when I was just doing programming work, it was pretty easy for me to poo poo. There’s a lot of jokes out there in the programming world about marketing and sales not really being able to do much more than operate the coffee maker and how come there’s no coffee today.
Jessica: As if there would be any purpose to our job at all without marketing and sales.
James: Yeah, absolutely. It’s given me a good understanding of what’s out there but also the limits of what marketing and sales can do as well.
Jessica: Do you think that’s core to Agile, that broader understanding and appreciation of more skill sets?
James: I think it’s core to what Agile is trying to achieve, I don’t think everybody on an Agile team needs to have that. But absolutely. Agile at its best, and Agile is sort of this big umbrella term that doesn’t mean very much anymore because so many people have co-opted it. I think Agile at its best is about bringing all the different perspectives that are needed to chip high quality, successful, valuable software, bringing all those perspectives together and having them work together effectively.
Charles: We’re going to go a whole lot in Agile today. You mentioned that Agile has been co-opted into something else. Where do you think that’s gone and why do you think that’s problematic or not problematic for people who are trying to do Agile?
James: I’ve been involved with this since 1999. It was named Agile in 2001 which gives you a sense of how long I’ve been involved. Back in the early days, it was mostly programmers doing this grassroots effort to try to get past what people were calling a software crisis, all these failed projects that were coming in late and not finishing what they were supposed to or what they had planned to.
That’s a fairly peculiar definition of failure that doesn’t involve anything about profit which was one of the problems. In reaction to this failure, companies had made increasingly bureaucratic binders full of software processes that people had to follow. It was fairly dehumanizing and not a lot of fun to work in. People got a bunch of folks, mostly programmers, some consultants got together and worked out, came up with a lot of different alternatives. Extreme Programming Scrum, Alexa Cobra’s Crystal Methodologies which I’m pretty sure he named Crystal Meth-odologies on purpose.
Charles: He did, totally.
James: Anyway, there’s DSTM. All these methods came out and the whole point was to make software development better and more effective, and that’s where the Agile manifesto came from.
In the beginning, it was very much a grassroots movement. You had advice like hey if people won’t let you sit together, come in on the weekend and bring your screwdrivers and screw the furniture. I tried that once, it did not make them happy but we got stuff done.
Around 2004, people were saying this is a movement that is about bringing people together and we’re not involving project managers enough, we’re not involving executives enough. There was this huge, huge push to bring project management into the field which succeeded. The conferences started having more and more project managers but it went too far in that success pushed out the technical content and the programmers largely left.
If you go to the big Agile conference this year in Atlanta, there is not very much technical content anymore. In fact, this year,
Jessica: I gave one of the very few technical talks.
James: Thank you for that because it’s necessary. In fact, to try to bring the technical content back, they’ve introduced a new conference which is the Agile Alliance Technical Conference which was in Raleigh I believe this year.
The question was is it good that Agile has been co-opted. It’s good that we’ve got all this participation but it’s bad in that the programmers who are really—this is software development. Agile works because we bring the physical realities of software development to our methods. We say things like Conway’s Law matters. Conway’s Law says that the structure of the organization reflects the structure of the code and vice versa. We say things like code quality matters, technical debt matters, and testing matters. Without technical folks, without programmers involved in the process, then that gets left behind and that’s what you see in Agile today. There’s an ignorance of how the technology affects what you do.
Sam: That’s been my experience that even when there are people in the meeting saying, at least in some of the organizations I’ve worked with, there’s still a strong pressure to say we understand that all of that stuff matters and still give us a number and it has to keep going up over time.
James: Up and to the right. One of the most common questions I’m asked is how can I get my product owner to let me pay down technical debt which is the wrong question to be answering, the answer is moo.
Charles: I love it.
James: Don’t ask permission to pay down technical debt, that is part of your work. You should always leave the code better than you found it. Agile does not work if you don’t do that. No process will work if you don’t do that, but that’s been forgotten because there aren’t enough programmers and other people with a technical bed involved in putting these things out. Obviously, I get pretty passionate about this.
Charles: I think it’s funny. The way you’re talking about this reminds me of a video I saw a couple of weeks ago. There’s this couple sitting on the couch, she looks at the guy and she’s like I’ve just got this pain and this pressure right here. The camera angle changes and there’s a nail in the middle of her head. He starts saying, “Maybe if you got the…” She stops and she’s like it’s not about the nail. It seems like a lot of this with Agile, they’re so focused on the practices that they’ve forgotten the technology. In other words, they’re so focused on the nail that they forget about the person.
It’s the same kind of thing, it’s not about the nail. Sure, if you remove the nail it probably helps. That’s not what it’s about, it’s about the person, it’s about the quality of the code we put out, the quality of technology that we can create, the quality of life that we can create for the developers doing the work.
Jessica: It’s not about the nail?
Sam: It looks like she has a nail on her forehead.
Charles: How do we fix this, then? Is it as simple as oh, let’s bring in more technical folks or is there more to it than that?
James: I think there’s a synthesis that most programmers. You go on Reddit or Hacker News and in the conversation about Agile will variably have somebody come up and say, “We did Agile and it was the worst micromanagement waste of time, spending all my time in meetings that I’ve ever experienced.” It would be nice if people who have that attitude about how to make things better to understand what Agile is really meant to be and help fix it.
Ultimately, this is something that although it’s too bad Agile has become so popular it’s been taken over by this project management mentality, it’s also meant that it spread. In many ways, what happens is when people work in this way, they get this sort of rush from having more control and more visibility about what’s going on. And then after a couple of years, their code base dies because they haven’t been paying attention to it.
It’s easy to laugh about it, it’s not so much fun for the people involved.
Jessica: That’s not the fault of Agile because if they were in any other process, their code base would still die after two years when they’re not taking care of it.
James: Exactly, or three, or whatever. What happens in most cases is people say oh, we really like this Agile stuff, now we need to figure out how to do the technical practices. It would’ve been great if they would’ve been doing that from the beginning. It actually brings in the stuff that they need to learn.
This is part of something that Diana Larson and I created called the Agile Fluency Model, actually which I can talk about more if you’re interested.
Sam: Yeah, I found that really interesting and useful when I saw it.
Jessica: There’s pieces of Agile that don’t belong to programmers, that are bigger than software development, that are about people working together like that appreciation of all the skills involved and the learning cycles. In my opinion, the one core practice to Agile is the retro, reflecting on how we’ve worked, deciding on an experiment for the next piece of time and then asking whether that helps and iterating on our process of iteration.
Jessica: That could apply anywhere. The technical practices still need to happen if you’re in software.
James: I absolutely agree with you. I think the most important—I wouldn’t say it’s the most important thing, there’s lots of things that I think are really important in Agile. The focus on people over bureaucracy is one of them. They call it people over process. The emphasis on the way people interact I think is super important as well.
Interestingly, retrospectives weren’t actually in any of the early Agile methods. The idea of continuous learning I think is super important and retrospectives is one of the most effective ways of doing it if you actually follow up on your retrospectives which a lot of people don’t. There’s some things that are very difficult to iterate towards.
Retrospectives and other continuous improvement, Kaizen improvements, tend to get you better at what you already know how to do. There’s some things to do that really high performing Agile team does that folks that aren’t an iteration of their existing practices. If you’ve never heard of test driven development, you’re not going to iterate your way towards test driven development. What you’re going to do is iterate towards more and more effective communication with the QA department but you’re not going to create test driven development. There’s a lot of stuff like that in Agile, particularly in the technical side although it also exists in the other aspects of Agile as well that fall into that category.
That’s what I would like to see people remember more about. One of them for example is evolutionary design which hardly anybody ever talks about but is one of my very favorite Agile practices.
Charles: Can you go into what that is?
Sam: I’m curious, I was about to bring up something about evolutionary algorithms and hill climbing but that sounds like a more interesting tangent.
James: I think some of the fallacies of hill climber, things that could go wrong with evolutionary algorithms can also go wrong with evolutionary design. I’d love to go there.
In a nutshell—I was talking to Ward Cunningham because he lives here in Portland as well. He was telling me about his definition of technical debt which for a long time I was confused by because he uses a different definition of technical debt than what I typically hear. He says that technical debt is what you deliberately leave out, it’s not bad code or code that needs cleaning. It’s the stuff that you’ve deliberately left out of your code.
What we call technical debt, just the bad code, he calls sloppy programing or sloppy code. He says don’t do that which I like. Absolutely.
As he was explaining this to me, I said, “Why would you want to deliberately want to take on technical debt?” He said, “It’s not sloppy coding, it’s this deliberately leaving stuff out.” You may have heard the phrase, “You aren’t going to need it.” That was one of the big catch phrases of extreme programming in the early days. This is the idea that we’re only going to build code for what we need to do today. We’re only going to solve the problem we have in front of us, we’re not going to anticipate the future. It’s really hard to do because as we look at our architecture, we can say we’re going to have a performance problem here, we’re going to need to scale here, and we’re going to build in hooks to make that possible.
YAGNE and Ward’s definition of technical debt says to get out faster, deliberately take on that debt and don’t build those things. Evolutionary design is how you get from not having those things to having them without rewriting your entire system. It’s a series of refactoring, gradual refactoring to improve the quality of design in your code so that you don’t have to build the world, so that you can ship with a third of what you need because you’re only delivering a third of your product right now. It’s very tied in with a minimum viable product, lean startup ideas.
Evolutionary Design is how you get from your one third product and one third code base or even one quarter code base or one fifth code base. The less code you write, the better. Because the less time you spend, the less bugs there are, and the more you can get out quickly.
Eric Ries who founded Lean Startup was a big extreme programming person. His ideas of continuous deployment and minimum viable product are very much a natural evolution of this evolutionary design and YAGNE ideas. I think it’s very important, one of the most powerful ideas on the technical side and extreme programming. But because it’s so difficult to see without a big example, it’s almost never taught.
Jessica: Does evolutionary design begin while you’re writing that MVP, that one third to one fifth?
James: It begins with the first line of code. The very first line of code you write. I always start out with Hello World. Let’s say I’m doing a software service application. The first thing I do is a build script, a real build script that generates a web page that says Hello World. It’s done in about ten minutes because it’s Hello World.
The next thing I do is that Hello World web page is now served by a server and I evolve that code from having just index.html to now having a server. From there, I stretch it out a little bit more and a little bit more. I was really skeptical about evolutionary design in the beginning but now I’m totally sold. I do it on everything. I don’t think I had more than about ten minutes.
Sam: You talked earlier about practices in Agile that you can’t iterate your way towards. We sort of mentioned the idea of hill climbing and local maxima. Are there things that evolutionary design will fall flat on?
James: Yes. I think so. There’s a famous example, unfortunately, that Ron Jeffries tried to use test driven development to write a sudoku application, a sudoku solver. If you search Ron Jeffries Sudoku, you’ll find it.
Ron Jeffries, for those of you who don’t know, is one of the creators of Extreme Programming. He didn’t have the background to do this sort of simple AI that’s required to do sudoku solving. He didn’t get anywhere, he evolved into a very low local maxima. Somebody else, I don’t remember the name of the person who’s very versed in these sort of AI solving problems knew the problem well came along and did it completely different and had a much bigger success.
I say don’t look ahead more than ten minutes, and maybe that’s an exaggeration, maybe it’s an hour. I do have experience with the types of problems I solve that allow me to avoid traps. That’s what you have to be careful about.
Jessica: It’s very easy for us to talk about don’t look ahead more than ten minutes when the fact is by virtue of our previous experience, we are already seeing months into the future without even thinking about it.
James: Yeah, absolutely. I think that’s a great way of looking at it. I don’t know that evolutionary design is something that a junior programmer, somebody without a lot of experience, can do successfully. I suspect they can do it as successfully as they can do upfront design because at least they’re getting the feedback from the code but you still have to have that experience to do good design.
When I’m doing evolutionary design, I have all these ideas about where the design might go but I don’t commit myself to them. I say I think this could happen and I’m going to wait to let the code tell me what’s going to happen. As a result, I come up with some really interesting designs because I’m leaving myself open for experimentation waiting to see what the features need which determines the code I write which determines what my design is going to be rather than the other way around which is usually features, design, code.
Charles: It’s funny that you mentioned that because as a hobby for many years, I have written fiction. That’s kind of the way that I write fiction. I generally know where I’m going to wind up but I have no idea what the next step is. When I’m writing that part of the story, I write that part of the story and then the next part of the story flows out of that.
James: It’s interesting you mentioned that, that’s a great analogy. You hear about authors saying I love being surprised by my characters, I love being surprised by my code. I love learning how the design can evolve in ways that I didn’t expect.
Jessica: But you’ve encountered negative surprises in the past so you know how to avoid those.
James: I would say that it’s more accurate to say that I’ve learned how I can paint myself into a corner and usually I paint myself in a corner by writing too much code. I become more and more confident about not writing the code that will paint me into a corner saying I’m not going to need it, I’ll wait for that to happen.
Your listeners might be interested in a video I put together years ago for the Norwegian Development Conference called Evolutionary Design Illustrated where I took several code bases, some of which I’ve done live on camera and checked in every 15 minutes. I took that code base and I created an animation that showed how the code base evolved over time which demonstrates—it was only 100 episodes so it was only 25 hours of work or 50. It still shows the early days of what evolutionary design can look at when you really push it.
Sam: I’ve seen some visualizations of how object graphs change over time. Those can be really, really fascinating to watch. Sounds very cool.
James: Yeah, I wish there was a tool that did this for me automatically. That would be just awesome to run that on a big code base and see how that changed over time.
Charles: We’ll inspire one of our listeners to write that for you.
James: Please do, mysterious listener. Let me know when you finish it. There this great animation tool that you’ve probably seen animate the Linux code base where it shows how files are added over time and there’s all these people sort of flying around in lasers counting out of their heads as they make files appear. It only operates at the level of file, I’d love to see something that shows the classes and methods and stuff in a system.
I think I saw something like that. Somebody took the code base for my Think Like a Git website and ran it through a tool that did essentially a big graph showing each of the classes and how they interact with each other. They ran that by commit and use that to build the firms of the animation, it was really cool.
Jessica: You talked about novice programmers probably aren’t ready to do evolutionary design. What can you do when you’re learning that will help you explore and get those pleasant surprises without despair?
James: I don’t know that evolutionary design is appropriate for beginners. It depends on how beginner you’re talking about. I was working with somebody recently, they were sort of struggling with how do I become more proficient. I said let’s pair together and I’ll see what’s going on and then I can give you some feedback. What I saw him do was something that I think a lot of beginners do but I didn’t realize was not obvious. That was that he solved problems by looking at recipes on Stack Overflow. I think a lot of beginners do this. Whenever he ran into an error, he typed in his error message into Stack Overflow and found a snippet and then would copy that snippet into the code base.
Sam: Which is totally valid, and I do that as well.
James: Yeah, it is totally valid. This is what I realized watching him was he didn’t have a mental model. He didn’t have the ability to predict what the code would do.
Jessica: Once you have that, it’s totally fine to cut and paste the syntax.
James: Right. My advice to him was yes, you could absolutely do that but you need to form a mental model. Before you run your code, you need to predict what it’s going to do. He was working with CSS so I showed him Mozilla Developer Network and said everything you type in, I want you to read the MDN page on that and follow any link that you think might be relevant and read that too. Understand what’s happening before you do it and form that mental model.
I haven’t talked to him since then, he hasn’t talked to me, maybe I’ve ruined a friendship. My theory is that he had sort of gotten to the point of advanced beginner. He’s working as a programmer for a company, he’s been there for about a year and he’s basically gotten as far as he can with that technique, the copy and paste technique. My hope is that this will get him over that hurdle to having that mental model and now being able to actually develop his knowledge in a completely different direction.
Sam: Maybe that’s the right tool for him to take up at this point in his journey.
James: Design isn’t even a question at this point I would say. That’s why evolutionary design isn’t appropriate for beginners if it’s not appropriate. It’s because design is just not a question yet. Design is about making code more understandable to humans, computers don’t care. You have to have experience seeing how code can be, particularly your own code, can become very hard to understand before you can really appreciate what design is all about.
Then once you have that appreciation, I think evolutionary design is a great way to practice it. I think it’s easier than doing a predictive design which is where you say this is what I need to do and here’s the design I’m going to create to do it.
Jessica: I tend to do some combination of this is where I want to get, break that down into pieces but then find a stepping stone, find something that maybe it’s not the cleanest, most direct path to where I want to go which is always a rewrite writer or you think it is. It’s a step that puts me a little closer in that direction and is individually useful and executable and runnable. Once I get to that stepping stone, I look around and is that still where I want to go? Often, I learn something that changes my direction.
James: That’s basically evolutionary design. Add in reflective design which is where you look at the existing code and say how is this designed and how can I improve it. That’s evolutionary design. Continuously paying attention to the code and reflecting on the exiting design and how it could be improved and not rewriting.
Jessica: Sounds very Agile.
James: To bring a full circle, this is the kind of thing that I think because we don’t have the technical practices front and center, they don’t need to be the most important thing, maybe. They definitely need to be as important as everything else.
Jessica: This is an example of technical practices that reflect the process model as well.
James: Yeah. I think that’s the real brilliance of Agile is that the technical practices reflect the social environment and the business environment and the business practices reflect the social environment and the technical environment and everything is connected.
Sam: The technical practices, especially like we’ve been talking about evolutionary design, those enable you to work in a way that is more agile because you aren’t planning ahead three months you can shift your direction and decide we don’t actually need that future that we thought was all that important because we delivered something and we found out that the market doesn’t actually care.
James: Yeah, and it also allows you to deliver sooner because you’re not building out all this infrastructure that isn’t needed for what you’re shipping right now. That’s the business environment, they want to see especially in a startup environment, you want to see something tested out in the market soon because you might be wrong about product market fit. Even if you’re right, you need the money coming in.
If you’re doing evolutionary design and not building more than you need, you can ship sooner. It’s also very cost effective because when you guess wrong with predictive design, it’s very hard to change your code.
Charles: One other thing with YAGNE that I found as I was learning that particular principle, it was mainly taught to me by David Brady who’s a regular on the show. Not only is it don’t take the time to create it but it was also that in many cases we found that because that code wasn’t run routinely, we built something in that was kind of a place holder or a connection point for something we hadn’t built yet, it turned out that that’s where our bug was.
Jessica: By the time you needed it, it doesn’t even work.
Charles: Or even worse than that, it doesn’t even work and it’s causing problems for the stuff that’s actually fully fleshed out.
James: I’m sure you’ve all seen, I’ve seen this myself many times. Somebody’s made a guess about where the design is going to go and they’re wrong. You can’t change it because its tendrils are all throughout your system. It’s a major refactoring effort to take this wrong idea out and change and put in the right thing.
If you’ve got a problem you want to solve, let’s say it’s internationalization which is something I actually solved with evolutionary design. I would rather have no internationalization in the code at all than bad internationalization that doesn’t work. Having to rip out the bad internationalization code is going to be way more work than writing the good code. Then from an evolutionary design perspective, if I’ve got to deal with internationalization, I’ll make sure that my code is factored such that all my currencies are in one object and all my dates are in one object. I have a single point to put that stuff in when I need it.
Jessica: That’s the kind of thing you’ve learned over time because you’ve implemented internationalization before so you know what got in your way then so you know there’s a hole that you need to fall into.
James: Right, but I don’t need to put the internationalization in. This is what I call Agile architecture. You’ve got a million things you could refactor. There’s always more refactoring to do than time to do it. What do you choose to focus on? Well, the experienced developer says I know that change will be most painful when internationalization comes on if we don’t have a single point of collection for data types. I’m going to focus my refactoring efforts on improving that duplication. You don’t write the internationalizations, you just make yourself ready to put it in if it ever comes up which it may not, you may decide that you’re going to pivot and do something else and this code is going to go away.
Jessica: Right, because there’s certain steps that aren’t painful. It’s not a lot of work and there’s other values to having the currency in one object for instance. There are other parts of internationalization that are really painful and those you’re not going to put in initially.
James: The project where we did internationalization using evolutionary design, we asked the customer can we please have a story for internationalization at least for Europe because this is a problem. She said no. You’re going to need this, she said no. This was my first XP project. “This evolutionary design stuff I think it’s ridiculous, I don’t think it’s going to work, but other stuff I thought was ridiculous like peer programming did work. Alright, I’ll try it.”
We didn’t put it in but what we did was we made sure—this was a web app—that there was one class in our entire application that loaded HTML templates from a hard drive. When it was time to internationalize those and localize them, we changed that one file. It was four lines of code to look at the location header and pick a different template based on the local.
Charles: One thing that I think is funny is we talked about evolutionary design and how it applies to fiction writing. Again, there are business strategies that are similar to this where if you read the book Procrastinate on Purpose, if you skip toward the end, there are five strategies. The fourth one is to procrastinate on purpose. It’s the same idea where it’s can I put this off, is it going to hurt if I put it off? When is the right time to do it and the right time to do it is as late as possible where it doesn’t cause a problem. With this, then you can prioritize the things that then need to be done now or are more important than that thing because you can continue to put it off with no consequences.
James: Absolutely. This is what Cunningham meant by technical debt when he coined the term. Technical debt is the stuff you deliberately chosen to put off because you don’t need it now.
Charles: Interesting, okay. One other thing you talked about when we were getting ready to come into the show was a lot of these practices, at least the way we’ve been talking about them, are mostly on the scale of when I work on my project and I leave this stuff out then I can pull it in later. I’m curious, how does that scale up to a team? Then, you have all this communication overhead with all the stuff.
James: I think at the team level it still works because these engineering practices were designed at the team level. It doesn’t scale so well to the cross team level though, that’s what I call large scale agile when you have interdependent teams. At the team level, we’ve got other practices, continuous integration, collective ownership, pair programming or mob programming is popular these days.
Getting everybody on the same page and resolving—when you have six people on a team and they’re all doing their own stream of work, what you really have is six one person teams. That brings in a lot of communication challenges. If you’re actually working with each other and everybody’s working on the code simultaneously, all the code they care about that week simultaneously, then it becomes one stream of work and the communication issues go away. This is where we get the overlap with the technical and management practices because it’s really valuable to only have one real stream of work simultaneously, it makes things so much easier from a coordination perspective.
Jessica: If we all work for the same dashboard, does that count as one stream of work?
James: Troll face.
Charles: Or if it works better if we’re all on one team, then all 200 developers in an enterprise company should all just be on the same team, right?
James: I wish we could, wouldn’t that be awesome?
Charles: I was going to say.
James: I say wouldn’t it be awesome because if you could actually have 200 people working on the same thing, there wasn’t the nine people making a baby in one month problem which there totally is which is why we can’t do that. But if we didn’t have that problem, that means instead of doing 200 features in a year, we could do a feature everyday and a half.
Charles: That would be interesting. What you’re saying is that if the overhead of the team didn’t scale linearly or logarithmically, in other words it was just constant whether you had nine people on your team or 200 people on your team.
Jessica: I’d be thrilled with linear or logarithmic because I’m pretty sure it’s exponential.
James: In fact, it is factorial. C Meth caps law.
Charles: I’m just saying best case scenario even if it’s the others, it still goes up every time you add somebody.
James: Yeah, that would be awesome but no we can’t do that. The maximum people on the team seems to be about ten programmers if you’re doing pairing, about six programmers if you’re not. Menlo Innovations got up to 30 people on a project which is pretty impressive. They’ve got sort of a 70 person company and team which is conversation for another time, I think. They’re doing some really interesting stuff that most people wouldn’t sit still for.
Once you get past about six programmers in a company, you need to start thinking about multiple teams. This is where large scale Agile comes in. I think there isn’t enough of the conversation around large scale or scaling Agile. Too much of that is sort of focused on the project management side and not on the how do we technically get 12 people to coordinate.
Charles: I have to wonder because if it’s between two teams and we have that coordination issue, it does seem like it comes down to communication. I’m wondering how much of that is artificial communication barrier that we put up ourselves and how much of that is just the nature of the fact that we have two teams and there’s not really a whole lot you can do to mitigate some of that.
James: That reminds me of essential complexity and accidental complexity. Fred Brooks wrote an essay called No Silver Bullet and he talked about essential complexity and accidental complexity. Essential complexity is the challenge that’s inherent in the problem. Accidental complexity is the challenge as introduced by the way you solve the problem.
For example, writing a software as a service app in C or Assembly introduces some accidental complexity that goes away when you use Ruby. The essential complexity of solving the business problem is still there. When you have multiple teams, what I see is that people tend to introduce a lot of accidental complexity but there is still essential complexity, you still have just this challenge of how do you coordinate people.
Sam: You go work on the routing code and I’ll go work on the persistence code and we’ll talk later.
James: Yeah, the typical thing to do is architecturally we’re going to split this in the layers.
Jessica: Divide it by technological implementation detail task.
James: Right. That’s a very good thing to do as a programmer because unless you’re using micro services, communication between components is instantaneous and perfect. People don’t have perfect and instantaneous communication so when you split up your team now that you’ve got 12 or 6 programmers who are all doing one person teams, and you say you’re the frontend person, you’re the backend person, you’re the micro service person, you’re the router person. When you do that, you now have maximized your communication links because every single features has to coordinate every single person, every single specialty.
One of the things that I think is good to do with a large scale system is to divide according to full ownership teams, teams that can own some segment of the problem completely from end to end and then be cross functional.
James: Right, when you have people on separate teams, they tend not to look at each other’s code. How does the communication happen? That’s where you start seeing breakdowns, that’s where somebody says I thought it was going to work this way but you did it that way. Also because when you have people split into lanes like that, somebody will get ahead just because it’s impossible to balance the work perfectly. They will jump ahead and they’ll start speculating. Oh, I think this is going to come up next. They’ll get it wrong and then they’ll start having to go back to previous tasks because the other team is still working on the previous team and they need something else. Now, you’re multitasking. Before long, you’re spending all your time just hopping from task to task and your life sucks. Let’s not do that because that’s not fun.
Charles: How do you manage the crossover? Another example that comes to mind is let’s say that you’re building a huge content system. You have a content team of maybe user management team. The crossover is authorship because there’s an author or owner for each piece of content. How do you manage that so that you don’t have these communication issues?
James: You’re going to. That’s where the essential complexity comes in, you can’t prevent communication issues. What you can do though is design your system so that you have as many intra-team communication points as possible and as few inter-team communication points as possible.
I actually have a video that just came out recently that I did in Romania about this topic. Maybe we can share a link that will go into more detail on this. The core idea is basically follow Conway’s law which I think we talked about earlier. The structure of the system tends to match the structure of the organization and vice versa. How can we design our architecture and our code so that the people who have to communicate most are on the same team?
Charles: That’s interesting. Basically what you’re saying is if we can simplify the way that the content system talks to the user system then we can simplify the communication lines have to happen between the content team and the user’s team.
James: Or maybe half of the content people and half of the user people need to be on one team and the other half need to be on another team. Let’s take the software as a service example, you could split it into frontend and backend. Let’s say you’ve got a startup, been going well, you’ve got this code base and the code base in addition to having frontend and backend has the on boarding experience which is sort of the front page and everything that you see when you’re not logged in and the actual app which is what you see when you pay and you log in. You could split that into two teams and what you’re going to do is minimize the communication between those teams.
Each one will have content people on it but different kinds of content. The landing page experience might have an accountant or a salesperson or a marketing person on it. They’ll probably both have user experience designers.
It gets complicated as you get bigger because you’re going to start having shared services. How do you manage that? That’s a big part of what I’ve been trying to figure out because it’s easy to say just split into full ownership teams but how do you do that in actuality? I go into some of that in my video but there’s still a lot to learn here, this is most of what I’m doing as a consultant these days. I’m trying stuff out and learning from it.
Sam: Right, because it seems like there are always going to be things, especially as an organization gets larger, that either multiple teams want to own or that nobody wants to own.
James: I recently worked with a company that had 42 teams. We did a before and after. Before they had a diagram that had an exponential number of lines of communication between all those people, or maybe it was factorial as you mentioned, Sam.
Sam: Actually, I have to correct myself on that. I was looking again at MetCalfe’s law and it turns out to be n2.
James: Okay, so exponential. I don’t know.
James: It’s to the end that exponential. They had these huge number of links so we did this big effort where we would just redesign in cooperation with the architects and lots of people in the company. Redesign team responsibilities to minimize that. We got it down from dozens to maybe a dozen.
Charles: Oh, wow. How do you go about making that change? Do you rearrange the teams like you said or does it involve refactoring the code or both? We keep talking at Conway’s law so it seems like the one will affect the other.
James: Absolutely. I think that’s one of the most important things that I would like the people working on scaling are able to realize is it’s not just a project management problem, it’s not just how do you tell people what to do which is unfortunately what a lot of these things are focusing on.
The way I’ve done it so far is by working with a company to redefine teams and designs and basically move code responsibilities around between teams. It’s fairly disruptive and not a lot of fun for the people involved who are going through the change but it’s over really quick. The team that I worked with that had the 42 teams, we started our design work in January and we rolled out the change at the end of April which is super fast.
Jessica: So it’s not just a process management problem, it’s also a technical architecture problem?
James: And a change management problem, at least when you’re implementing this kind of thing. If you can grow into it, I don’t think you’re going to have this same sort of—if you can grow your company with the knowledge of needing to architect your system in this way, then I think you’ll be less of a problem. I would love to learn how to evolve an organizational design in this direction but I don’t know how to do that yet.
Sam: When Chuck asked how do you go about making that change, I was struck by the fact that you can ask that question in terms of how do you physically rearrange the teams but then also how do you convince the organization and the executive layer that this is a change that needs to happen and how you get them to go along with it. This is a question you can actually ask at quite a few different layers up and down the stack.
James: Yeah, absolutely. How do you get people to go along with it? That’s something I get to do as a consultant because when people hire you as a consultant, they’re more willing to hear ideas that seem out there because they don’t know you and familiar leads can tempt.
Sam: And they’re paying for you.
James: Exactly. There’s that aspect of it.
Jessica: Michael Feathers said something in conversation the other day that might be part of the answer to this. He said gelling teams is hard, what if we do it more? What if instead of saying that team gelled, don’t touch it, we got good at gelling teams? As individuals and as an organization if we acquired that skill, then we could evolve teams more.
Sam: That comes back to something that I’ve heard a couple of people including you say which is if there’s something that’s difficult or painful, you should do it a lot more.
James: That was one of the early XP ideas, I think it may have come from Kent Beck. He was time a continuous integration specifically I think. Really, all kinds of things. I’m a little skeptical about this one.
Jessica: Everyone’s skeptical about it which means that if it’s right, it’s a high value surprise idea.
James: That is so true. When I hear an idea that I want to steal, that’s what I say, ooh.
Charles: What he’s hearing is cha-ching, cha-ching, cha-ching. I know how this works, I have a whole show about freelancing.
James: Menlo Innovations, which I mentioned earlier, they’ve got a 70 person company that operates as a single team. I think they’re doing what you’re talking about. I think they—
Jessica: It’s fluid teams.
James: They reassign pairs, they have week-long pairs which is also something I wouldn’t have done. There’s many things they do that I think are weird that they do successfully, ooh, let’s look at that more. They have their project managers assign pairs with the goals of sharing knowledge. They effectively act as a single person team because they’re switching pairs every week and moving people around amongst all these projects.
I was talking to the president or CEO there and Rick shared, he’s got a book called Joy, Inc. which I hear is good, I haven’t read it myself. It’s recommended. They’re about 70, 80 people and they’re not growing any further because he’s afraid that more growth will prevent their formula from working.
Jessica: Good call.
James: There is this number of people, people have this natural limit of how many folks they can remember who they are and have positive social relationships with, the Dunbar number. It depends on the person but it ranges from about 150 to 250. I think when you’re looking at these large scale systems, that is a key cutoff point.
There’s a company, Gore, that makes gore tex, the waterproof fabric. They practice a flat organization. Whenever they pass more than 150 people in a building, they spin off a new building because of Dunbar’s number. I think that would be another natural limit. 12 is one natural limit and then 150, or maybe even less, maybe 100 is another natural limit.
Sam: I feel like 30 is another one along that phase transition point along that continuum but I have no research for that.
James: I think you’re right because I was talking to Rich about his company and I said okay, you’ve got 70 people acting as a single team but how many people have you ever had on a single project? His company does outsource product development with an emphasis on user experience and creating good social results. He said 30 people, 30 people is the most he’s ever had on a single one of his projects even though he’s got 70 people in a company.
Jessica: Two is an interesting one as well. All that communication is free while it’s just you. As a developer starting out, if you can add one person to your project, you will learn a lot about coordination.
James: That’s true. You can get away with not having source control with one person. I’m not saying you should.
Jessica: No, future me would hate past me. That’s close enough to two people right there to need source control.
James: I believe in the value of learning through failure. As a beginner, not doing source control and then discovering the reason for it hopefully without a hard drive crash.
Jessica: We’ve probably all written essays and lost our work. I think we’re already there.
James: If anybody listening to this is not using source control, go beg forgiveness and start using source control.
Charles: Let us save you the headache.
Sam: Jim, you mentioned something about how Menlo Innovations are doing some things that seem really weird to you but they seem to work for them so maybe that’s worth checking out. That made me think of something that I read years and years ago about an evolutionary hardware design problem that somebody basically set up an evolutionary algorithm to run on a set of field programmable data rate ships. The task was to differentiate between two different frequencies.
One of the solutions that came out of this process, it actually took advantage of a CPU that was close enough that one of the chips could pick up the clock frequency from the CPU. It used that as a baseline to determine what the two other frequencies it was distinguishing between were. The upshot from that of course was that evolutionary designs are really good at taking advantage of local quirks and things that can’t necessarily be replicated, they depend a lot on the underlying materials. I wonder if there are things that work at Menlo Innovations that work because of the people who are there and maybe that’s another part in why they might not want to necessarily scale because as you hire different people, that makes changes, and maybe some of their practices that worked before stopped working because people who upheld them left.
Sam: I think that is absolutely true. Maybe not the specifics of Menlo but the idea that when people get into advanced Agile and they evolve their process, they do their retrospectives and actually pay attention and follow through on whatever changes they suggest. They come up with a very different kind of process. Often, what happens is they take this out and say this is what works. This is what worked in our context and it can be really hard to know what that context was.
You may have heard of the Spotify model which is a model for scaling Agile. The Spotify folks are great because they know that they’ve come up with this model through lots of emphasis on learning and change. Whenever they talk about the way they work, they make a point of saying don’t copy the model because that was what worked for us. What you need to copy is the way we evolved, not the details of what our model is.
Absolutely. Any highly developed process, Agile or otherwise, is very much in context.
Jessica: It’s meta, you’ve got to copy it down a meta level. Copy how they got there, not where they got to.
Charles: Certified Scrum Master, that’s how they got there.
Jessica: I want to comment on Sam’s point about the Menlo Innovation process, working for the people there. This is, in my opinion, not about the personalities and characteristics of those individuals that they came in with. It’s about those people having been there, having been part of the system in the past, having been part of its evolution. It’s about their shared experience that when as a new person, you come in, you don’t have those instincts yet. It doesn’t matter if you hire people who seem just the same as you, they still don’t have your in context experiences and it’s participation in that system that is making that system work for them, not some inherent person.
James: You are so right. In fact if you’re hiring people who are just like you, then you’re missing out on opportunity for lots and lots of new ideas and new perspectives. What I’ve noticed with teams and I assume this is happening at Menlo. Rich shared and had a talk at Agile India this year which is really fantastic, I recommend that you watch it.
When you have a gelled team, what I’ve noticed is that when a person comes on to the team, they get boardified. The rest of the team assimilates them and they come into the process and start following the culture of the team. As long as you grow slow enough, what you need is just people who are willing to be assimilated, not people who are just like you. Actually, my opinion, and I have no evidence on this, is you need people who are absolutely nothing like you or at least willing to try and participate wholly.
Sam: That’s what we’re all about. It was right there in the show, they were talking about we’ll add your cultural and technological distinctiveness to our own.
James: That’s the lesson, and it isn’t being a consultant, I’m also saying you should be assimilated. It’s a really positive show here.
Charles: This has been really fascinating. I’m going to go ahead and have Sam start with picks.
Sam: I was going to pick a couple of things from my recent vacation. As we were talking, I was reminded of a few other things that now I’m going to pick instead.
The first of those is a blog post by Sandy Metz from earlier this year but is talking about something that she put in a talk about two years ago called The Wrong Abstraction. It’s well worth reading. The gist of it is something that she threw on a slide in one of her talks that I picked up on and this sort of became one of the Sandy memes. “Duplication is far cheaper than the wrong abstraction.” There’s a lot to unpack in there, I’ll let you read the blogpost, it’s really cool.
The other thing I want to pick is Tom Stewart who we had on the show. He apparently has a podcast, a very occasional podcast that he calls Why Are Computers. It’s a very regular thing. After he was on our show, James, the former Rogue, had a bit of a beef with him so they had a really interesting conversation that came out of that in which they wound up mostly agreeing but in really interesting ways. THen, he also more recently had Sandy Metz and Katrin Owen to talk about their book, 99 Bottles. At any rate, Why Are Computers episodes two and three are the ones I’ve listened to and they’re both really fascinating.
Jessica: My pick today is NetStat. This is a unique C command line tool. You can use it to find out what program is listening on what port. The trick is you can’t just type NetStat, you have to do NetStat—listening—programs or NetStat—lp if you want the abbreviation.
Charles: So handy, though. There have been a few times where I’ve run into issues where it’s like why won’t this start and then it’s like oh, something else is on that port, what is it? Oh, I’ll go kill that.
Jessica: Yeah. Or yesterday somebody gave me a link to a website that they wanted me to look at. I was like hmm, what program is running this? I logged into the host that it was running on and did NetStat—listening—programs and found out where this is coming from.
I could use this internally to trace routing. When I make this call, what code is executed so that I can, do we get blame with it and find out, who to talk to to change it?
Charles: Jessica is now the server whisperer.
My first pick is we had a team building session as part of Wood Badge, I’ve talked about Wood Badge a couple of times since I’ve been back. A lot of the examples they used was from the movie Remember The Titans. If you’re looking at team building, especially where half the team is White and half the team is Black and this is right around when they’re doing the desegregation of the schools. There’s a lot of antagonism and hard feelings there, and just seeing how they progress into a well working cohesive team and some of the issues that they have there. There are a lot of parallels between that and programming teams or other groups of people. I’m going to pick that movie.
I think that’s all I got today. Jim, what are your picks?
James: Well, I’ve got two picks. I didn’t get warned about this at all but I’ve got two for you. The first one is that Agile India 2016 talk by Rick Sheridan about what they do at Menlo. It’s just fascinating.
My second one is a little known book that I think deserves a lot more attention. It’s called Liftoff and it’s by Diana Larsen and Ainsley Nies. Diana Larsen is one of the people who introduced retrospectives to the Agile movement. Liftoff is the other side of it, retrospectives is what you do as your project is going and liftoff is what you do to get your team off the ground. Really great stuff about how do you charter teams and how do you get them all coordinating well and working together effectively.
I think this book deserves a lot more attention. They recently came out with a second edition and it’s published by Pragmatic Press. Highly, highly recommended.
Charles: If people want to follow up with you, find out what you’re doing, hire you as a con-sultant, what do they do?
James: You can reach me at email@example.com, you can go to my website jamesshore.com and see what a bad user experience designed I am. I would love to hear from folks.
Charles: With that, we’ll go ahead and wrap up the show. We’ll catch everyone next week.
Charles: Mind blown. Awesome.