CHUCK: Hey everybody, and welcome back to the Ruby Rogues Podcast. I’m your host, Charles Max Wood. And this week’s panel is, in no particular order , Avdi Grimm. He’s the author of Exceptional Ruby, he blogs at Avdi.org/devblog, and you can find him on twitter as @avdi. Welcome, Avdi.
AVDI: Hello again!
CHUCK: We also have Gregory Brown from Ruby Mendicant University, and he is the author of the Prawn Library and Ruby Best Practices, and has a lot of experience helping developers kind of go from that beginner or intermediate stage and move up. Welcome, Greg.
CHUCK: We also have from Shiny Systems, David Brady. Shiny Systems is his company. Are you still hiring, Dave?
DAVID: Actually, I’ve got some really interesting developments going on this weekend. I’m trying very hard to hire somebody and trying also to avoid getting hired out right. Apparently when you challenge the universe, the universe challenges back.
CHUCK: All right, so you can check them out. He blogs at heartmindcode.com, he and Pat Maddox have an awesome podcast at ADDCasts.com. And he is also the author of the tourbus library for testing your application. So welcome, Dave.
DAVID: And I’m @dbrady on Twitter.
CHUCK: @dbrady on Twitter. All right. We also have Josh Susser. Josh is one of the organizers of the GoGaRuCo, The Golden Gate Conference. He blogs at hasmanythrough.com. You can find him on Twitter @JoshSusser, he’s contributed to the Ruby on Rails library some and as we’ve said before, he have seven patents, which I think is rather unusual for software developers. Welcome, Josh.
JOSH: Thank you. Good to be here.
CHUCK: And we also have… we have a one more than we normally have; we have James Edward Gray. James is the author of the Textmate book. He ran the Ruby Quiz for a while and wrote the best of Ruby Quiz, he wrote the FasterCSV library, and you can find him on Twitter as @JEG2. Welcome, James.
JAMES: Hey! It’s glad to be back.
CHUCK: And I’m Charles Max Wood. I have screencast and podcast at teachmetocode.com, I’m also the host of The Rails Coach podcast and this podcast. And you can find me at teachmetocode.com. I’m on twitter at @cmaxw. All right, welcome everybody. This week as we said last time, we are going to be talking about metaprogramming. This was one of the topics that was requested on the Ruby Rogues website. And it seems like you guys have some ideas of where you wanna go with it. So, I’m going to turn it over and just let whoever wants to jump in, get the ball rolling.
JAMES: So I thought an awesome question to start us off is, “What is metaprogramming?”
CHUCK: Ooh, there’s a deep one.
JOSH: What a meta question.
JAMES: [Chuckles] Right.
CHUCK: [Chuckles] Since this is all meta, we are just going to sit here and think about the answers for a while.
JOSH: Can we think about thinking about the answers?
JAMES: It’s getting deep.
CHUCK: Let’s not get carried away here. [Chuckles]
GREG: Okay, so I´ll try an answer. This is Josh. Metaprogramming is programming other programs.
JAMES: Yeah, that’s pretty much what I always used to find it as was code that writes code. Although, I kind of think Ruby blurs that line in a lot of areas.
GREG: Yeah, I think that the question is going to depend on whether you are talking about metaprogramming in general, or metaprogramming in Ruby, because metaprogramming takes on I think a broader meaning in other languages; where in Ruby, I always tend to think about the metaprogramming functionality as just an API for dynamic programming within Ruby itself.
CHUCK: I always thought of it as basically writing code that defines other programmatic constructs.
DAVID: This is going to be a fun day today because for me, metaprogramming is almost always… I mean a lot of us… we’ll probably talk about DSLs and that kind of thing but for me, metaprogramming is largely about monkey patching, duck patching or monkey patching, which is programming other people’s programs for them, whether they want you to do it or not. But then in Ruby, I love the take… I think it was Josh that wrote… he blogged about it and basically said, “There’s no such thing as metaprogramming; there’s just programming.”
JAMES: So let’s try a concrete example to see if we can zero in. Like in Rails, where you have your ‘has many’ and ‘belongs to’ or ‘validates numericality of’ which is so popular. Are those metaprogramming?
DAVID: I don’t think of them as metaprogramming. They are just class methods.
JOSH: So, I think that things like the association has many, those things, I do classify those as metaprogramming in their implementation. Using them it looks just like an API, and when you call ‘has many’, however has many methods in your class, has many comments, then that defines a bunch of methods on your class.
DAVID: Oh, that’s true!
GREG: That’s what I was thinking about. I was thinking about how from the outside in, you can’t actually necessarily tell. Like I would consider it metaprogramming if it actually defines method or otherwise messes with structures. But say for example, it was just storing… like if you have a class method that takes blocks and then stores those blocks and calls them later, I don’t know that I would consider that metaprogramming.
JAMES: So that’s interesting. By that definition Greg just gave, the association methods are metaprogramming, but the validation methods would not be metaprogramming.
AVDI: So if it generates a class or generates a method?
JOSH: What about method missing?
CHUCK: I was going to ask that because effectively, you are…
JOSH: Sorry, go ahead and ask that.
CHUCK: [Chuckles] Well, you are effectively modifying the interface, right? I mean, it accepts a message that it wouldn’t have accepted otherwise if you monkey around with method missing, but you are not defining a method itself.
DAVID: You are not defining a method; all you are doing is you are saying, “My interface is extremely promiscuous.”
GREG: [Chuckles] That doesn’t define methods for you. That just changes the way that dispatch works. And then, the question is, if method missing is a form of metaprogramming, then you need to sort of think of ‘send’ as also being a form of metaprogramming. In a way, they sort of go together. And I don’t know that I feel that those things… like I actually am of the opinion that there really is no such thing as ‘metaprogramming’ — in Ruby at least. I just think of them as APIs that do various dynamic things. And I think that they are actually entirely different sort of things; like I feel like finding a method and defining a hook for when a method doesn’t exist, are two very different things.
JAMES: So maybe for a working definition for today, maybe we can agree on something like, some chunk of code that modifies the object structure, in ways like adding methods or responding to methods or things like that, making new classes available, possibly?
AVDI: What about re-write? Reg is rewrite, which actually does do kind go metaprogramming in a more classical sense, where it is generating new code.
GREG: Right. That’s a question about code generation and whether code generation should be considered metaprogramming. I think it would actually be more productive to talk about including all of the concepts of the right code to make modifications of code dynamically, that would or narrowly be done by something explicit. So metaprogramming is always implicit and dynamic. And it’s goal is to provide a more generalized way to do something that could have been done statically.
DAVID: Right. Basically, if it’s something that will blow the mind of a Java programmer, I consider it metaprogramming. [Laughter]
JAMES: That’s awesome.
JOSH: So the entire Lisp language is metaprogramming?
JAMES: Yes, absolutely. I agree.
DAVID: Uncontested. Yes. [Chuckles]
JAMES: All right, let’s try new questions and come at it from a new angle. It’s clear we have no idea what we are talking about today.
DAVID: Which is fantastic. That is a very important fact. We have established that we cannot define what this is, so that is the first thing you need to know about metaprogramming.
CHUCK: So you just got the meta definition for metaprogramming?
JAMES: That’s right.
JOSH: Well, if you take it from the thing that all of Lisp is metaprogramming, if you say programs that treat code is data.
JAMES: Yeah, that’s interesting.
DAVID: That’s a good way to put it, yeah.
JAMES: But by that definition, I can think if quite a few blocks and areas just using Ruby’s blocks, that I think would qualify as metaprogramming.
GREG: So why don’t we just change the topic to ‘code has data’ and then just roll with that?
AVDI: But Ruby doesn’t do code as data.
DAVID: It doesn’t.
JAMES: Interesting. Why do you say that, Avdi?
JAMES: Actually, that’s not true in 1.9, is it?
GREG: No, that’s not true. You can get source code, you can get AST, you can get all…
AVDI: Can you ask an object for it or do you need to load in Ripper?
GREG: Well, Ripper is a standard library, so…
JAMES: I think you can actually get the source itself without Ripper. They now have a source…
GREG: They have a source location method…
AVDI: Source location, that’s a little different.
JAMES: Yeah, right.
AVDI: Especially if the source has been generated or changed at runtime…
DAVID: So we have a list of symbols that get mapped out into define_method, then you get that location and you are like, “Oh, this is a ‘for’ loop. I don’t really see the source code here.”
GREG: That’s the thing about the source code stuff. Now, so I guess… Let me think about this… The source code stuff definitely works if you’ve got something that’s in a file and then you can get the source because you can get the source location and then parse it from there. But if you are dynamically generating code, could you get the source back? Probably not, without some sort of Ripper hack on that. I’m on even sure that you could do it.
AVDI: And also, you can’t marshal code in Ruby.
JAMES: That’s correct. There’s a very good reason for that. The concept of a block in Ruby is a closure, so how do we properly marshal a closure and then bring it back with all that state?
AVDI: Yeah, exactly.
DAVID: I’ve actually considered forking Ruby just to add two little changes to it, which is that everything should be treated as an expression should return it’s value. Therefore, when you issue the class statement, when you say class, pick truck, blah, blah, when you type ‘end’ that should evaluate. And you usually get back the definition of that class. And the same thing when you define a method with ‘def foo’. Once you are done, you should get back a proc object. And Ruby doesn’t do that. It returns nil from both of those of those evaluations – which is unfortunate.
CHUCK: So I have a question on that then, Dave. If you monkey patch something then, would it return the value of whatever it is that you monkey patched? Like the new class definition or whatever?
DAVID: In that case, I don’t know that… I definitely don’t have the chops to just monkey patch Ruby straight. I think you’d actually have to patch… actually have to fork Ruby and change the source code, I think.
JAMES: So let’s move on to a different angle outside of definitions. I though another interesting question might be, “Do you see too much or too little metaprogramming in Ruby?”
GREG: That feel like too much all the time.
DAVID: I halfway agree.
GREG: The use of metaprogramming that I see is either wrong or has an equivalent that is less confusing. But then again, that was intermediate and beginner developers mostly.
JAMES: So that’s an interesting point. I think we have a tendency maybe to reach for the big tools when there’s a simpler tool that would do?
CHUCK: I like power tools.
GREG: I did a lot more metaprogramming before I got more experience with Ruby.
CHUCK: So Dave, you were saying too much and too little?
DAVID: Too little, yeah. I don’t know. I talk to a lot of people that see metaprogramming as just black or white. It’s either completely evil and we should get rid of it, or it’s god’s gift to mankind and we should use it for everything. And it really is a Swiss army knife. The beauty of metaprogramming is once you figure out how to do a meta, you don’t have to write regular code anymore, because you can just do metaprogramming. And that I think what Greg is mentioning that, “Oh yeah, you figure out how to solve this problem using metaprogramming. But you know what, there’s actually really straightforward way to do it in the language, it would be nice if you would just take time to go learn the library and do it that way.” But at the same time, I also see a lot of people just really sweating and breaking their heads, trying to write something in a straightforward manner, because metaprogramming is either difficult or unavailable. One of my big problems with rdoc for a long time is that if you metaprogram something up, there’s not really any way to document that. Like if you had a code generation, you actually had to generate static code and static documentation or rdoc can’t see it.
JAMES: That’s an interesting point. And at times, I’ve actually argued that’s a feature that maybe if your code is so clever rdoc can’t read it, then maybe that’s a hint, you know?
DAVID: Yeah. It is a hint. It’s a hint that rdoc isn’t smart enough. [Laughter]
JAMES: Well, I’m not sure it’s that hint.
JOSH: That’s actually a problem or a characteristic of Ruby that makes it difficult to build a good development environment. When people run around and dynamically create methods and classes, and new pieces of code, and those things don’t live in file somewhere and the people don’t declare where the source or location of bits of newly created code are, it makes it really hard to… there’s no way to do static analysis with your program and figure out where all the methods live and are defined. So you can’t build a browser. Smalltalk, all the program structures lived in memory in the image, so you could build a browser that knew everything was.
DAVID: I was going to mention Smalltalk specifically for that because it’s very dynamic, but it’s got a test and set of tools. I’m very excited to see Ruby moving towards more than an AST reflection, that sort of thing. And yeah, if we can get the ability to grab a running system and query it’s object structures. Oh, tingly, that’s just great.
CHUCK: Okay. I have to ask my new question now. I’m vaguely familiar with what an AST is, but I am not familiar enough to have enough context to completely follow what you guys are saying.
DAVID: Abstract Syntax Tree. It’s the graph of the code, broken down into like a flowchart… into a tree, basically data structure, reflecting what code is, what the source code compiles into. Basically, what objects in memory, what instructions have to happen in what order.
JAMES: So just to give a super simple example, if you have a state like 1+2, it generally compiles into a tree with the + at the top and a 1 on the left under it, and a 2 on the right underneath it, meaning, “Do this plus operation on these two operators.”
JAMES: Also, that’s used in a lot of areas. We tend to think of it just as a programming and stuff like that, but the changes that Aaron made to Rails 3’s where order type methods on active record, basically all he did was have those methods build an abstract syntax tree, so that then he could walk that tree and translate it to SQL.
JOSH: Usually, ASTs are used as an intermediate representation for a program in the compilation process. You look at the source code, you parse it out, you create an abstract syntax tree and once you have the AST, you can generate the byte code or the machine code that you need to execute the program.
AVDI: It’s probably worth noting that in a Ruby AST, the 1+2 — and correct me if this is not quite right — but the 1+2 is more likely to be… the top of the tree is ‘method call’ and then the name, the 1 is the receiver, and the + is the method, and the 2 is the first argument.
JAMES: Yeah, I did simplify it a bit much.
DAVID: Yeah. The other advantage specifically looking forward from a AST is what this would give us if you could reach into a running object system and grab it’s AST, is you could create an object, spin it off into memory, and then you could monkey patch the class or monkey patch that object, or you could delete the source code. This is how Smalltalk works. You can change the source code, but that object is still hanging out in memory based on old source code. But you could still grab it’s syntax tree and interrogate that object and say, “Oh, you know what, you don’t mess the source code that you were supposedly built from because you are older, but you are still running a real object and now I can actually step in and kind of debug you through your syntax tree.”
JAMES: So now this is interesting. So we talked multiple times about getting the syntax tree. I believe 1.9 does give you methods on the VM to compile to the byte code, but I don’t think it gives you the syntax. Well Ripper kind of does, right? I think. So that’s one way. And I believe Rubinius may have some capabilities.
JOSH: It does.
GREG: It has stuff, but Ripper itself, the problem with Ripper, I mean we played around with it as one of the exercises at Mendicant university. And the more we looked into it, like the way Ripper was set up right now, it’s not really meant to be used as a general way to access the AST. And so there’s like weird things about the way that it does stuff that introduce ambiguities that make it sort of impossible to have it go around trip.
AVDI: Can we talk about guidelines for metaprogramming a little bit?
JAMES: Absolutely. Let’s do it. Go ahead.
AVDI: So, this just kind of occurred to me as we were having this discussion. One guideline out there is if you are redefining methods or aliasing methods, you are probably doing it wrong. If you are generating modules, you are probably doing it right.
JAMES: Agreed. I absolutely agree.
DAVID: Okay. But with emphasis on “probably”.
AVDI: Well yeah, I mean there are always exceptions. I mean, a classic example of this was the alias method chain, that you saw all throughout Rails up until 3, where a lot of times, methods were being aliased and redefined when you could have used dynamic module generation to do it in a much cleaner way, where you’d still have access to super, so the methods could just cleanly call the original definition but you are inserting a module that puts the new definition in.
DAVID: That’s fairly clever. Now, can you stub out super? This is something I’ve found in Smalltalk last week.
JAMES: No, I don’t think so.
DAVID: There’s absolutely no fear of sub classing in Smalltalk. And we do see a little bit of fear in sub classing in Ruby. We see a lot of monkey patching than we do sub classing, I think and inheritance. And part of it is I don’t think you can stub super. But in Small talk, you can.
AVDI: When you say ‘stubs’…
DAVID: For like testing. Basically I’m going to call new on fixnum, and it’s going to call super up into integer, but I’m testing something… these are bad examples because they are in the standard library. The idea is to basically say, “When you call super, nothing happens. I’m not going to let you go up the chain. “
AVDI: What you can do… So when I’m testing something like that, I’m usually just… I create just an object to insert my module into. What you can do is you can actually ask Ruby if super is defined and not too many people know this, but Ruby has the defined operator — which is not a method, it is an operator — and you can actually say ‘define?(super)’ and that will tell you in any method dynamically whether there’s a super to be called. So you can say ‘super if defined super’ kind of like you say ‘yield if block given.’
JAMES: I just learned a new trick. [Laughter]
DAVID: Can you then undefine super? No, because you have to be inside the context of a method, don’t you?
JAMES: I wanna go back to where Avdi has been saying though because I actually believe this is a really important point. If you’ve seen any of my talks at conferences, I bring this up a lot. Ruby’s method call system is really important. And you have to get your head around it at some point about how it does these lookups. And what Avdi is saying is eventually, you realize that Ruby’s method call system is a straight line with stops along the way. And what makes it beautiful is that once you get the hang of it, and once you get the hang of modules — especially things like extend and include — then you realize that you can put a module at any point on that line.
JAMES: Almost, that’s right. There are couple of very minor exceptions. But usually, you can put a module at any point on that line. And what’s great about that is that means you can put modules at the front of the line; so you can do things like around filters, do something a little before call this method, do something after blah,blah, blah, or you can… David asked can you stub out a super? Well, sure. Put a module between the point where super is going to be called and the thing above it, right? So yeah, sure you can.
DAVID: That’s cool.
JOSH: So the tricky thing there is… and I think one of the reasons that we saw alias method chain in Rails 2 a lot, is that if you have a method and in the leaf class that you are looking at, and it’s not been built in a way to allow for extensibility, it doesn’t call super maybe, then there is no way that you can go and insert something above it to compose in some behavior. With Ruby, you can call extend on the instance and smash some behavior onto that that will override the methods define in the class of that instance. So that’s one approach.
AVDI: And thank you for noting that because a lot of people…
GREG: Yeah, that was a concern that I was going to point out as well that basically, Avdi, the guideline that you gave is completely valid, assuming that everybody plays nice.
AVDI: Well yeah, and there’s this kind of old rule in Ruby that unfortunately [chuckles] gets forgotten a lot which is calling super is usually a good idea. There’s I mean, Ruby is one of those language, a lot of language in the class or the object initialization, they just automatically do super class initialization for you, so if you’ve got constructors in all you super classes, they’ll just automatically call those things. Ruby says you can decided whether to call those super class constructors or not. And so there’s this guideline that you have to remember, which is that unless you really have a good reason not to always call super in initialize, and that really goes for a lot of other methods as well.
GREG: I mean, I think that’s because Ruby splits out the concept of object creation into… basically, allocate is called in the class and then initialize is an instance method that gets called. And Ruby separates out the … of the object creation and sort of the hook provided to the user to set some data. So, it’s Ruby constructors are not proper constructors in the sense of a lot of other languages or at least…
AVDI: A lot of other languages do that. It may be a little bit more hidden, but a lot of other language do that same kind of thing.
GREG: Right. But I guess the point that I’m making about this idea of naturally calling super is a little bit strange in Ruby because the hooks that you are defining is this by definition user defined hook, you are not typically messing directly with the constructor.
JOSH: It’s nice though being able to control the sequencing of things in your initialize method, that you can do little work, call super at the right moment and then do a little work after it.
DAVID: So basically, by injecting these modules, it’s almost like giving you the advice from Lisp, right? Like you define and then advice which basically is just a wrapper around a method. That is very cool.
JAMES: Just to hit home how important this point is we have been talking about right here, I mean I hate to be like you know… “You really ought to learn this,” but you really do, because if you wanna know what happened between Rails 2 and Rails 3, they rewrote pretty much the guts and the internals of Rails, to do exactly what we’ve just been talking about for the last 8 minutes or so. So this is really important; you reach a point where you realize that this is how Ruby is supposed to work. And once it works that way, you have a lot of power over what you can do with the system — and that’s very important.
DAVID: That is very cool.
CHUCK: So I have a question. With the dynamic module generation that we’ve been talking about, I definitely see where the supplies, within say a class, but within the main context and defining classes within dynamically defining classes, does the same concept apply and how is that?
JAMES: So you are saying if you just defining classes normally and stuff?
CHUCK: No. If you re dynamically defining a class, then you are not going to generate a module to generate the class, right?
JAMES: I see what you are saying. So like if you are doing some trick in summoning objects out of the user or something?
CHUCK: So for example, [chuckles] I did some playing with Rack and I dynamically defined some middleware to help manage some of the routing and dispatching that I was doing with the request that were coming in, and so it was just dynamically defining the classes within the methods that I had added to the main context of the program. And so we were talking about dynamic module generation within the sense of adding a stack or adding these levels and adding functionality to classes, but do you do the same thing with the main context of the program itself?
GREG: Well I don’t know that it makes a difference and I think that it dynamically generate a class is basically in the end, the same as regular class. So you could apply this technique if it makes sense. I mean, we were also talking about say you wanted to add behavior to or change the behavior of the existing object using modules for extension make sense. If you are talking about creating classes dynamically like for example… I mean camping used to have those weird routes that would basically take a regex and then turn it into like a controller object for you or controller class. I don’t know where modules comes into play, but if they did, it will be the same way that you could work with regular classes. So I think they are just two separate concepts. — unless I’m not understanding something.
CHUCK: Yeah, I just wanted to make sure that I understood that there was a difference between the two.
AVDI: Yeah, I just don’t find myself generating classes very often. It does happen, but much less often than I generate modules.
JOSH: Well, do you use Rspec?
AVDI: I do.
JOSH: You generate classes all the time then.
AVDI: No, Rspec does.
AVDI: What I’m saying is… well actually, Rspec I think does probably… well… I think they do a fair amount of module generation as well. But the point being, I’m not writing module generation or rather class generation that often.
GREG: Well it’s almost always that use case where you have some sort of program, where you wanna have some domain language and you basically wanna hide the class creation; so, Rspec does that if like I got some helpers that I use on top of Minitest that when you say contacts actually creates a sub class of the test case class — that sort of thing. But it’s typically just used for that. So I mean, normally, dynamic class generation is just syntactic sugar, is what I’ve seen it used.
AVDI: Yeah. Plus even in cases like that, sometimes I´ll do something along the lines of generate a module and then create a generic object and stick my module onto it. Just because, that gives me… I can then take that module generation, I can do something else with it as well, I can throw it into something and test it or something.
JOSH: David, you’ve been playing with Smalltalk lately?
JOSH: Have you discovered ThingLab?
DAVID: No, I have not.
JOSH: ThingLab was the first really mind blowing thing I ever saw done in Smalltalk and this was back in the mid-80s, giving you a hint about how out of date I am about all these stuff. [Chuckles] ThingLab was a simulation environment built in Smalltalk. It’s really worth looking up and reading about because it was so awesome. Randy Smith was the guy behind that. It was an object-oriented simulation environment and you can do physics type simulations in it, and one of their classic simulations was “walking in the rain” where you would see… if you walk slower or faster, would you get wetter or…
JOSH: And the way that they did this was you would create objects through some UI and okay, I got a box and I give it some velocity and all that. And as you did things and added behavior to the simulation, it was generating new classes and generating methods within those classes to embody the simulation.
JOSH: So that was an incredibly great use of metaprogramming. And it took a lot of work for them to do it given the primitive nature of metaprogramming that was available in Smalltalk.
DAVID: Yeah. So I have a kind of a follow up question from that. We’ve talked a little bit about when not to metaprogram, and we’ve talked about how to metaprogram. I kind of wanna throw out the question of why? Why would you metaprogram? What cases would you do it? I know why I monkey patch, I know why I do it, but I’m curious to ask you guys when do you metaprogram?
GREG: That’s a really good question. I think that’s one that like that’s… the one that people fail to ask themselves before they actually make use of metaprogramming, that leads to bad metaprogramming.
JOSH: I have one answer for that, that is sort of from a language implementer’s point of view, and that’s the virtual machine or the runtime implementation of a language, is optimized to perform certain kind of operations really well. So, like method lookup in an object oriented language is this really heavily optimized thing because it’s at the heart of everything that you are doing. And it’s pretty much just a like a case statement. So that’s a really great thing to do if you can generate code that takes advantage of the language’s implicit to really fast method dispatch, then you win. And that’s a lot faster than building data structures within the language and using case statements or conditionals. So if you have something like the ThingLab, where you are building simulation and there’s a couple things that might happen, building those things as methods on an object or a class that you generated and letting the language just go fast at that, that’s great, as opposed to cobbling together a whole bunch of data structures and populating them with data and then….
DAVID: Well, actually, my very first thought with that was that there is a data structure pattern that I use frequently which is building a hash of function pointers, to get away from the case statements. Basically to replace a case statement with a hash lookup.
JOSH: So why don’t you generate a class that has…
DAVID: Right, exactly. Does anybody else have [inaudible] for the why and when?
GREG: No, I think the example that Josh gave is a really good one, but I think that that’s not the majority of what people use metaprogramming for. What I see people using it for primarily is to reduce repetition or to give syntactic sugar to their APIs. And In those cases, the thing that bothers me the most is that sometimes people start with the idea of what they want their domain language to look like, or what they want to reduce the repetition or they can envision at some point, there will be repetition because there’s going to be lots of methods with similar definitions and things like that. And they go straight away to write in some sort of metaprogrammed code. But for those cases, these are the cases in which there is not… the primary advantage is just making it so that you can write code more succinctly. In those cases, metaprogramming should always be a refactoring stuff, not an initial design step for two reason: one, even if you make something that is this nice dynamic API, it’s whenever it makes sense to do so, it’s nice to have a static symbol API underneath it, that you build the metaprogramming on top of. And the other reason is that, often times, your metaprogramming will be shorter but much, much harder to understand. And so the maintainability argument has to be weighed out. You can say it’s the last code and it’s less likely that it will break here and there. But if code is really trivial, then you are not really introducing new possibilities of having bugs by just explicitly writing it out statically. So I think it’s one of those things where that needs to be a tradeoff. And sometimes people don’t take that into account.
JAMES: I think I really agree with Greg there that i do try to always to do it as a refactoring step and I’m we are all used to like an extract method refactoring where do the same transform in a few places, we see that code is totally identical and we are like, “Oh yeah, that’s extract method.” We pull it out into a method, put that chunk of code there and then we re-used it in all those places, right? Metaprogramming for me is really similar in that, I see myself using the same pattern, but it’s not quite extract method in that, there’s some complex challenges in there where it’s going to have to take it’s queues from where it is, or something like that. And then once I see myself doing that, then I will exchange extract and do some metaprogramming to intelligently do what I intend in each. But like Greg, I would much prefer to do it as a refactoring, because I need to see the pain that it’s saving me, and I find that that helps me do it in the correct way. Whereas if I start from the other side, if I just say, “Oh, I know I’m going to have this repetitional over the place when I do that,” that may even be true, but I don’t know exactly what it’s going to look like very time yet. So if I tried to think at that way, it’s very likely I do it in a non-optimal way.
GREG: Thinking about introducing some metaprogramming, the question that I ask myself is when the user interacts with whatever the thing I created does, is it going to conceal more questions or more complicated questions than it’s going to raise. Because every time you see dynamic code, it’s going to raise some questions, it’s going to increase the documentation barrier. And the question is, “Is the use of this going to make it so that a regular user can intuit or quickly understand my API faster than they would have to understand if I do do it that way?”
DAVID: Right. And what I found is that people… I’m actually glad that you guys said those things because I was worried when you said ‘syntactic sugar’, that we are going to have a fight in our hands, because my reason for monkey patching is almost always, I do it whenever I cannot cleanly do or cannot cleanly express what I want to express in the given syntax that I’ve already been given. But I do it the same way that you guys say; it’s a refactoring step. If you start out with the monkey patch, if you start out with the metaprogram, then instead of having this crappy repetition all through your code, you have this crappy monkey patch all through your code, because it didn’t come out organically from it. I was going to add, i think it was Avdi who asked about rules for metaprogramming, and I wanted to throw on a couple of other ones that I use rigorously which are if you’re going to monkey something, if you are actually going to put a method on an object, the first thing you should do is test to make sure a method isn’t already there because there’s…
AVDI: Then you are not monkey patching..
AVDI: If the method isn’t already there, you are not monkey patching.
JAMES: That’s using Ruby’s open classes instead.
DAVID: Yeah… AVDI: That’s just using Ruby the way it’s intended.
DAVID: Yeah, well okay, so in 1.8.6 index, the array index method did not take a block. So if you wanted to say, “Give me the index of the thing.” “Here is a block describing the thing that I want,” you had to do a three line two step and call detect which gave you the position or which you give you the object and then you had to call index on that thing and then you had to return the index for that thing. And I patched… I rewrote array index to take a block. And if no block was given, I call to the original index. And if it was given, then it did the whole two step of detect in index. And I mentioned that… I posted that on my live journal, and I mean I just got flamed just off the face of the planet for daring to monkey patch…
AVDI: I was probably one of the ones flaming you.
DAVID: Yeah, [Chuckles] and I wrote a week later, I posted back to the blog, because I actually wrote the title of the Live Journal post was, “Is this monkey patch good or evil?” And people came in and said, “That is totally evil.” And I said, “Okay, well hang on. The first thing I did after I wrote this is I wrote a test to see can I call index and pass it a block and get a meaningful result?” “If I can, don’t inject this patch.” And sure enough, when 1.8.7 amount and patched it, my monkey patch just silently deactivated itself. The second thing I did is I gathered the entire team together and said, “Guys, look at this. Be aware of this.” And we put it in a patch that we had a policy that if you patch anything, you have to put it in lib patches, and it had to be in the name of the class. So lib patches array.rb basically, so that people know, “Oh, somebody is monkey patching array. We need to see what is going on in there.” And the communication, letting people know that it’s there. So two weeks later, I wrote another journal post that basically said, “It’s good, you can kiss my butt because they had just released…” literally two weeks later the my boss called me and he said, “You spent how many days working on that index with the block? Because it’s in the standard library,” and I’m like, “No, it’s not.” And he pulled up the 1.8.7 documentation and said, “No, they just added it. Check it out.” And so, two weeks after I wrote the patch… I’m not claiming that I inspired anybody or there was any kind of causation, a precedes b, therefore a caused b not necessarily true. But certainly, there was some simultaneous parallel evolution going on. So I felt pretty good with that.
GREG: That was probably one of the other 500 million methods that 1.8.7 added.
DAVID: Well yeah, they added it and then they take it out, and then they added it back and then they took it out.
AVDI: Did you create a module for that?
DAVID: No, I didn’t.
GREG: That’s the thing I was going to say. There is two points Dave, about what you’ve suggested. One, a respond to check or something like that is almost always insufficient because that one…
DAVID: Yeah, my test actually passed a block to index and in 1.8.6, if it is unpatched, you pass the block and it accepts the block and does nothing with it. And so it was really [inaudible] to write a valid test.
GREG: So it’s very hard to actually check and adequately check for that sort of thing, that’s why I support it.
AVDI: There was an issue with the [unintelligible] method in Rails that gave me and a lot of people…
GREG: [inaudible] patching myself for like back ports from 1.9 just to limit the amount of back ports that I needed to do, but I found that if you don’t get the signature exactly right on things — and several libraries do that — then you may end up getting someone else’s back port that isn’t exactly compatible with yours. And it’s very hard to know exactly what the signature of these methods are. I mean, Ruby still isn’t is the world’s best documented language.
DAVID: So that actually raises two more points — and I know Chuck is going to start cutting us off here pretty quick — but the… when you talk about monkey patching, people start freaking out and saying, “If we start mixing matching things, then things are going to break in weird ways,” and I keep just looking at people and saying, “It doesn’t happen.” Okay, I mean, it does happen, it happens about once a year, right? You combine, you pull down bundler and you combine it with ruport or something. I’m not saying bundler and ruport conflict, but I mean, you pull that two libraries, you find out, no, they don’t work together or you pull down the Beanstalk client and you pull down… simple common JRuby, you pull those down, that is not monkey patching, that’s just a SimpleCov has it depends on coverage which hasn’t been ported on JRuby yet. This happens once a year and it’s not… the errors don’t happen in like impossible, well they do happen in kind of impossible to find places… but you sure are going to give up and you find another way to do it. The second thing I was going to say is I did not do it using a module because — and I’m going to confess some ignorance here — I’d never thought about doing that until you just mentioned it just now.
GREG: The module gives you… if you say you create an array and then you mix in whatever the hell you want to, as long as it [inaudible] it, now you can go to [inaudible] on it and you get the same visual behavior. It was basically the same as the monkey patch, but it’s localized the effects greatly. And even if you, leak those arrays out so that your arrays will pop out or your special arrays. Even if you pollute the arrays that have passed in, at least it is limited to your library. It’s isolated to interactions with your library.
AVDI: And there’s another advantage, people can say can inspect that array, and they can see that it has your patch… my array patches in the ancestor chain. Whereas you cannot see that of you just reopen object or reopen array.
DAVID: It’s even better than that in 1.9 because you can say ‘object.method’ and give it the symbol and it will come back and it will say, “Dave’s weird ass module line 89”.
CHUCK: All right, we are 46, almost 47 minutes in and probably should get into the picks.
DAVID: Let’s go for two more minutes because I don’t have a pick today.
JOSH: Okay, I have something really concrete that I wanna talk about, and that’s the do’s and don’it’s. So class_eval def versus define_method.
AVDI: Yeah, define_method.
GREG: Define_method, unless you are running a framework and really care about the performance hit that using define_method is going to give you.
GREG: Which is almost no one. If you are not building Rails, you don’t need to care about that.
JAMES: Yeah, I generally prefer define_method in that particular case. Although I do use class evals sometimes, like if I’m writing active record module or plugin, and I wanna do some things like declare a bunch of validations or some associations or things like that, I´ll go ahead and just do a class eval so I can open up that class and just pretend like I’m inside of the model itself and program normally.
DAVID: Is define_method actually slower? I thought class eval was the one where you gave it a big ass string and it had to parse through the whole string and eval it every time. But define_method, you gave it like a lambda.
GREG: Eval is slower on its own because it has to fire up the parser…
JAMES: And that only happens once…
GREG: [inaudible] on every single call because it has to pull in the closure that define_method provides you. Where eval, once something is evaled, works pretty much like you typed it in a file.
DAVID: Okay, so I’m going to change my stance from, “always one or always the other” to “it’s a tradeoff,” and we’ve just discussed what the tradeoffs are.
JOSH: There is also potential for memory leaks when you are using define_method.
JAMES: That’s true.
DAVID: The closure can leak, yeah.
JAMES: Right. So what were your other questions, josh?
JOSH: No, that was the one concrete issue is just do we wanna do… what do we recommend people do? Class_eval def or define_method? I tend to stay away from…
GREG: One very obscure thing that I just wanna mention. We don’t need to discuss it. If you are working with objects def find finalizer, that can be very scary and almost certainly leads to bugs, unless you know exactly its Idiosyncricities.
JAMES: Going with the define_method issue, it used to be a much bigger problem because blocks couldn’t take blocks, right? So that complicated that a lot.
AVDI: Yes, it did.
JAMES: But that’s mostly gone now.
DAVID: So I have one last hard and fast rule to throw out. If I may quote from the gospel of beck, which would be the Smalltalk Best Practice Patterns book from the 1990s — remember this book is 15 years old — he actually, in one of the chapters, he talks about convenience constructors, which is where you basically monkey patch a class, some other class to return an instance of your class. Like how integer gets the monkey patched to with the at method to return a point. So you call 3@3 and that gives you a point by basically calling a method on integer. And he gives a rule at the very end. He says, “In a medium sized project, you should not do this more than three times.”
DAVID: Yeah, it’s worth thinking about. Like if you’ve monkey patched 58 things in a 1,000 lines of Ruby, you are patching too many things.
JAMES: That’s a lot like DHH’s rule of, once there’s 13 things in the list or something like that, then it’s too many.
JAMES: Interesting. Well, we should probably turn it over, but I think the moral of today’s lesson was modules are good…
DAVID: And be responsible children.
CHUCK: Yeah, it’s definitely something that you can cause not only problems for yourself, but for others if you are being reckless with this. But yeah, modules are awesome.
CHUCK: All right. So I am going to take over now and push us towards the picks. For those of you who are new to the podcast, basically, picks are just cool stuff that we’ve run across or have experienced. A lot of times they are related to code and writing code, but sometimes they are just stuff. I think we had like three or two or three TV shows or movies or something as picks in the last one, but we also had some great libraries and some stuff recommended as well. So we’ll go ahead and start off with Avdi.
AVDI: All right, so I think I’ve got two. First of all, as a consultant, I find myself working on a lot of different projects and often, sometimes it’s not convenient to have all those projects sort of running together on the same machine; because a lot of times, projects will have little changes that they wanna make to the machine, like maybe they wanna set up certain DNS aliases or something like that or specific MySQL configuration or something like that. So, something I’ve been playing with lately and I’m hoping to use more and more is something called Vagrant. And Vagrant is basically some scripts on top… a project that uses virtual box and provisioning tools like Puppets and Chef to basically make it very easy to spin up a new VM for your development environment. And then I basically, in three moments in the last couple of picks, I’ve been putting together sort of my ideal starting development environment with an eye towards being able to just spin up a new one every time I start a new project and then it will be completely isolated, and it won’t touch any objects that I have, and the other benefits of using VMs for stuff like that. It’s very easy to then… if you bring somebody new, it’s very easy to give them a VM and say, “Hey, just work on this. Don’t worry about the whole long setup process.” So, Vagrant is very cool. Something else, and I don’t know if this has been mentioned. Well, I mentioned this at the beginning of the show. I don’t know if it has been mentioned as a pick, but for a while, I’ve been slowly reading my way through the Best of Ruby Quiz, which was edited and I guess a lot of it was written by James Edward Gray. And it’s just a great collection of Ruby idioms, and like different people’s style of solving different problems. And a lot of times, the topic comes up, where do I go to look at… to read other people’s code, to learn better styles or different styles. And this is just a great curated collection of other people’s code that you can browse and learn from. CHUCK: All right, those are both terrific picks. And yes, so go check them out. We’ll have links at RubyRogues.com all right, David go ahead.
DAVID: so i wasn’t kidding when I said I didn’t have a pick. I do have an unpick today. I don’t wanna start anything… I don’t wanna start some crap here, but I have an unpick for today, which to say that I’ve been doing a lot of remote pair programming and the tool of choice that we use for that are Skype and Team Viewer Pro. And my unpick for today is Team Viewer. I love you guys, Team Viewer. I love you like sliced bread, I love you like cooked food. But the pro version is absolutely… it’s over $700 per seat. It is absolutely unattainable by anybody on any kind of a single contract or developer budget. So I’m throwing this out to the audience, if there is a good screen sharing thing that’s easy to set up, easy to use, allows people to switch sides, allows mutual control, please let me know. I’m @dbrady at twitter or email@example.com. I would love to hear if there are some better solutions. Team Viewers guys, I love you. If you could tell me how to get a group discount, we can get this thing down to like $100/copy, I know I can find a whole bunch of people that would give you some business.
DAVID: That’s it.
CHUCK: All right. Thanks Dave. Greg?
GREG: So, I’ve been using IRC cloud for a couple of weeks. And it’s okay, the main reason why I was using is it is basically a web based IRC client that persist your connection, so that you can see your messages from when you’re away and all that sort of stuff. And it has some other features, but I recently stopped using it because I find their beta to be kind of buggy, and also because like about a year ago, I tried to get away from doing… I used to do everything in the web browser, and I sort of went to the other direction. I find that specific application… [dog barking] oh god, my dog is barking.
GREG: Yeah, hold on one second. [Dog barking] Hey!
JAMES: Greg’s dog disagrees with his pick.
CHUCK: Yeah, absolutely.
GREG: So I’ve been using site specific applications which means that I would prefer to do IRC in my own app. [Dog barking] God that dog is killing me. Sorry. [Chuckles] [Laughter] So, basically, I would much rather use IRC on it’s either on the command line application, graphical application, doesn’t matter. What I want it to have, which I always thought would be hard is just to be able to have messages when I wasn’t there and just sort of keep persisting connection. I found that using GNU screen, plus basically any command line IRC client will do that for you, you can just put it on just any where you got a shell account and it just works great. And it didn’t require any configuration; we had basically typing screen and then typing IRSSI. And it’s totally worked well. And I mean, I think IRC cloud is probably well worth it for someone who wants a web-based irc experience, but if you were just looking for that functionality, you can get it for free — fairly reliably — just by using Screen and any commend line IRC client. And a lot of those will allow you to proxy as well so you could use like whatever your favorite client is going to connect to a running instance of it. So that’s one pick. The other one that I have which is completely non-technical is that I think that people should check out square foot gardening if they have it. I’ve been learning that for a couple of months now, and it’s awesome. Basically, it’s a way that you can grow a whole bunch of stuff in tiny 4×4 boxes, the initial setup of it is a little bit expensive, but you have to use sort of a complex blend of soil but, once you do that, because the boxes are lined at the bottom, you get no weeds. And basically all I need to do is water my plants a couple of times a week. And I’ve been eating vegetables from my garden every day for the last few weeks. It was very minimal work — and it’s fun.
DAVID: That’s great.
CHUCK: That’s awesome. All right, I just got to spent a lot of time on my garden. All right, James go ahead.
JAMES: So I’m going to go with an unusual pick this time, as far as a technical pick, but I had somebody email me recently and say, “I’ve been trying to learn Ruby, I’ve gotten so far but I’m kind of stuck on some things.” “Would you be willing to just come and let me come over once every couple of weeks and show me some stuff.” And I hadn’t done any mentoring in a while, so I said, “Sure, why not. Let’s do it.” And we’ve been doing that and for a bit now, and I’m going to do it right after this podcast. And I’m enjoying it. I’m learning a lot, and I’m remembering how much fun it is to mentor somebody, and they are getting a hang on things they were stuck with. I’ve forgotten how valuable mentoring is, and I think it’s really wonderful in our community. So my advice is if you are one a low level and you would like to gain some knowledge, ask somebody who has it that you could get it from. A lot of us don’t mind teaching and helping you over a hump or something. And usually, we learn things from doing it too; I learn things about areas where we are doing a bad job of teaching certain things or what kinds of help people need. I try to find new ways to explain things, which helps me when I give conference talks or do trainings or whatever. So I think it’s helpful for both sides and everybody learns a lot. So ask somebody, or if you are at a higher level, maybe offer services for somebody, I think it’s a good thing that you can do for our community. So that’s my first recommendation; mentoring, do it or get involved with it. I think it is valuable. And then as far as books, if there’s any fantasy readers out there, I hope you found George RR Martin by now. If you haven’t, you need to. That’s his main series the Song by Ice and Fire, and starts with the book called The Game of Thrones. The 5th book in the series just came out, Dance with Dragons, and I just got it in the mail, so I admit I’ve not read it yet, but I absolutely love that series. So if you are in to great fantasy, I think you can’t go wrong with the Song of Ice and Fire. And I just do need to say that it’s a very adult series, so if that kind of thing bothers you, there’s a lot of things in it that could definitely be taken as objectionable. And you stand warned, but if that doesn’t scare you off, then go read it. You’ll love it. And Greg is telling me that mentoring should talk at the RMU, people on Freenode, which is of course, that’s exactly what RMU does. So, that’s great point. You should definitely go hang out and #rmu on Freenode. That’s a great place to find mentors or become a mentor, so yeah, good stuff.
DAVID: Greg is RMU on Freenode a good place for somebody who is not an RMU alumnus, but a fairly senior development? Would that be a good thing to come in there and hang out and chill out and help people?
GREG: Absolutely. We actually have a lot of people who hang out in that room from both sides; people who are looking to learn and people who are willing to help, because they know that that’s what our program is all about. So we have this sort of extended network. And the thing that I just sort of ping James about over I am was that we are gradually extending our mentoring program, so that we can start taking people in. And doing like hour long sessions or something like that instead of our crazy, 20-hour a week, three week long course that we typically put people through. And having people around just in the room, would be very helpful. I mean, it is worth keeping in mind that like we have specificals in mind, so it’s a good idea to lurk a little bit and see how they work, or just talk to me. But at the same time, we’ve got some great people from the Ruby community who aren’t directly involved in RMU helping out with our students. And as long as we keep it at a good environment for learning, I would definitely welcome people to do that.
DAVID: Yeah, because I’ve hung out with like Brandon Hayes from RMU and I’ve given them tips and pointers, but I’ve always had to be kind of circumspect and say, “You know, I’m not part of the curriculum, so I might be giving you bad advice.”
GREG: [Chuckles] Yeah and the RMU channel in general, I think that like people who are just interested in helping others out, they are more than welcome to come. I mean, you may just might want to check in with us and see what we are doing or whatever. But other than that, then yeah, absolutely,
CHUCK: All right. Sounds good. And it’s always good to have another place where you can get help. Josh, go ahead.
JOSH: So my first pick is… let’s see… if you are listening to this on Friday, the coming weekend is the Rails 3.1. Hack Fest. And this is organized by some Rails Core Team people, and it’s going to be a weekend of doing a lot of focus development effort around the 3.1 release. So there’s a post on weblog.rubyonrails.org about it. And there are a lot of places where there are places where you can go and hang out with people. I see a list of like 20 or 30 right here. Here in San Francisco, both Heroku and MochaLeaf are going to be hosting places. Pivotal Labs is doing it in Boulder, Shopify in Ottawa, Pivotal Labs in New York — all over the place. You have in Chicago. So, it’s going to a lot of fun. And all weekend, people are going to be porting their apps to Rails 3.1 and working on fixing bugs, and working on documentation if you are not comfortable working on the Rails 3.1 code base. If enough people get involved and contribute, then 3.1 release should probably come pretty soon.
CHUCK: That sounds terrific.
JOSH: Yeah, so that is one pick. And then, my other pick is since you are listening to this on a podcast, you may appreciate knowing about another podcast which is Escape Pod. And this my favorite science fiction podcast. This has been going on for years, and they have so many stories that they podcast. It’s just free and they have some awesome stories. It is started by Steve Eley, and he did just a great job for years and then he turned it over so other people can be more focused on it. I think Steve is now doing Ruby on rails development somewhere, so maybe he is even listening to this podcast now. So, Hi Steve! Thanks for all the great work on Escape Pod. So yeah, go check it out. It’s on iTunes. Escapepod.org is the URL for the site. They even have some sister podcast like pseudo pod and podcast all for different genres.
AVDI: I have to second that. That’s a great one.
JOSH: Woohoo! First. [Chuckles]
CHUCK: [Chuckles] David Brady.
DAVID: Use modules, kids.
CHUCK: Avdi Grimm.
AVDI: It’s been great.
CHUCK: Gregory Brown.
GREG: No problem.
CHUCK: James Edward Gray.
JAMES: It was geeksplosion this week. [Chuckles]
CHUCK: It was. [Chuckles]
DAVID: There’s geek all over the walls.
CHUCK: Yeah, and I’m Charles Max Wood. Hopefully next week, we can come back, we’ll be talking about something interesting. We don’t have a topic pick for next week, so we’ll have to surprise you, but anyway, thanks for listening and we’ll catch you next week. You can get the show notes at rubyrogues.com, you can find us in iTunes and go ahead and leave us a review there, and probably I’m going to try and get us in to other podcast aggregators. So, if you’ve been pulling the feed down onto your Zune or something, then hopefully we can get listed there are pretty quick. But that’s it, so we will catch you next week and thanks for listening.