050

050 iPhreaks Show – HTTP APIs


The panelists discuss HTTP APIs.

This episode is sponsored by

comments powered by Disqus

TRANSCRIPT

CHUCK: My calendar says it’s episode 50. PETE: Wow, the big 5-0. CHUCK: Yeah. PETE: What’s that called? Is that a half centennial? It’s not a bicentennial. CHUCK: I don’t know.CHUCK: Hey everybody and welcome to episode 50 of the iPhreaks Show. This week on our panel we have Andrew Madsen. ANDREW: Hi, from Salt Lake City. CHUCK: Jaim Zuber. JAIM: For the 50th episode, I’ve got balloons everywhere. You can’t see because it’s a podcast, but they’re there. CHUCK: There you go. Pete Hodgson. PETE: Good morning from the city that knows how. CHUCK: I’m Charles Max Wood from DevChat.tv, and this week we’re going to be talking about web APIs with your iOS app. This is something that I’m particularly interested in since I tend to do more of the web API end of things as opposed to the objective-C, iPhone app, end of things. I can complain about some of the stuff that people do just because I know it’s wrong, and you guys can also talk about some of the stuff that people do but you wish they didn’t. Maybe I’ll learn something. PETE: So it’s kinda interesting, just before the call someone – I can’t remember who said it – said something about talking to the server. Jaim was talking about how to talk to your server team and Chuck, you just kind of touched on that as well. Kind of like, there’s things that the server team wish that the iOS guys would do, and there’s things that the iOS guys, the client-side team, wish that the server side guys would do. So this is an interesting application of this thing called Conway’s Law. Have you ever heard of Conway’s Law? CHUCK: Yeah, that the structure of the program will basically imitate the communication structure of the team or teams that are working on it. PETE: Yeah, exactly. That’s a very good, that’s a much better description and I’ve only come up – I’ve always struggled to describe it. So I always think this is very interesting, right? Like if your server team has a hard time – if your back end guys don’t have a good communication channel with your front end guys, then your back end code tends to not have a good communication channel with your front end code. CHUCK: Yeah, I have to say that typically on the teams that I’ve worked on that we’ve had to expose an API for, whether it’s like a front end JavaScript or an iOS app, usually what happens is the front end team will say something to the effect of ‘we need an API that give us this’ and then the back end team will go implement it then tell them where it is. Does that more or less reflect your experience, or have you seen it work other ways? PETE: I’ve got an experience of this – few different things, depends on the type of the organization. I’ve worked in little startup environments where there isn’t a server team and a front end team – there’s just a team, and they tend to be oriented more around functionality. So you might have the photo browsing team and they’re responsible for the photo browsing capabilities of the app, and that encompasses the back end stuff and the front end stuff. And that’s kind of – again, that’s applying Conway’s Law, or wielding Conway’s Law to make you more effective. So you shape the team vertically, then you tend to add really good, efficient communication across the stack in terms of the team, which means you tend to have good, efficient communication across the stack in terms of the actual APIs built. JAIM: Say your team’s organized vertically – what does that mean? PETE: If I had a whiteboard, I would draw a layer cake – kind of that classic layer cake that people draw. This is our data base here, and this is our back end, middle tier, and this is our client side tier or something like that. You normally draw this kind of horizontal rectangles, so when I say vertically-oriented teams, I mean they’re kind of, if you imagine you’ve got these layer cake rectangles. A team slices vertically through the entire stack so it owns the front end that relates to that functionality, plus the back end, plus the database, maybe. JAIM: Okay, so you’re working with people that understand what’s happening in the database, what’s happening on the web service, and understand what the client-iOS portion is. PETE: Yeah. Yeah, and a lot of times, that will also be, there’ll be some Android guys and some iOS guys working on the same team like cats and dogs. CHUCK: [Chuckles] PETE: And a lot of times when I tell people that, they don’t believe me. It’s like, “That’s not possible! Oil and water – they cannot mix!” Dfinitely. I’ve worked on teams that done that, and I actually think it’s pretty effective if you can pull it off. CHUCK: Do you ever run into problems with that kind of setup between the different teams needing a similar functionality, or sharing functionality? PETE: Sometimes what you have is that you have these vertically aligned teams, and then you have either some other support team that goes across the top that implements common things, like more infrastructure-y technical stuff. The other thing that I’ve seen, or I’ve heard of working pretty well – I haven’t actually seen it myself – is this model that’s Spotify talked about a lot where they have teams that are grouped around functionality, but then they have this communities of practice, or tribes I think they call them, where I might be the database guy for the photo browsing team and that’s my team, that’s the guys I work with everyday, the guys and gals I work with every day. But I’m also part of the database community of practice, so I’ll get together on a regular basis with my database buddies and we’ll talk about the problems that we’ve been solving in the photo browsing thing, and some other person from the login team will describe how they’re building some new caching thing that everyone else might be interested in using [inaudible]. I’m sure there’s a risk there that you end up reinventing the wheel, but I think it’s a good tradeoff in exchange for not having to, not having super ineffective –. I think everyone’s had that experience with being, where you have the server team builds an API that doesn’t work very well and then you want to change it but they don’t have time because they’re working on stuff for some other team. You lose so much of efficiency that way; I feel like it’s actually better to just orient yourselves vertically and do things that way. CHUCK: Yeah, one thing that I’m a little curious about with that though is that you make up the team, then up the people that have the database skills and the back end server skills and the front end server skills? Or do you kinda make everybody learn – to some degree anyway – all the different parts? PETE: Yeah. So I think it’s a bit of both. I think we already do this a lot with – nowadays, we do this a lot with database stuff. It used to be old school, app development. There was a DBA department – DataBase Administrator department – and if you needed to get anywhere near the database, let alone write a query, let alone insert anything into the database, just write a query that was touching the database, you had to go to the separate team and describe your requirement. Then they build something and throw out over the wall to you, and you’d say, “That’s not quite right” and you throw it over back to them. Which sounds really clunky and antiquated, but that was the same kind of thing we’re talking about with front end and back end. And now, what tends to happen, I think, with most teams is you have the teams that are responsible for doing database stuff – there tends to be one person on the team who happens to be more interested or more capable on writing efficient sequel statements of tuning indexes or whatever. So he or she will tend to be the person who’s like the expert in the team and who ends up writing a lot of those stories that are kind of database-focused, and you can translate this to iOS. It’d be like the person who’s really good at core animation, or the person who’s really good at core data tends to do a lot of those kind of bits of work. They’re like the expert, but it doesn’t mean that they’re the only person that’s allowed to work on the story. And they also might have – again, going back to this community of practice thing – they might have connections to experts on other teams who can help out in a pinch. So if there’s some super gnarly, core animation thing for example, they’ll kinda reach out to that core animation community of practice and say, “Hey, I can’t get this thing up to 60 fps, can you guys help me with this CA layer issue” or whatever, I don’t know. It’s a very, very long-winded way of saying, I think, yes you need a poly-skilled team, but it doesn’t mean every individual on the team needs to be an expert on everything. And it also doesn’t mean that you have to have all those skills in the team. You can also reach out to support people if you need to. JAIM: So I like the idea that you’re talking about, like a vertical team where [inaudible] data, server and [inaudible] people on the same team. One challenge I run into where if you’re doing something kind of Agile sprint-based, you do one-week, two-week sprints and you say, “Okay we’re gonna do this feature this sprint.” And your server team goes to work, and the client team doesn’t have anything to test with or even wire up until halfway through the sprint or near the end of the sprint. How do you deal with those types of challenges? PETE: That actually was what I was saying I was going to touch on earlier with this idea of how do you –. Chuck, you said something about the server side team built you this API and then it’s not quite right, and then you go through this period of – well they decide what the API should look like and then they’ll give it to you. There’s this approach called consumer-driven contracts, also sometimes called contract testing, and the idea there is the team that’s going to be consuming this API – so in our example here it would be the iOS team or the front end team or whatever – they actually define that API in terms of tests that describe what the API could do. So it’s kind of like using behavior-driven development – well it is using behavior-driven development, but rather than describing the UI and saying, ‘when I click on this thing, this error message should be displayed’ or ‘when I log in, but my credit is expired and I should be told that my credit is expired,’ you’re instead saying, ‘when I send this chunk of JSON to the server, then I expect this chunk of JSON to be returned.” That’s one way of doing it; it’s a pretty extreme approach. I think most people would be pretty skeptical that it can work, but I have seen it work. And it actually works really well when you’ve got those disconnected teams, because it’s a way for you to communicate with code unambiguously, rather than communicating via wiki pages, or excel spreadsheets describing API. CHUCK: One other thing that occurs to me is that if you follow some sort of standard across the team or across all the teams – for example, we’re going to use REST and we’re going to do this or that for the things that don’t fall neatly under however we define or understand REST, that makes things a lot simpler too because the back end team can move ahead knowing that they’re probably going to get something similar to what the front end team is going to request, and it streamlines a lot of that communication. So you can actually write test harnesses that basically implement the test around REST, and then that saves the front end team time in specifying the APIs, and the back end team can also use standard libraries to do that on the other end. PETE: Yeah, and there’s a few standards that are emerging out there for JSON through RESTful JSON APIs. There’s this standard called HAL, [inaudible] called how; there’s another one called SIREN; there’s a few of them out there that kind of defined how you – kind of a common way of structuring JSON APIs, and I think that really helps because you can build client-side libraries around it and server-side libraries around it. I don’t know if there are any for iOS, but there definitely are in other languages, kind of standard consumers for things like HAL and SIREN. JAIM: So Pete, before we get into the stuff like HAL and SIREN – I understand HAL. Generally, we should probably define what we’re talking about when we talk about REST. [Inaudible] anything from having resource-based web servers to going kind of a more hypermedia approach, hideous hypertext, as [inaudible] of application state. Can we talk a little bit about that, like what the difference is when people say REST? It’s kind of a conflated term – you’re not sure what they’re actually talking about. [Crosstalk] PETE: It’s like Web 2.0 back in the day, right? It doesn’t actually mean anything at this point, more like DevOps. It’s a term that has been so overused that now doesn’t mean much at all. It means not [inaudible], and sometimes it means not XML although you can do XML. So my definition of REST – there’s a guy called Sam Richardson, I think. He wrote this really good book – RESTful Web Services, I think it’s called. I have to find the book and Post it in the show notes. He described these levels of maturity; the basic form of REST that I think most people kind of agree on is this idea of you’re Posting and sending and receiving documents, JSON-based documents, using URIs. You’re using some kind of shared document structure against known URIs – so it’s like the basic stuff. That’s sometimes called POX or POJ, like plain old XML or plain old JSON. You know what? I should actually look up these levels because I’m trying to remember what they are but I’m not [inaudible]. Basically, that’s the basic stuff and that’s what a lot of people call REST. A lot of people who would – REST docs, RESTifarians – get really upset when these people call this stuff REST because it’s not really REST, right, according to them. So the next level up from that is you start using the different HTTP – I think this is the next level up; if get it wrong, then it’s not like I’m being recorded on a podcast –.. CHUCK: Yeah, no one will complain. PETE: Yeah. So, if I remember correctly, the next level up from that is using the verbs that are in HTTP, and using them semantically. So when you get something, it shouldn’t have any effects on the thing that you’re getting. So if I – if I do a get on /user/pete, then that should modify that resource, that kind of thing. Now that sounds kind of obvious, but there’s loads of APIs out there where you do get/user/pete?changepassword=newpassword or something. That violates a bunch of stuff that the web assumes is gonna happen, and it happens that you lose a lot of really nice things that REST gives you, like free caching and the ability to retry requests without worrying about them. JAIM: I don’t think that’s obvious at all, if you have been doing this type of development for a while. I think a lot of us – we’re doing object-oriented programming; we’re used to calling methods and objects, and one of the things that they talked about in that book is that’s how we started doing web services. We call a method on a web service, and that’s how we distribute things around. So that’s just how everyone did it – go do this on this web service. So it’s pretty – it’s a very different mindset than what they’re talking about with RESTful. We’re using kind of the architecture of the web, HTTP stuff, saying, ‘don’t call a bunch of methods on this object; this is a resource, and one of the things we can do, we can get, but getting will not modify it at all.’ So I can call get 20 times, it’s not going to modify what you’re actually doing. PETE: I guess you’re right. I’ve been doing REST-based stuff for long enough; I kinda forget how it is, like there is this whole other mindset that a lot of people are used to of essentially the web as like a big remote procedure call kind of thing where you’re just calling a method on an object, it just happens to be somewhere else not next to you. JAIM: I mean, I had to go through that book at least twice before I started to actually kinda get it. PETE: Yeah. Yes, I mean, and it’s definitely true. This idea of thinking of – I guess, maybe a catchy way of describing it is REST is about nouns rather than verbs, so in our world, we’re very used to thinking [inaudible] programming, in general. We’re very used to thinking about things in terms of actions that we’re taking, so, login or change password. With REST, the way you get the real value out of building a REST-best API is by flipping your thinking, so rather than focusing on the action, you’re focusing on the resources, the things that you’re acting upon. So you’re primary thing is the user, and then maybe you might be changing the password on the user, but fundamentally, you’re acting upon the user. You’re not performing an action, you’re performing an action against a resource, a known kind of entity. JAIM: Okay, we talked a little bit about the get – that’s one of the HTTP verbs, but you’re talking about something that changes state, like a password. So you have a user resource that you do a get, which would get you your information – how do we actually change the password? How does that work? PETE: I think with all things REST-based, it depends on how dogmatic and RESTifarian you wanna be about it. The way that I would do this with most things is with a Post. A Post is basically the sledgehammer of HTTP. You can use it for everything; it doesn’t have any semantics, so it doesn’t – you don’t have – like the thing I was saying with get where you have all the things that you touched on as well would get, kind of you can get something 500 times and you’ll always get the same result, assuming nothing’s changed elsewhere in the system. Post doesn’t have any of those rules: you can do whatever you want to the Post, do it all better off, you’re allowed to do whatever you want. So that makes it really useful; that also means that it doesn’t have as much value because all the stuff in the middle doesn’t know whether you’re changing something or not, so you lose some benefits from using something like Post, but Post is the sledgehammer that you can use for all these kinds of stuff. You could say, you could have a resource/user/pete, and then if you wanna change the password, you could just Post the new password to /user/pete/changepassword, for example. So probably some REST people are freaking out right now because I’m not using the right verb, but I’m a lot more pragmatic in my RESTifarianism than dogmatic, I guess. JAIM: Yeah, it’s a good point to remember that if you need to call a method on the external object and that old kind of thinking, you definiltey do it with a Post. PETE: Yeah. JAIM: So if you want to call a change of password and Post that, that’s the way you do with that kind of things. PETE: Yup. The other two verbs that are there that would be another option is Put and Patch. You think of Post as kind of ‘do anything,’ but generally, it means ‘change something.’ Put is like ‘update all the things.’ So let’s say, if I wanted to update user/pete, so I wanted to change my mailing address. One way that I could model that is I get that resource – so I do a get on /user/pete – and I get a whole representation of that user in JSON form or whatever, and then I modify the things that I wanna change, so the email address or the password or whatever. And then I put that entire chunk of JSON, that document, that thing that represents that user – I put it back to the same URL. So that’s the other way of doing it. Now the problem is you have to – you don’t get to just put – with the Put, you don’t just get to say, just Put the password. If I just Put the password, then I’d essentially be saying, ‘delete everything else about this user and just-’ it’s like an overwrite. It’s not updating in place; it’s like overwriting whatever’s in that current resource. And the nice thing about that is, because you’re specifying the whole state of the resource, you could send a Put, and if you’re not sure if it went through or not, you can just send it again, because you’re not specifying a change; you’re specifying the new state of the world. So, a really, really obvious kind of example here that helps explain it with this Post versus Put thing is, let’s say I’m modeling bank accounts – so this is the old, hackneyed example – and I’ve got a balance on my bank account. If I was taking $20 from Chuck and giving $20 to Pete, I might model that by sending a Post to Chuck, saying, ‘debit the account by $20,’ and then sending a Post to Pete, saying, ‘credit the account by $20.’ Now, if I send that Post – and I’m not sure if it goes through or not – what do I do? If I’m not sure it went through, but it actually did go through, and I send it again, then I’ve given Pete 20 more dollars than I was supposed to. So you get all these problems of not knowing whether an operation’s completed or not, because that Post isn’t [inaudible]; you can’t do it multiple times and have the same effect. If instead, rather than saying ‘give Pete $20,’ you said, ‘Pete’s current balance is $80, so set his current balance to $100 using a Put.’ If that Put fails, I just send it again, because it’s identical; I can do the same thing over and over again and have the same effect. That’s a big benefit of using something like Put over something like Post. ANDREW: And so, you also mentioned Patch? PETE: Yeah. So Patch, it gets around this problem that I mentioned earlier was that, like if I wanna modify one field of my account or the user or whatever, I have to download the whole object, or the whole resource, and then modify it [inaudible] JSON and send the whole thing back up because I’m overwriting the whole record, the whole resource. Patch is like that, but it’s like an update in place rather than overwrite everything, so I can just send the field I wanna update, and it kinda makes a Patch, kind of like a Patch on source control. It just modifies the stuff that’s specified. When I send the Patch, that just says my username also is just as my email address and just as my password, it’ll know other fields; it will change those fields in the resource, but it won’t change any of the other fields. So that’s the way you can avoid having to download everything and then re-upload it again. JAIM: So that’s very cool. I remember a couple of years ago when I was looking at this same problem like you’d have a big entity and you just wanted to change one thing. All the RESTifarians said, “Hey, do Patch” and I looked at the client libraries and support was spotty, at least back then. Most client libraries can give you a Get, Put, Post – that type of stuff. How is the library support for Patch? PETE: I actually have no idea. It’s interesting, with the Rails world, for a long time when folks were first doing REST in Rails, even if the Put wasn’t very well-supported. And the way that they got around that with Rails was kind of basically doing a Post with this special magical indicator that said, “Hey, this is really a Put.” Even if your library doesn’t support Patch, you can do the same thing, and you’re still getting most of the value. The RESTifarians will scream at you because you’re not honoring the one, true way of whatever, but if Patch isn’t available or you’re not sure it’s going to be available, just use Post and use some kind of scheme to indicate that what you’re doing is actually a Patch. You lose some of the benefits, so all of the systems in-between don’t know that you’re actually doing a Patch, so that you lose some kind of semantics where the systems in-between can use for caching and stuff like that, but at the end of the day, it’s not that big of a deal, to be honest. I don’t think it’s a big deal anyway; I’m not an expert on – I’m not a total REST expert, so maybe I’m missing some huge issue, but I think even if it’s not available in your library then you can kinda simulate it in your code. CHUCK: Yeah, well and then the other thing: REST initially was based off of a paper by – it was a PhD thesis by somebody –. PETE: Fielding. Roy Fielding, yeah. CHUCK: And apparently, what we understand as REST especially in the Rails world – because Rails [inaudible] their own sort of thing at RESTs – it doesn’t quite match up with that and that’s where hypermedia APIs come in, and so there are a lot of different ways of following a standard for doing this kind of thing, and it really just depends on what your preferences are and what the tradeoffs are, and how well you understand them. PETE: Yeah. And that last piece, that hypermedia is like that final level. That’s like when you’ve kinda reach Nirvana and you suddenly realized, just like in The Matrix, when suddenly you realize it’s all code. It’s all hyperlinks – everything, it’s just hyperlinks! CHUCK: No! PETE: Yeah! JAIM: Let’s talk about hypermedia. You mentioned the term REST itself is overloaded, even though that’s what Fielding meant is the highest level of what RESTful is. In current practice, it’s just using HTTP verbs, but hypermedia’s something all different. Let’s talk about what’s the difference between a RESTful HTTP approach and doing hypermedia. What’s the difference? PETE: I think that I give it a lot of credit to some folks in the modern, current REST community. I think Steve Klabnik has been one of the guys doing this. They’ve basically embraced the fact that REST doesn’t mean what they want it to mean, so they made up a new word that means what REST used to mean. So hypermedia APIs is what we used to call REST APIs, as far as I can tell. What’s hypermedia? When you go to Amazon.com and you wanna search for books in a certain category, you don’t go up to the browser and go /?booksequels whatever. You just go to the homepage, the root of Amazon, and Amazon gives you a bunch of options like, “Here’s things you can do.” There’s this form here where you can fill it in and if you send me some query string in this from, then I will do a search for you. Hypermedia is kind of like that, where instead of me, the client, going to the browser and going, “Ok, I want to look at the fishing books section, so I’m going to go up to the browser and type /fishing.” Instead, I start a group resource and just follow links to where I want to go. The key thing here, the big difference, it seems really stupid and I think most people, when they first start doing this, they think it’s silly and inefficient and just something that people do because it’s trendy, but the big change it makes in your API is your client no longer has a bunch of hard-coded dependencies on the server. The server’s deciding what the client can do, and that means the server can evolve the API over time without your client having to move in [inaudible] step with the server. D; So the difference is, if I’m understanding you – so I’m going to Amazon, I’m clicking on something where my API is going to the service where I’m looking for fishing books and I don’t get back a bunch of information on fishing books, I get back a bunch of links – is that right? PETE: Yeah. So let’s say you’d go to the root of your API, and what you’d get back is a bunch of links. Part of that response would be like, ‘Here’s all of the different departments in the book store, and here’s the names of those departments, and here’s the URL to get the list of books in that department.’ And so rather than the client creating their own URL and then kind of just going to the server and saying, ‘hey I want /books/fishing,’ it asks the server to kind of describe what links are available to it and then it looks through those links for the one it wants, and then it follows those links. So it seems like kind of the same thing, but the difference is the client isn’t deciding the shape of the links or the structure of the regional space, if you want to get all fancy about it. JAIM: Okay, so before I’d say, ‘okay, give me a list of books’ and then I get the book that I wanna look at, look at the ID, and then I go books/id/whatever and get that. But in this case, I’m actually getting a link and I’ll just say, ‘oh, if I pass this link to my NSURL connection, and I get the data that I want without having to specify what the URL is. PETE: Yes. JAIM: Ah, very cool. ANDREW: I was just thinking a little bit about documentation, which I hope we talk about, but it seems like that design is also easier to figure out if you’re just new to the API. PETE: Yeah, it’s a really [crosstalk]. ANDREW: Because it’s sort of self-documenting, right? PETE: Yeah, that’s a really good point. Has anyone ever worked with an API, an internal API that’s documented on the wiki, where the wiki documentation was actually correct? Like it didn’t have any stale things – it never is true. The documentation, in general, gets out of date eventually, but API documentation, it seems like it’s never, ever correct and it just gets worse over time. So I think that’s one of the real benefits of this kind of hypermedia thing, is this self-documenting APIs. JAIM: Yeah, and you don’t need to write code to see what the API is doing. You can go to a browser, get the resource, copy-paste the link which is JSON, and say, ‘okay, I’m going to the state. I’m going to the state. Oh, this is broke.’ And so I can paste whatever links and not have to write any code [inaudible]. Anyone could run it; it’s easier to write tests around, you do some script-y type tests, so it’s easy to kinda tell your server team what happened if you broke something. PETE: Yeah, and if you are using one of these standards like HAL or SIREN, you can actually get this generic HAL browser, for example, that you can load up. It’s a little JavaScript t# that you load up in your browser, and it will kind of let you follow these links with quite a nice UI. You’re not like searching for JSON and copying and pasting links; it’s pretty similar to actually browsing a website, except obviously it’s machine-oriented, so it’s not going to be pretty, or have super verbose names, or maybe it will, but I’ve seen people use those kinds of things to actually do QA on an API, for example. JAIM: So how does a HAL response different than the standard web service response? PETE: It’s just like a convention for putting in your links, basically. I’m more familiar with HAL than the other ones, but I think most of them are pretty much the same. If you’re not doing hypermedia, then there’s no real benefit using HAL, because you’re just basically sending a chunk of text, or a chunk of JSON or XML or whatever, down to your client, then the client is doing something specific with that to pull out the IDs and form URIs or whatever. And once you get to the point where you want the client to be able to follow links, then you need a convention for what those links are going to look like. HAL basically gives you a convention of saying all the links will be in this subsection of the JSON document called underscore links, and each of those will have a name and then an href, so a URL. It also lets you do templating, so there’s this standard called URI templates, which lets the server say – let’s say you wanna do a search for the weather for a city. You could say, you want the server to tell the client how to do that search. So if the server just says ‘the search thing is on /weather/city,’ then the client doesn’t know how to use that to do a search. You can’t list a link for every single city because you’d have this 500 Mb JSON document with a link to every city. Instead, what you do is you say ‘here’s the template for building a URI to search the weather in a city.’ So you say, ‘go to’ – the template would be like /weather/city and then {?city [inaudible]} whatever. And then the client can use that template to say I just plugin in the city search string here in this URI, and that’s how I do a search for weather in a city. JAIM: Okay, so we talked about one aspect of the response has [inaudible] different links, like underscored links. What about the data that we actually wanna look at? Is it all links, or could we actually have data that we can actually use? PETE: One of the things that I really like about HAL is it’s very, very lightweight. If it’s just like data, going  back to my user example, if you’re trying to do representations for /user/pete, the first name, last name, email address, all that stuff – you can just roll that into JSON any which way you want, just like you would if you weren’t doing HAL.  Regular fields just go in as regular fields; you don’t have to do anything special there at all. Links go into a special subsection called underscore links, and then there’s also this concept of kind of embedded resources, so you’re asking the server for a list of only users friends, so you wanna get friends of Pete. Actually, that might be a good work example [inaudible] how this would work. After the client has logged in, it might go into the server and say, and do a get on /user/currentuser or might just be /currentuser. The server says, ‘okay, you wanna know the canonical resource for the current user, so I’m going to redirect you to /user/pete, because that’s the person currently logged in.’ So now the client has called down the JSON that represent /user/pete, and in there there’s a link called friends, and that goes to /user/pete/friends. So now the client, your iOS app follows that link, and it gets back a list of all the friends of Pete. So in this case, the client has not specified any of those URLs; it’s just following a set of links that the server got back, or the server sent back to it. So at this point, the client has pulled down /user/pete/friends, and what you wanna see is a list of those friends, and you wanna know their name, and their avatar or whatever, so you can display it in the app. So you can think of each of those friends as another user resource, and essentially you wanna embed a set of resources inside of this response. It’s not just one user, it’s like seven users. HAL lets you do this thing called embed it, just like I’m just going to go embed it, so that you can tell the client that this is not just a list of stuff. Each of these individual items is its own resource; it has its own URL that you can actually return all of that data in one document rather than making the client say, ‘okay, here’s a list of 50 URLs, and now I have to go and pull down the URL for /user/chuck, the URL for /user/jaim, etc. ANDREW: So if we got a list of Pete’s friends, and we’ll say those are embedded, do you generally have the entire [inaudible] for the friends? Is it a subset of things? PETE: Yeah, that’s a good question. I know that there’s definitely an option there of doing what I kinda called abridged resources, so you don’t wanna send down the entire resource. You know, if it would be 50 users, it would be quite inefficient maybe to send down everything about every user if all that you really care about is their name and the link for more information. So you can do that, you can just send down the entire thing, or you could, I guess, just send down links so that’s like the ultimate version of the abridged resource, which would just be, instead of  sending down the actual resource, you just send down about your links, so that if the client wants to, they can follow that link to get the full information about that person. I actually don’t know where the HAL has any conventions around that, but I know that when I’ve done this kind of stuff before I’ve definitely reached for those kinds of ideas of abridged resources. So you send down just enough so you can display on the client a list of all the users with their name and a photo, for example. Then if you need more information, when the user taps on that user, then you’d go and follow the URL that came with that resource to get the full information about the resource. CHUCK: Yeah, I’ve also done it though the other way where, let’s say I have a dashboard or some page that displays multiple pieces of information rather than make the canonical request, and then make the other request to get the other information when working with an API like this. I’ve also set up an endpoint that’s basically like get dashboard info or something, so that then I make a request and I get all the information back. Because I know I need it for that page and I can tailor it to the behavior that I have, and I have to make multiple requests to get everything I need. PETE: Right. Going off of the REST topic for a second, that’s a really good example of what I would expect to see more of a vertically-aligned team. Particularly if it was a team where you’re the team – you’re doing the server side and you’re doing the client side – you can build that API to work with the client. If the client is always going to make the same five requests for the same five pieces of information, just make an extra endpoint that represents all of that information bulk for bulk. It’ll be more efficient, it’ll be less load on the server, etc, etc. If the client and the server teams aren’t able to work together well enough, then they end up not doing that and so the communication – because the communication between those teams isn’t very efficient, the API that they designed isn’t very efficient and it ends up having to do a bunch of extra work. CHUCK: Yeah, I think the other danger with a lot of this stuff is that we tend to think of our resources in the same way we think about our database tables, and that does not always line up with what’s the most efficient way to build our API. PETE: Yeah. I think that when you’ve reached the point that the way that you modeled your API doesn’t map directly to the way that your database is laid out, I think you’re in a really good spot from a resource modeling, like a REST point of view. If you [inaudible] that and you can kind of project the data into a different form so that it can be consumed better by clients, then, at that point, you can say that you understand how REST works. JAIM: Okay. So you’re saying that REST doesn’t have to be just [inaudible] over your database. PETE: Yes. JAIM: And it probably shouldn’t be. PETE: Yes. JAIM: Alright. PETE: It’s a good start; it’s better to do that than it to just be basically modeling method calls, but I do think there’s this kind of level that you go – there’s kind of evolution you go through where you start off with method calls, and then you turn it into, essentially crowd over your database. And then suddenly you have this revelation that you can model these abstract concepts as resources. You can kind of rarify the concept of logging in as a resource, even though there isn’t a login table in your database necessarily. JAIM: Okay. Yeah, that’s very cool. One thing I wanted to ask people on this show – we’ve got some good back end people, we got some client people, we’ve got people that do both – do you think that a mobile app should generally be reuse the API from the main website? CHUCK: What do you mean? Like, for the JavaScript front end? JAIM: Yeah. A lot of times, a mobile app will do some subset of what the main website would do. You have some web application – a mobile app will do not everything; it’s going to do a small piece of it. So we talked a little bit about HTTP – you do a Put, okay, so I get the user Pete, and I know about these fields. The mobile app may not know about every single field that might exist from the website, so you have problems where you get a Put from a mobile client which maybe doesn’t know everything, where the full application would have a bunch of different fields that the mobile app doesn’t really care about. How do you manage that type of thing? CHUCK: Let me talk a little bit about the way I tend to design my APIs, and that is if somebody doesn’t need it, I’m not going to expose it. So let’s say we have a full-on web application. We have a little app we’ve built called Facebook or something, and we wanna build an iPhone app that does all of the things that Facebook does. Well, I’m not going to build those APIs until I know somebody out there needs them, and so in this case, if I’m only exposing APIs for my private application, then I’m only going to build them as the iPhone app team needs them. That way, I don’t have to maintain anything that is not being used. I don’t have anything sitting out there that could be a potential security vulnerability if I don’t upgrade it. Usually, APIs in the apps that I build are not as fully featured as the actual app unless, for some reason, things haven’t moved ahead on the web app and they have on the iPhone app to the point where they’re basically reaching [inaudible]. JAIM: Okay, you generally have a different API for the web application and the mobile client? CHUCK: As far as having an API that’s separate for both, that really depends on what I’m trying to accomplish. One of the tricks between the two is that typically, on the web, your authentication can be different from the authentication from the device. The flip side is that if you’re using something like OAF or something, then you can store the OAF tokens in the cookies or somewhere where the browser can get them, and that way, your JavaScript is then capable of making the API requests using the OAF tokens that we’re talking about, and then your mobile app can do the same thing. And then you just make sure that those tokens expire after a reasonable amount of time for security reasons, but then you kind of can. The other thing you have to keep in mind though is if the behavior between the two is different enough, then you may wind up exposing different APIs. The flip side is that you wind up maintaining a surface area that’s sort of, to some degree, at least twice as large as the others. So as much as you can make them overlap, you’ll get the payoff there. But it’s not always simple because the capabilities and requirements aren’t always the same. JAIM: Right, yeah. That’s something to figure out to get the tradeoff on hand, you’re reimplementing a large part of the application for a mobile or a full web app when maybe you don’t need to. But yeah, it’s good to get some perspective on what people are doing, because I’ve had problems with both approaches, so it’s always good to hear what people are doing. CHUCK: Yeah, I’m not convinced that there’s a direct or right answer to that, but if you know enough about what you’re trying to do and enough about the systems that are accessing it, then you can make decisions based on the tradeoffs that are there. JAIM: Okay. Very cool. So are there any other benefits to doing this kind of RESTful approach? PETE: Quite funny you should ask! One of the things that I found really, really valuable with this REST-y stuff – not necessarily all the way up to hypermedia, but at least kind of embracing the nouns and that kind of thing is [inaudible] treating things as nouns rather than verbs is caching. The web is like one of the most, probably the largest distributed system in the world, and it works pretty well considering how crappy most of the stuff is out there. Servers go down and things have slower response times and yada-yada-yada, and the big reason for why the web manages to scale so massively is that it has this amount of caching infrastructure built into it. If you also do browser-based development, if you do any front end JavaScript [inaudible] kind of stuff, then you probably know – or even if you do web development in general actually, then you know about the way that browsers can really be intelligent about caching stuff that hasn’t changed, so there’s always caching headers and pragmas that you could use so the server can say, ‘this response is valid for the next 10 minutes’ or ‘this response is valid until this date’ or you can use things like e-tags which are ways for the server to say, ‘here’s a tag that represents the state of this object. The next time you ask me, if the tag is the same as last time, then you don’t need to re-download it; just use the thing that you have cached locally.’ JAIM: Caching – what benefit do we gain from caching? How does that even work? PETE: There’s a few things – I mean, the cheapest network request is the one you never send, right? So, if you set up your NSURL connection or whatever to honor the caching information that returned from the server, and you set up your server to actually send caching information, they you can do things like –. Let’s say you have a newsfeed that only updates once per minute, and you know, on the server side, you know that it’s not going to change more frequently than once a minute. What you can do is when the client makes a request and says, ‘hey give me the latest from the newsfeed,’ when the server responds, it can say – in the caching header, it can say, ‘this isn’t going to change for a minute, so don’t even bother to ask me for another minute.’ And then the next time the client wants that newsfeed data, if the cache copy is still fresh based on that information, it cannot make the network call at all. JAIM: Okay, so you don’t see that as a huge [inaudible]? So you’re making a call your network request, but it’s not actually going out to the network; it’s just returning what is cached locally on your device. PETE: Yes, exactly. And what’s really cool about – there’s two really cool things about this if you do it in the HTTP level. Number one, your client code doesn’t need to do anything at all. Your client code can happily do what looks like really inefficient operations, like making a URL request every single time, but it’s never going to go out on the wire. It might not even go to disk; it might just go to memory, so you get this kind of free caching on the client side if you do the work on the server side. And then the second benefit is the server, which knows the most about the state of this data, gets to decide how to cache it, and it can change that over time. So if in the future, you decide – let’s say, so this is kind of maybe a little bit of a convoluted example, but let’s say your service is undergoing a bunch of extra load and you’ve got something. You’ve been featured on the front-page of Hacker News, those are the people that have downloaded your app, and they’re all hearing the newsfeed, and you’ve realized that you just can’t scale to number of users and you wanna dial back the freshness of the newsfeed to be only five minutes versus one minute. If you’re doing HTTP caching headers, then you can just change that on the server, and all of the clients will start honoring that straight away. So that means, like you don’t need to release a software update; you don’t need to do anything at all – you just need to change the caching stuff on the server and stuff just magically work. To me, it makes sense to put the controls of the cache-ability or something in the hands of the server, because the server knows whether something’s changed or not and how frequently it’s changing way more than individual clients. JAIM: Do we have to do anything on the client side to honor the caching stuff? PETE: You have to tell iOS to use caching – I can’t remember if it’s by default. I remember, you might need to do some extra stuff to ask it to use a separate store or something like that, or it uses a separate store by default – I can’t remember. I’m pretty sure iOS does a pretty good job of just doing this for you, as long as you’re using – as long as you’re not writing your own network [inaudible], which I don’t think everyone’s really doing. JAIM: Caching is really powerful. Like, no matter how complex your server operations are – you make a request, do a bunch of crazy database stuff and processed off in memory, and you spit it back. Whatever you spit back, it’s generally just a text file, and that’s something that almost every web server can dish out very cheaply, so you can do a ton of it without having to go back and forth your database and web server and that kind of stuff. PETE: And there’s also all this stuff in the middle, which we don’t really think about that often but is actually particularly relevant for mobile. When I make a request for some resource – the newsfeed or whatever – there’s a bunch of intermediary caches that might get hit, so the carrier, AT&T for example, probably has a cache where they’re trying to cache very, very commonly-used or commonly-requested resource. Someone might be using Akamai or other edge caching technologies to kinda try and keep the content physically closer to you so that you don’t have to go all the way back to the UK or Australia or whatever for the content. Once you get into the server’s infrastructure, they might have a caching there in front of their application server, etc, etc, etc, and if you use HTTP caching, then you get to leverage all other people’s caches for free. So you get to not just get free caching on your phone, but you get free caching with Akamai, you get free caching with whatever caching the network provider is giving you. JAIM: So it sounds like you got a ton of stuff thrown in if you just play by the rules of the web and [inaudible] so they don’t change. PETE: Yeah. And that big thing for that kind of a weird, just one weird trick to improve your caching is don’t use query parameters on gets unless you have to. Because, I guess back to what we were saying at the start of the show, so many people for so long who are used to making gets that actually did stuff. Generally, the way that people have always done that is through ?changepassword=newpassword or whatever. Even though you’re doing a get, you’re actually changing things – you can’t cache that because if you cached it, then the next time that you try to change the password, the password wouldn’t get changed, right? By definition, that kind of operation always has to get all the way back to the origin server, because it’s changing something on the origin server. Because so many people abused this functionality and did the wrong thing, it’s now built in to almost every caching technology that they won’t cache and get with a question mark, even though in theory they should. So if you put a question mark in there and then you’re like breaking the cache-ability of your resources#. JAIM: That’s true! I know with [inaudible], on the Windows side, you can actually cache different query parameters. You can explicitly state, I wanna cache this one – are you saying that? PETE: It could be true with the technology that you own, but it might not be true with Akamai, for example. Akamai might have some [inaudible] and you can probably tell Akamai, ‘hey, please’ even though it doesn’t have a query parameter, ‘please turn off the magical thing,’ whatever, but I definitely have heard, at least anecdotally, that putting query parameters into a URL where you don’t need them hurts the cache-abililty of those URLs. JAIM: Okay. It sounds like even though it may be possible to do more caching with query parameters, if you’re trying to do it, you’re probably doing it wrong. PETE: Yeah. I mean, there’s some stuff that you kinda need to do to use query parameters searches, like, the obvious example – you can’t have every search query in the world under a URL – I mean, you can, but it’s just really weird to do that. There’s the semantics kinda don’t make sense so modeling that [inaudible] makes a lot more sense, I think. CHUCK: Alright, is there anything else that we should cover on this? We’ve been talking for an hour. PETE: I kinda wanted to talk about faking out back end service, but I don’t think we’ve got time to do it justice, so maybe we’ll have to save that for another show. CHUCK: Yeah, is it a topic that we can cover in an entire show? PETE: If you let me talk, and then yes. CHUCK: Ok, good enough. We’ll get it on our next schedule. JAIM: Put it in the queue. CHUCK: Yup. Alright, well, we’ll wrap up then and do the picks. Jaim, do you wanna start us off with the picks? JAIM: Yeah, I’ll start off with the picks. I would probably make two of the picks that Pete’s going to make, so I’m not going to steal his picks. I’m going to pass on them; I think I made picks the last time we talked about hypermedia, but I’m going to talk about a grain. It’s not specifically a beer pick, but I like rye as a grain; grows kinda north of here. A lot of beer makers are starting to make beer with rye and I like it a lot; it’s kinda nice and spicy. But you don’t have to just have rye and beer, I’ve made cereal with my pressure cooker up, put rye with oats and stuff; it’s pretty good. It makes good bread; it’s just your all-around good grain. I think that should get more props. I’m going to just pick rye, and specifically, yeah –. The brewery out here Summit is made of Frost Line Rye Ale, and it’s actually pretty nice, so I’ve been enjoying quite a few of those lately. So, that’s probably the next thing in brewing, you know? Everything’s getting over quadruple IPAs, but not at all doing rye beers, so check them out. PETE: Plus one! CHUCK: Alright. Pete, what are your picks? PETE: Back In The Saddle, the rye pale ale from Maverick’s, I have picked it before but don’t care, I’m going to pick it again. RESTful web services was the original book that I really liked from a few years ago; RESTful web APIs is the new one that he just released recently, which I’ve heard is really good, and is [inaudible] by these guys that are at Spotify called Scaling Agile at Spotify, and that talks a lot about vertical teams and this ideas of committee of practices and tribes and guilds and all sorts of crazy concepts. If you’re skeptical about this whole vertical team thing, then you should read this paper at least, and then disagree with me. CHUCK: Alright. Andrew, what are your picks? ANDREW: The first on is just a post by Justin Williams from his blog a couple of weeks ago that I really liked, that kind of summed up my feelings about using third-party libraries. In iOS apps, I think people who know me know that I’m not super enthusiastic about using lots of third-party libraries in my apps, and this – he kinda has the same feelings I do, so I thought that was a good post. The other two are sort of music picks. The first is a pair of speakers that I recently bought that are really cheap. Amazon’s got them for $87right now; they’re made by Pioneer, but they sound super nice for $87 speakers, so I’m very happy with it. And then last is Discogs. Discogs is a website that has discographies for –. I think the goal is to have all the music ever really sung in any format, in any country – they’re not there yet, but it’s open, sort of like the PDA. You can submit and edit the information there yourself and they let you build up a list of all the music in your collection and you can do buying and selling. Anyway, it’s just been – I’m a record collector, so it’s a great resource, and I’ve been enjoying adding my stuff to it lately. Those are my picks. CHUCK: Awesome. So, I’m trying to think of a few picks. One things that I’ve been playing a little bit lately is Diablo III. I know the game’s been around for a while, but I needed something to kinda break up the heads down, coding time a little bit and that’s been kind of fun. So I’ll get on and play for a half hour here or there, just to kinda clear my head and relax a little bit. Another pick that I have, and this is a tool that the team and I’ve been working for the last week is using this called Flowdock, and it’s kind of a team chat deal and you have different – I think they call it flows or contacts – but basically, you can reply to somebody else’s Post, and it’ll go into its own little stream, and so you’d wind up having two panes. One pane is all of the conversations in the same thread, and the other one, if you click on the little bubble next to the message that somebody sent, then it’ll actually show you the context that they’re speaking in, and you can reply to just that context. Anyway, it’s pretty nice, so I’ve been able to get help that way and people can carry on other conversations in other contexts, and it works out pretty nicely, and so I’m pretty happy with that. So those are my picks. I also want to remind everybody, we’re reading a functional reactive programming book. We were going to do that this week, but there was a technical, or he had some family emergency and couldn’t make it this week. So we rescheduled that, and we’ll be talking to Ash a little bit later about the book. So keep an eye out for that, and thanks for listening. We’ll catch you all next week! [Hosting and bandwidth provided by the Blue Box Group. Check them out at BlueBox.net] [Bandwidth for this segment is provided by CacheFly, the world’s fastest CDN. Deliver your content fast with CacheFly. Visit cachefly.com to learn more]

x