060 iPhreaks Show - Core Data with Marcus Zarra
The panelists discuss core data with Marcus Zarra.
[Would you like to join the conversation with the iPhreaks and their guests? Want to support the show? We have a forum that allows you to join the conversation and support the show at the same time. You can sign up at iphreaksshow.com/forum] CHUCK: Hey everybody and welcome to episode 60 of the iPhreaks Show. This week on our panel we have Pete Hodgson. PETE: Good morning, from Alamo Square. CHUCK: Andrew Madsen. ANDREW: Hi, from Salt Lake City. CHUCK: Jaim Zuber. JAIM: Hello, from Minneapolis. CHUCK: I'm Charles Max Wood from DevChat.tv, and this week we have a special guest, that’s Marcus Zarra. MARCUS: Good morning, everyone. CHUCK: Do you wanna introduce yourself really quickly? MARCUS: Certainly! My name is Marcus Zarra. I've been working with Core Data since its inception way back in OSX 10.3, back when it was in beta. I've written a book on the subject for the Pragmatic Programmers; a second edition of it came out about a year and a half ago. I've been consulting, advising on it, teaching at conferences, teaching at colleges – pretty much you-name-it when it comes to Core Data – then helping a lot of developers out over the years who are working with this API. In addition to that, I also am a managing partner of MartianCraft, one of the larger Mac/iOS development shops here in the United States. CHUCK: Awesome. Now –. JAIM: That’s kind of a large side-project, don’t you think? MARCUS: Indeed, it is. It takes quite a bit of my time, unfortunately. CHUCK: So I'm going to jump in here and just apologize to everybody. We got about a half hour into the show, and then realized that somebody forgot to hit the record button, so, we apologize for that and hopefully we’ll sound as smart the second time through. PETE: I think we should all switch roles. [Crosstalk] JAIM: I guess now I'm the Core Data expert, so, ask me questions. CHUCK: I know that you listened to the Saul Mora episode that we did. Is there a good jumping off point from there that you can think of? MARCUS: By this point in time, Core Data has been around for many, many years. We’ve been using it since iOS 3; we’ve been using it on the Macs since before most people had a Mac at this point in time, considering the huge growth spurt we had on iOS. Core Data is a fairly well-known API. I hesitated to use the word ‘popular’ because a lot of people tend to blame it for things, but it is extremely well-used. The jumping out point at this point probably is looking at what Apple announced for coming out later this fall in iOS 8 and the new Mac OSX Yosemite. They mentioned a couple of new APIs, which are interesting and worth talking about, but they also made a statement during the talk that, to me, pretty much had me standing up and cheering. They gave us a definitive answer when it comes to threading with Core Data, and while not the first time that they’ve done that, but this is the first time that they really said, “Stop doing x, we want you to do y.” To explain that, it’d probably be helpful to go back to the beginning. When Core Data first came out, there was no explanation about threading. There were no rules about it; we had to kinda figure them out on our own, so we walked to a minefield that was very densely packed and things would explode at us. In the beginning, threading was very, very difficult, and pretty much the general rule is, you don’t know how to do it, you're going to screw it up, don’t do it. A little bit later, the APIs got refined, it got a little easier, and we started, “Okay, this is how you do threading.” There became this rule, this mantra, that I have been preaching myself is, “One context, one thread.” A context can’t jump threads; if you're in a multi-threading environment, you have to have one context for every single thread in your application. And that was still hard, because threading is hard. Most people just can’t wrap their head around threading; they can’t visualize thread barriers and think like that – it’s a very hard concept. So back in iOS 5, Apple came out and said, “Okay, we’re giving you a better way. We now have blocks.” Blocks make threading easier because blocks are very easy to conceptually visualize as opposed to threads, which are invisible. With blocks, we now gave you this new API with Core Data to make it easier to work with blocks, and now we have three different ways to work with Core Data. We have your main thread – what I like to call your UI thread – and context. You can define it, say, ‘this context is for the main thread.’ Then we have another one which is a private thread. That one’s kinda fun, it’s like, ‘Okay, I'm going to define this one, but I can't touch this thread; I have to call it through this block’s API,’ which is fine – very easy to understand. But they also left in the original design, which they now call Thread-Confined, which is back to that old rule, “One context, one thread.” And so, there you go. Now you have a definition of your threading model; go forth, and be happy. People were still screwing it up, and because they use that thread-confined one, and they would still have the problems with threading –. At WWDC this year, they finally got up on stage and said, “We’re now changing that rule. We’re making it even more defined. That thread-confined context is going away.” It was a very definitive, and to me, a very powerful statement of saying, “Stop using this.” Some people got it right, some people got it wrong, but just stop. You don’t need it anymore; we don’t want you to use it anymore, and in fact, at some point in time in the future, this is probably going to get deprecated. So we now have a very defined statement from the Core Data team going, “This is how we want you to use our API; stop breaking it, stop making us cry,” which was nice, because before then it was – you could get away with a lot. You could say, “Well, I'm using a context on multiple threads because I'm locking correctly and I know what I'm doing” and people wrote this really unmaintainable mess and they succeeded in writing what I call “Write-Only Code” surrounding Core Data by doing all this locking and this threading complexity and stuff like that. That, to me, was probably the greatest thing that came out of that session was them saying, “Stop. Just stop. You don’t know what you're doing; you're screwing it up; stop doing this. Only use main context, product context – that’s it. No more. No more else. Stop using the other stuff.” And they're actually writing APIs to that point where you can’t do it, so they're actually supporting that statement in the actual API itself. That, to me, that’s probably a long-winded way of saying, “That’s where we should start. We now have a very clear guideline, going forward, on how the threading model, with Core Data, should work.” JAIM: So when we’re actually creating our application, what are the differences between the different approaches, the old way and the new way? MARCUS: In the old way, you created your context – and generally, people would only have one because that’s what the template had and they didn’t understand Core Data enough to just fully grasp the concept, so they'd have one they'd pass it all around. If you're not using threading at all, that'd work perfectly; you'd have no problems at all. But then you're like, “Oh, this process over here is slow. I'm going to add threads to make it faster,” right, because threading makes things faster. Then we’d get into trouble with it, and I was like, “Oh, what happened? My database is corrupted” or “All of a sudden I'm locking across threads” and they just wouldn’t understand what was going on. So that was the old way [inaudible] they had a very harsh lesson on – threading is a wonderful tool, but it is also a very, very sharp tool and it doesn’t have a handle. The newer model, which is telling the context, ‘this is what you are going to be’ – it got rid of that ambiguity. If you said, “I'm defining you as a main thread context” and then if you try to use it on another thread, the application would crash, and it would be very sharp-edged, say, you are doing this wrong. That’s why they introduced it in iOS 5; they’ve since pulled back from that a bit. It doesn’t crash anymore. You don’t have to turn on a debug flight to get the crash back, mainly because when they turned that on, a lot of apps crashed. I mean, a lot – a very, very high percentage of the App Store stopped working, so they had to pull back from that sharp edge and they had to put bubble wrap around it. But they now have that definition of saying, ‘okay, I can turn those on but I gotta make sure that I'm doing it right.’ I've got this context that says, ‘I'm going to be on the UI thread only; I'm only for the purpose of the UI,’ and the context would now say, ‘okay, this is the way you're going to work with me, and if you need to work with me from another thread, here’s how you do it.’ So now, we have a very defined way to do that. They also introduced the Private Queue Context, which is just the exact opposite – you can't use them at any thread at all. There's no thread that you have access to that you can access that context directly. You must access it through a block; you must create a block saying, ‘this is what I want to do, it’s this Core Data stuff’ and then hand that block through a method through perform block, so you couldn’t actually touch that context at all without it blowing up, and it forced you to say, ‘okay, here is my self-contained little bit of code, hand it off, go forth and do this’ and then it would come back. That’s the direction they're heading and they're keeping you away from the threading, but still allowing you to say, ‘go do this in the background.’ ANDREW: Something you said earlier raised a question in my mind; you said they are actually kind of going in with this new, I guess you'd call it self-deprecation of the old thread confinement model with the new API. Can you talk about some of those new APIs that really sort of require this new private queue/main queue sort of model? MARCUS: Yes. One of them that they introduced, which a lot of people were cheering about and that kinda made me giggle, is a new batch updating. The history behind that is, a fairly famous developer, Brent Simmons, several years ago wrote a blog post about why he had to walk away from Core Data in his application at the time called NetNewsWire. It’s a common story – well, common to me, because I always hear the bad things about Core Data, because they call me when they think it sucks. A common philosophy of Core Data is thinking that it’s a database. A mantra that, like I said earlier, I should tattoo on my forehead is, “Core Data is not a database.” Core Data is an object-relation – an object model, an object hierarchy, and it can persist to disk – not required – and if you persist to disk, it can persist to a database. There's two can’s in there – there's the fact that it can persist to a database structure is a secondary function of the API; it’s not the primary purpose of it. At first, it’s like, ‘well, it sounds like I'm sweating hairs, but I'm really not.’ You have to approach it in that that direction of ‘this is an object graph.’ In Brent Simmons’ case, he had a RSS reader, and he wanted the ability to go into a site and say, ‘read all’ and what that would do under the cover is in the code, it would pull in all the articles for that site, flag them as ‘read’, which is a Boolean, and then save them back up to disk. Well, in the database world, that’s very, very easy to do no matter how many articles are in that site, whether it be 10 or 10 million, it really didn’t change much because the database could flip that bit very easily. With an object graph, that’s painful, because it has to load in all those records, change literally one bit, and then write all those records back out again. You're looking at 10 million records being pulled into memory, which is the problem he was running into. And then of course he wrote this blog post that went viral for a few years, and to this day, people still cite that as a reason why they won’t use Core Data, even though they’ll never have 10 million records to begin with. The new API introduced in iOS 8 has finally solved that problem, but they solved it in a fairly raw or risky way. You can now write code saying, ‘I know I'm using SQL out at the back end; I know that this going to be an expensive operation, so I want to do a batch update to all these records and I want to change this property to this value,’ which is great; it works very, very fast. They did a test onstage where it showed you can update a million records in less than a second, and there were lots of cheers and applauds. But that API has a lot of very dangerous, very sharp edges, which I expect to get a lot of phone calls about in the near future and the fact that it is updating that record on disk, it is using the SQL APIs, it’s doing a SQL call and it’s saying, ‘update table, blah-blah-blah’ – it’s not updating the object graph. Underneath your object graph, it’s changing the bits on disk and that’s it – that’s all it’s doing. It’s not telling your object graph, ‘hey, I've changed. You might want to go refresh.’ That’s an exercise left up to the developer, which is how they were able to make it so fast, because they're not actually doing the hard part for you; they're letting you do that. PETE: This is kinda weird because I know the direction our conversation is going to take because we just did this half an hour ago, but without recording it. How would you mitigate that problem? If I want to use this new batch API, is the right thing to do – if I'm Brent Simmons and I've got my NetNewsWire app and I want to update five bazillion records, should I use the batch update thing? Should I use some other technique? What's the best option? MARCUS: There are two answers. My preferred answer, and the answer I expect to be giving a lot in the near future, is the old joke with the doctor, “‘My arm hurts when I do this.’ ‘Don’t do that.’” And my answer here is the same – don’t do that. Anytime that you're having to update 10 million records just to change a bit, you really are using a cannon to kill a mosquito. You want to design that differently, so my first answer is, “Let’s look at your object design and let’s fix that.” Secondarily, if you still have that problem, or if you're already pinned into that corner, then yes, we need to use the batch update. Let’s be happy that it’s now in there, but now we need to make sure that we refresh all the objects in memory. What that means, as far as the application is concerned, is if we’ve got any kind of view that is showing one of those objects, we need to tell that view, go refresh yourself. That may be as simple as going into every single [inaudible] of your context and just saying, ‘reset’ and then telling every table view to reload data, causing them to go back out to disk and grab the data. It can be just that easy, depending how well our database or our application is designed. If our application is holding on to objects and we’ve got a lot of strong references and things like that, then we get into some more hairy code, and we may need to an auto-notification where everybody who’s holding onto an article object will need to listen for that notification and refresh its object or something else that’s equally ugly. It kinda goes to the application design, which then leads back to the ‘let’s solve the root problem.’ Let’s solve the problem where we’re required to go into 10 million records and flip a single bit. JAIM: So what are some better approaches for this? MARCUS: I was setting you up for that question so you could ask that question again, because it is a great question. As far as the famous one for Brent, there's not an easy answer for that one. However, the answer that I recommend on that one is that we have a higher order of that state. The state he’s trying to change for this website, whether it be an Apple RSS feed or something, I want to say ‘this site, everything from x date back, I don’t care about it anymore. It’s read. I'm abandoning it; I've failed; I could not keep up with the feed,’ so that’s where we want to set that information. We want to set that information on that site object and say, ‘read date of x.’ We can say, ‘for this Apple developer news RSS feed, I'm setting a read date to June 16th 2014.’ And then when I'm fetching articles, if I'm fetching my read counts, I then change it around, saying, ‘for this site, give me everything that’s unread that is newer than June 16th 2014.’ You would flip that around so the date calculation’s being resolved first, and that gives us a higher order. We only care about that bit as a secondary piece of information. The primary piece of information is ‘when is the last read date for that site?’ and we can go further than that – we can do that on every site, so whenever a site gets to the state of having a read state of zero, we set that last read date. That will improve our fetch performances across the entire application, because now, five years from now, when the user has 10 million articles in that developer news, we’re not checking 10 million articles for that read bit. We’re checking 10 million articles for their date, which is an index column, which is very, very fast, and then we’re checking for that read state later. That read state could be, we could end up adding additional states to that read; it could become more complex, so we want to be able to roll that data out into the site. PETE: So it’s another example of the same thing, right? If that’s a hard problem to solve, maybe don’t solve the problem that way or not model the problem a different way so that the solution is easy. MARCUS: Yeah, it’s a denormalization, which is – the database guys that are probably listening to this are probably crying, but Core Data is not a database, again. I will say that a lot. We actually want to denormalize the data as much as possible. If it’s a calculated field, we don’t want to have to calculate that every time because it’s an object, so we want to store that calculation in the database. Same thing with the last read flags and things like that – we want to store that data; we don’t want to have to calculate that data, so we’re denormalizing a significant amount. PETE: I'd say I've made the wrong decision. The past me was stupid and decided that they were going to calculate this thing every time, and now, present me has seen the light and realized that I should just denormalize it and just say the result to the database. That means, if I've already got 10 bazillion records, I need to migrate all of those existing records. What are the approaches there if in Core Data land? MARCUS: If we’re dealing with this pre-iOS 8, which is where we’re at today, the migration itself, the changing of the database itself – let’s start there. I woke up one day going, “Okay, calculating my invoices every single time is an expensive operation and really pinned me into a corner, so I'm going to change my invoicing application so that the total for the invoice is stored.” First of all, I need to change my database. In Core Data, what that means is I create a new model that's based on the old model, and I'm going to add this one property to the invoice saying ‘total’ that creates a migration situation inside of Core Data. Now, there are two different ways – actually three, the third one doesn’t count, but I’ll explain it in a minute – there are two primary ways to do a migration Core Data: one’s called Heavy, one’s called Light. The names should give you a hint as to which one you want to use. A heavyweight migration is the original migration strategy that they added in Lion, which is, Core Data would say, ‘I need to do a migration;’ I've got my old store – my old SQL-like file – and I'm going to create a new SQL-like file that’s based on my new model. And then I'm going to walk through that old store, I'm going to grab each object, pull it into memory; I'm going to create a new object in the new store in memory; I'm going to copy the data over, and then I'm going to save it back out to disk. Worse, I'm going to do this in three passes. The first pass is object creation and copy properties; second pass is set up all the relationships between my tables; third pass is I'm actually going to run all the validation rules, so it’s actually keeping the entire store in memory the entire time. If that sounds expensive, you're right. That’s horribly expensive. PETE: That’s why they call it heavyweight, right? MARCUS: That’s why they call it heavyweight. It’s so expensive that it’s extraordinarily rare that'll even work on iOS. Most of the time, you'll pop memory on iOS. So they came up with a new one, also pretty much at the same time back in Lion and it’s called Lightweight Migration. What that one is is that Core Data says, ‘I've got two different SQL-like file designs and I'm going to compare them and go, ‘Oh, you're just adding my property. Well, I can do that.’ And it’s going to go down on to the disk, down the SQL-like API and say, ‘modify table’ and then it’s going to set that new property to a default value. Now, you can actually change it on disk and that’s very, very fast. It’s usually a few milliseconds; it’s not even perceivable when it’s done right. That’s a lightweight migration, which is, it’s great, it’s wonderful, everybody likes it. The problem is that it doesn’t do any mathematical calculations, which is what the heavy could do. The heavy, you could actually inject code into that migration process, and you could do logical decisions going, ‘While you’ve got this object in memory, please calculate the invoice value and stick it in this new property right here.’ Or if you were splitting one table into two tables – there are a lot of different things you could do. As a matter of fact, it was pretty much wide open. You could manipulate the database anytime you wanted because you had the whole freakin’ thing in memory anyway, so why not? The new one, you can't do any of that. It’s, ‘I'm adding a column; I'm adding a table; I'm removing a column; I'm removing a table’ – it’s that kind of stuff. It’s very, very clever as far as how much of a database change you can make, but it won’t change the data other than deleting columns, deleting tables. That leads into the problem that, ‘okay, how do I get that calculation into that column?’ You could load everything into memory and force calculate all of those invoices taken back out, but then you're going to run right back into your memory problem, your performance problem, and you're just going to say, ‘you know, I like this app, but that 10 minutes spent on launch kinda sucks.’ How else do you solve that? The answer to that is doing a lazy calculation on that. During the life cycle of an object that’s called NSManagedObjects – during their life cycle they have certain methods that get called on them. One of those is awakeFromFetch, and it does exactly like it sounds. When you retrieve that data from disk and you realize the object becomes into memory and the values are populated, awakeFromFetch gets called. That’s the perfect opportunity for us to do that calculation, so we can go in and say, ‘I'm an invoice; let me check my total. Is my total nil, as opposed to zero?’ We can make it an optional and say, ‘Okay, it’s nil, so therefore, since it is nil, I'm going to do this calculation and I'm going to now set that property on launch.’ Now, we’ve got a lazy loading of that total property and we’re no longer calculating that on every launch. JAIM: Okay, so at that time you actually used the object. You can verify that, “Oh, this has not been set,’ run your logic, set it, and go on your way. MARCUS: Exactly. What we can do there is we can actually do a couple of passes – there are clever and interesting things that we can do on our application, depending on our business need and our business logic. For instance, we could let that go for a little while and then in our next version, we can modify the database again and say, “Okay, that column is no longer optional and now I've got data protection, now I've completed this two-step migration.” And we can go even further, saying, “Okay, this is the 15th time you’ve run the application since the last migration. In a background process, I'm going to go grab all those invoices that you never touched – which are probably old and you don’t care about anymore – and I'm going to go ahead and force calculate those now and get it over with. That’s something else that can be done afterwards, but you're allowing the application to manipulate that data in a way that's not affecting the user or not impacting the user by doing those kinds of clever things. JAIM: One concern I have with what I can do in the batch updates is if you're doing something with SQL, you don’t really have the transaction support. If something happens, your apps get killed or crashes, you're kind of out of luck, but this sounds like a way that you can kind of work around it and you update what you update and if [inaudible] maybe you can just do it again. MARCUS: A lot of the designs that I come up with when it comes to solving problems, I'm thinking about it from that direction of I want to make sure that if it’s not atomic, which nothing with a persistent storage was truly ever atomic, I want to make sure that it’s repeatable and that it’s recoverable. The same thing when I'm importing data, if I'm pulling data in from the server, I always want to make sure that it never gets into a state where I'm hosed. I can always have a recoverable state and I can always make sure that if the user force quits it or the battery dies or the connection drops, or any of other thousand things that are fun problems for us now that we’re in the mobile space, that we can recover from those gracefully so that we don’t get those one-star reviews going, “I ran out of battery and now I've lost all my work,” which is a very real problem that you see. PETE: That’s the really scary thing about things like – not things like Core Data as a technology, but basically the persistent state is the chance that you can screw something up and someone’s app is permanently hurt. MARCUS: I guess. PETE: It’ll crash on start, and it will always crash on start forever, the end. JAIM: Delete and re-install. PETE: Yeah. MARCUS: That is the stuff of nightmares for someone like myself who’s – I'm the one who gets called going, “My application crashes on launch and everybody’s data’s gone. What do I do?” That’s the stuff that just makes me cry, and it’s a very, very real problem. If you don’t think about it in the beginning, it’s part of some of the things that I try to teach when I'm at conferences and colleges is, Core Data is very much like the back of a cabinet. I'm stealing that quote from Steve Jobs. People don’t think about it; people don’t care about it. The only person who ever cares about the back of the cabinet is the developer – hopefully, but not always – or someone like myself who’s coming in afterwards when the application’s on version 3 and now they're in this really, really bad spot. Ideally, the developer, the designer of the application, he should be looking at the back of that cabinet and give it equal – if not more – importance than the front of the cabinet. People can forgive a UI glitch. They go, “You know, this button is off by two pixels. It’s ugly, but the button’s still there.” When you say “I'm sorry, your [inaudible] is gone,” people aren't too terribly forgiving for that. Even though it’s not invisible, even though people don’t go, “Oh yeah, they added the Core Data in my app; it’s so much better” – you never hear that. It’s, to me, from my perspective, it’s far, far more important to the happiness of the user to make sure that we get that backend just absolutely bulletproof – as far or as close to bulletproof as we possibly can get. And we never get into the state of permanent, unrecoverable data. JAIM: One of the scary things when I think about this – and we were talking about migrations and how to take someone from version 1 through to version 4 – when you're testing your app or you're testing some change to your app, you're probably testing against a new version of the data that hasn’t been migrated all the way through from version 1. Are there any tools or tricks or something out there for someone who wants to make sure that not only is the new version of their app going to work with old versions of the operating system, but also, is it going to work with version 3 of your app that was migrated up from version 1 where the data model is totally different? MARCUS: Yes, you can. There are a couple of different ways to do that. My first recommendation to people always is to keep a copy of every database, every version of the database. If you’ve got version 1, 2, 3, 4, 5 – you’ve got five different versions of it, you should add a minimum of five databases – one for each one of those versions. You keep them as part of your project, as part of your test plan, and then you inject each one of those into the application before launch and make sure that it migrates properly to the latest version. That’s step one; that should be fairly obvious. Step two is customer support. Whenever you get a customer saying, “My database is hosed; it’s broken. It doesn’t work; it’s really slow” – anything – any excuse you can to get a copy of your user’s database, do it, and then hang on to those suckers. Every one of those is gold, because now you’ve got a potential problem database, and you can use that as part of your testing cycle. You say, “Here’s Mary Jane’s database,” where she’s Brent Simmons’ – one of his famous examples – “she has every RSS feed on the planet and every article on the planet, and they're all marked as unread.” PETE: [Laughs] MARCUS: You laughed, but that’s actually based on a true story, and I'm going to use that as one of my tests. You build up this collection of strings and you hang on to all of these guys and you test them all and you build tests for them. The testing framework, instead of Xcode, is more than sufficient to be able to build up these tests, and they're really super simple. You have a startup method that says ‘build me a context based on model x,’ and then you build up on that and you say, ‘okay, now I'm going to hand it this database file from version 1, while I'm version 3.’ What happens? It doesn’t migrate properly [inaudible], and you build those up. Every time you do a new version model for production, you add another test to the top layer of that, so that you say, “Now I've got version 5, so I'm now going to test version 1, 2, 3, 4 against 5 and make sure that each one of them would stand up, each one of them migrates properly and make sure these tests are repeatable.” That’s the bare minimum. Doing that will guarantee, or get close to guaranteeing, that your users are not going to have a migration problem. Secondarily to that is having a fallback plan. So, the user has something bizarre that I didn’t expect – what do I do? The common answer is do nothing and just take the hit for the App Store review, but there are alternatives, and this also goes back to the third option for migrating as well. For each version of your database, each version of your data model, write an export routine for it, and the export routines are actually very, very simple. There's a baseline for it that I posted on Stack Overflow a few years ago, and the idea is walk through down a model and export it up to JSON – the whole data [inaudible]. No matter what version of the database you're on, just export it up to JSON. This is useful on a few different areas: one is useful if you ever need a push to a server, or push to another – a non-Apple device, because JSON is a format that everybody recognizes; number two is it’s really useful for migrations, so if you're in a situation where version 1 was stupid simple, version 5 is super complex, and there is no migration path for that because going from version1 to version 5 just doesn’t work. When it doesn’t work, you just throw it out to JSON and then consume it; consume that'll SON back into version 5. Make sure that the latest code base can handle every version of that JSON, which is not hard, because if JSON doesn’t have a field, it doesn’t have a field, you move on. You set it to a default value and continue. You can do that kind of a migration pass, so that’s your fallback for migrations. It’s also your fallback for ‘the database doesn’t want to migrate properly,’ so you got a user on version 3, there's something funky with their – they’ve got Arabic characters, you didn’t expect Arabic, whatever. Export that sucker out to JSON, JSON is perfectly happy; try to export it into 5 – if it still fails, now you have a really nice format that they can send to you and you can look at it and fix it and send it back. But most of the time it’s going to work, because now you're dealing with just JSON and now you can just suck it in. You're not dealing with an opaque format; you're dealing with a transparent format, and you can write import routines that are fairly simple for that. It’s a little bit of effort to do at the beginning, and then it pays off in huge dividends as your application matures, because now you’ve got this other format that you can do all kinds of wonderful things with. PETE: That’s a really neat solution. MARCUS: It’s a belt-and-suspenders approach to persistence, which, if you're dealing with people’s data with things that are potentially unrecoverable, you should be looking at a belt and suspenders, and tailor, and anything else to make sure that everything’s already [inaudible]. It’s the one thing – I don’t know how many people remember, but remember iOS 5, when iCloud came out, and the stories of “I had all of my grades, for all of my schools, for all of my students in Numbers on iPad and now it’s gone.” Not “I had to recover it; I had to pull back” – no, it was gone. Deleted. Completely unrecoverable, I didn’t backup, blame the user – fine, but you just lost their livelihood. And that was Apple who did that; that was Apple’s mistake, so if Apple can make this mistake I guarantee you everybody else can, too. This should be the stuff of nightmares and we should be trying to make sure that it’s bulletproof in a ridiculous number of ways. PETE: I'm bummed we didn’t have time to talk about iCloud – maybe not in detail, but about the intersection of Core Data and iCloud, because I think that would be interesting to talk about, but maybe we’ll have to get you back on to do another show. MARCUS: Certainly. There's a lot to discuss in that arena, and it’s an interesting subject. PETE: I actually don’t know if we’ve done any episodes on iCloud. CHUCK: I don’t think we have. PETE: Yeah, that’s kind of surprising. JAIM: The Dark Arts. PETE: Yeah. CHUCK: Well, we need to do MobileMe, too. MARCUS: [Laughs] You'd go all the way back to [inaudible][chuckling]. I will, instead of leaving the listeners in the dark, Apple is not good at networking, which is kind of hilarious because they have some of the smartest people on the planet working for them. For some reason, when it comes to networking, it’s just – no matter what it is – they're just blind-eyed to it and it’s really, really bizarre. I don’t know if it’s because they have really good network connections at their campus or what it is, but –. PETE: I think it’s not just networking; I think it’s just server-side stuff in general. If you think about all of the iCloud stuff, also that crazy outage for iTunes connect for multiple – how long was that? That was like weeks. MARCUS: Yup. PETE: That’s pretty awful downtime, right? Most other companies would be pretty ashamed. I guess, maybe, Apple is ashamed as well, but [inaudible]. MARCUS: I'm sure they were, but this is a repeatable experience. It was kind of funny that if you go back to the very beginning days when MobileMe was announced after Toolkit, a lot of us were like, “Okay, great. If I only fixed it.” And then iCloud came out and we’re like, “Oh, hey, they finally fixed it!” And now when version 7 came out, they're like, “No, no, no, we really fixed it this time – we promise!” And all was like, “No, I just don’t believe you anymore.” PETE: And they did this thing when they announced the new one, we’re like, “Yeah, we know. The old one was kinda crappy, hah-hah-hah.” It’s better now, and we’re like, “Hm, maybe.” CHUCK: But it’s not networking per se, it’s the cloud services that they struggle heavily with. PETE: Yeah, it’s just backend stuff. CHUCK: Looking at this stuff in the keynote from WWDC where they have the iPhone and you can make calls from your computer through your iPhone and stuff like that – they really get some of the integration stuff down, and they get it down really well. But anything that has to connect to a cloud service, it’s hard. JAIM: I think the problem is they're trying to do all their backend coding in objective-C, so I think that’s where Swift came from. [Laughter][inaudible] MARCUS: Don’t say it, that'll make me cry. PETE: I'd love to know what all of the backends for iCloud is implementing. MARCUS: Well, you know, WebObjects is all Java now, so you never know. PETE: That’s even worse, right? Well, not worse but –. MARCUS: But it could be the whole thing with Garbage Collection – when they came out with Garbage Collection for objective-C, everybody immediately poopooed on it. Honestly, if they had called it anything else – if they had called it ‘fairies and rainbows’ – they probably would’ve got a better reception for it, just because they called it Garbage Collection instead of automatic reference counting, then we should [inaudible]. They could’ve called it anything else and people probably would have had a bigger shot at it, but it’s one of those things of ‘Java burned us hard on Garbage Collection.’ To this day, it’s still there. You operate a Java server and it still stops the universe while it does its Garbage Collection calculations. It’s better; we don’t have the five minute pauses that we used to have and the 100% CPU [inaudible] - I used to write server Java back in the day – but it’s still there. There's still that, ‘Ooh, what's that spike? Oh, Garbage Collection, yeah. It’ll go away in a second.’ Whenever you hear the words Garbage Collection, it sends chills down people’s spines for that reason. That’s why when ARC came out, everybody’s like, “Oh, I’ll try ARC; I’ll never try Garbage Collection, but I’ll try ARC.” There really isn’t – technology-wise – a lot underneath it, but as far as the developer’s concerned, there's really not much difference; you're still trusting the system to do it for you. PETE: It’s slightly less magical, right? You still need to understand the weirdness of lbocks and the abstraction leaks somewhat, whereas garbage collection of these – the abstraction leaks from an operational point of view. Like you were saying, these memory spikes and stop-the-world sweeps, but at least as a developer it is pretty much magic. As opposed to – I think ARC is an amazing technological accomplishment but it makes me a little bit sad that you still have to understand how it works in order to not screw it up. Kind of like the Core Data thing, right? You need to know that the batch update is doing this extra stuff behind the – there's some sharp edges though, I guess, as well. MARCUS: Yes. Call it having a great beard, call it having a lot of scars – whatever you may call it – I like those sharp edges. I have probably a very antiquated opinion of “coding should be hard.” It’s right up there with “brain surgery should be hard;” I really don’t want everybody doing brain surgery, and I don’t really want everybody doing writing code, because we’ve been down that path. We went down that path with Visual Basic and it didn’t turn out well, and I don’t think that’s a path we should go down again. This should take – I hesitate to use the word “intelligence” but that’s one word for it, but also it needs to take energy and devotion and dedication to the art and the craft of it. Every time we come back and say, “Okay, now it’s easier,” I kinda cringe a little bit, because every time we lower that barrier, we get a little bit more of not good coming into the space. And I know the business reasons behind it, they make sense, they know there's a real brain drain problem in the Bay Area these days – I get all that, but every time you lower your standards, you're lowering your standards, so it becomes a real problem. PETE: I think, for me, the goal is not to lower that level so anyone can go in there and make a mess of things, but I do think there's a lot of that in flattening the learning curve, right? Eventually, you need to get up to that level of really understanding the craft, but if we can make it so that you don’t keep on having to slam your head against the wall and hold it together, then I think people could get up that slope faster. MARCUS: True. PETE: I totally agree; I don’t want a bunch of people who don’t understand any of the fundamentals of what they're doing to be writing apps for me because I don’t want to have my apps crashing all the time. Things like Swift, I think, are a great example of just removing accidental complexity so we can focus on the fundamental complexity of building software. MARCUS: Yes, and striking that balance is the hard part. PETE: Yeah. MARCUS: How do we make it into a gently curve, but not too gentle? We’ve definitely – and when I say we, I'm talking about all developers – we’ve gone to the too gentle way before where ‘look, you can just draw widgets on the screen and they’ll work’ to ‘Ooh, you actually have to write this in binary,’ so striking that balance is definitely the challenge. The question as with any new language coming out is, “Did we get the balance right? Did we make this better? Did we make it worse?” PETE: Yeah, I'm really interested to see how Swift developed over time. The thing that I was most encouraged by is Apple quite explicitly saying, “We are going to break language compatibility. We’re not going to do the job of trying to always be backwards compatible from a language point of view.” That’s cool, because it sounds like they're preparing themselves for a few years of evolving a language based on how real people are using it, rather than just kind of sticking to their guns and saying, “Nope, this is the way to do it; if you're not doing it this way then you're doing it wrong.” MARCUS: Yes, that’s definitely something that we can thank both Java and Microsoft for is we now know where that goes. So them saying, “Yes, we’re going to cut this cord” – it was nice and it definitely offers some hope. PETE: Yeah, well I mean, objective-C’s got the same issues, right? C, C++ - most languages have had that, have been shackled by the same thing. It was like, “Oh, we need to make it a subset or a – subset or the previous version of the language, so we’re going to make the syntax ridiculous in order to achieve that. MARCUS: Yup. Especially on the OSX side, we’ve got some gnarly code in there. It’s like, ‘why do you – oh, because we have to talk to [inaudible] and all this other stuff. And Apple did the same thing with iOS when they first came out with it. They're like, “This is App Kit version 2, and it doesn’t look like App Kit version 1 at all, and we’re going to call you iKit so you make sure it’s clear that this it’s not the same thing.” So they did cut a lot of cords there as well, so they do have a history of that, which is kinda nice. CHUCK: Yeah. The thing that frustrates me with trying to maintain backwards compatibility isn’t the gymnastics that they go through in order to do it. It’s usually when we get to the point where the technology we have has now outstripped the capability that we are trying to maintain with the backward compatibility, and in order to move forward in a meaningful way, we kinda have to cut that cord, and they don’t. That frustrates me because we could have so much nicer things if we would just let go of that in this. MARCUS: Definitely. Then there's also the other side of the coin of people going, “Hey, we’ve let go of that.” You never need to touch these things again, which we’re seeing a lot now with Swift again, where a lot of the – I would caution to say, younger, less experienced developers are going, “Hey, we don’t ever have to touch pointers again” and kind of the old, grey beards are looking at them going, “Yeah, we’ve been there before. We’ll see.” CHUCK: Yup. [Chuckles] MARCUS: Yeah, nobody’s saying they're wrong, but everybody’s kinda chuckling at them going, “I remember being that young.” So, we’ll see. We’ll see what the language looks like. I'm excited by it, but I'm cautionarily excited by it, because I've been around for a while and I've seen this before. It was always a step forward, but it was always a step forward across broken glass. I like my analogies. CHUCK: Alright, well, should we get into the picks? MARCUS: Certainly. CHUCK: Alright, Jaim, do you want to start us with picks? JAIM: Sure. So we don’t have Ben today, so I'm going to do a beer pick. I'm going to do a pick from a company that’s been around for a long time – a brewery – Abita, out of Louisiana, and this is one of the older craft breweries. They were a brewery before craft beer was even a word. I think they used to call it microbrews, but that’s something that when I lived in south Florida in the ‘90s, craft beer, microbrewery – this wasn’t really a word, but we can get one of their beers there, Purple Haze, and if you are 21-year-old me, you'd drink anything named after a Jimi Hendrix song [chuckling]. Since then, they’ve expanded quite a bit around and [inaudible] Purple Haze, kind of a fruit, wheat beer, which isn’t really my favorite anymore. But I was in Florida a couple of months ago and I saw some Abita and thought, “Yay, when in Rome, I’ll try this again.” So I tried their Amber, which is a Vienna Lager, which is a really fantastic beer and style, kind of a nice lager with a little bit of body to it, so, very nice. At 4.5 alcohol content – Pete, what is that? PETE: That’s a session beer. JAIM: A session beer! So if you want to just hang out and not get messed up after one or two, have an Abita Amber. Very good. That’s my pick. CHUCK: Very nice. Pete, what are your picks? PETE: As always, +1 on Jaim’s pick. I really like Abita, too. They have a really good – I think they have a [inaudible] that’s really good. Or [inaudible]-style. Anyway, my picks have nothing to do with what we’ve been talking about, or beer. Actually, I maybe have a beer, one at the end. I watched this talk from a conference called infoq, also a conference called QCon, and way back in 2009, I went to this talk by a guy called Kent Beck who is an amazing, amazingly smart guy about how we build software. He’s a very thoughtful, philosophical dude, one of the originators of XP – eXtreme Programming – and he did this talk called Responsive Design. It has nothing to do with responsive design like the UI thing if you know websites that will respond to the size of the screen – unfortunate naming collision there; he should’ve namespaced it with KB Responsive Design or something. But it was a really, really great talk about how approaches to build, to evolving the design of your software, so a really, really interesting, philosophical take on how we get from the design we have today to the design we want in our software. So, really cool talk; I really recommend watching it. Thinking about conferences made me think about the two things I like the most in conferences, and that’s stickers and conference t-shirts, so those are two of my picks. I really like stickers and I really like conference t-shirts. My last pick – I'm going to get even more ridiculously esoteric on the beer stuff. I've now stopped picking beers and I'm going to pick hops. My hop pick for this week is Galaxy Hops. This really crazy – they smell like passion fruit. I'm not kidding; I brewed some beer this weekend and I used Galaxy Hops to brew the beer, but when I opened up the little bag that the hops come in, it smells like hops but it actually full-on smells like tropical fruits like passion fruit, so hopefully the beer will turn out well. That’s my last pick, the Galaxy varietal of hops. CHUCK: Awesome. Andrew, what are your picks? ANDREW: Got a couple of picks today. My first one is a website called practicalswift.com, and I think there's been, already in the last two weeks, kind of an explosion of people trying to start websites for Swift, but this one’s a good one. It’s a lot of just sort of practical tips, especially now, right now, while we’re in the early days of Swift. I think it’s kind of a fun time because there's not a lot of documentation for Swift and things are still changing and kinda buggy and everybody’s new to it – that’s a fun time to be involved in a new technology, and this website’s just got cool stuff that you can’t really learn about Swift anywhere else. In particular, the way this post has a list of all the built-in functions in Swift, like 90% of which are not documented at all, so that’s practicalswift.com. My second pick is actually our guest’s, Marcus’s website, Cocoa Is My GirlFriend.com – cimgf.com – and I don’t know how long he’s been doing this; it’s quite a while. It’s one of the blogs that I have been reading for a very long time and have learned a ton of stuff from, so it seems like Marcus, you don’t do frequent posts you guys, but the posts that are on there are excellent and you can learn stuff no matter how long you’ve been doing Mac and iOS development. So, Cocoa Is My GirlFriend. Those are my picks. MARCUS: There's actually, if I could interject for a second, there's actually a reason for the decrease in post. In all honesty, [inaudible] Stack Overflow. We’ve actually drawn a line of if I can ask or answer this in the Stack Overflow post, I'm going to do it, because their search rankings are really nice and high and we can get a bigger audience, and if it’s a more complex issue, or nobody has asked it yet, then we’ll put it on Cocoa Is My GirlFriend. We’re trying to take a little bit of a backseat because of the way that Stack Overflow is designed, we really like it, and we just use Cocoa Is My GirlFriend to answer everything else. ANDREW: Well, I still find it really useful, so you’ve actually got your newest post about Core Data and Swift and that was just a great intro. Then you talked about the sort of boiler plate code for Core Data and Swift; I really appreciated that post. Even so, check it out. CHUCK: Yeah, you know girlfriends don’t like to be neglected, especially if you're going over to a hotter, sleeker, Stack Overflow girlfriend and doing your stuff over there. MARCUS: Yeah. Now cue all the bashing for us calling Cocoa Is My GirlFriend. CHUCK: [Chuckles] Yup. Alright, I've got a couple of picks. I finished the Steve Jobs biography, finally, this weekend. I know I picked it last week, but it was really good so I'm going to pick it again. Marcus, what are your picks? MARCUS: Well, for the first one I'm actually going to pick one of our own applications that’s being developed by MartianCraft. It’s an application I had to use quite a bit. We wrote it for ourselves, and then we shared it with everybody else, and it’s called Slender. It’s a Mac application designed by developers for developers. What it does is it goes through your Xcode project and finds duplicates, it finds dead assets, it finds bad assets, and it notifies you or fixes them. What it does is it reduces your overall project size and it reduces your overall application size, which for iOS projects could be incredibly valuable. After a three-month, six-month, 12-month development cycle, you may not realize you’ve got 15 copies of your icon in there and they're all not exactly the same. And it catches a lot of other things: it looks out for magical strings and other common issues, and just helps make your applications smaller, slimmer, faster, better. That’s my first pick. My second one is a tool for Core Data programmers. It’s been around almost as long as Core Data itself – it’s called mogenerator, written by a fantastic developer out of Chicago by the name of Wolf Rentzsch. What it does is it produces the object files for you, so it produces the subclasses of NSManageObject, but it does it differently than Xcode does. Every time you do that export from Xcode, it'll erase your existing object files and replace them with new ones. If you’ve got custom data in there or custom methods in there – they're gone. So mogenerator solves the problem from a different direction because it actually creates two subclasses: one’s called the machine file, one’s called the human file. The machine file subclasses NSManageObject; the human file subclasses the machine file, so whenever your data model changes, mogenerator only updates the machine file, and your human file which has all your convenience methods and your other code in there remains intact, so it’s really, really useful as you go into different iterations of your application when you're making modifications to your data model and you're really exercising Core Data to its fullest; it’s a very, very good tool to have. Its command line interface works great; I've been using it for years; I highly recommend it. Lastly, since there are beer aficionados in here, I've gotta throw out a non-beer. I'm allergic to wheat – I can’t imbibe wheat at all, so I've become a bit of a cider snob. I thought I was a cider snob, and then I went to England and then I regally became a cider snob. Over here in San Francisco, we don’t have a lot of choices. However, there's a relatively new one that’s come out and it’s quite nice. It’s called Angry Orchard. They're making the rounds, they're getting more and more popular, and they’ve got different flavors, but they're probably the closest thing we have to a proper English cider that we have over here. That would be my third pick. CHUCK: Very nice. Well thanks for coming, Marcus. We really appreciate you taking the time. MARCUS: I'm happy to be here; it was a very fun conversation. CHUCK: Alright. If people want to know more about you or MartianCraft, or any of the other things that you're working on, what are the best ways to do that? MARCUS: Well, MartianCraft itself, you can go to our website, which is martiancraft.com. You can learn all about what we’re working on, the things we’re developing; we do a lot of products as well as third party development, so there's quite a few products we have over there. Most of our products are for engineers, so if you're a Cocoa developer, you should take a look at our product line. We write tools for ourselves, and then we productize them afterwards, so there's a lot over there. Cocoa Is My GirlFriend is also a great resource. We add things there as we come up with – sometimes we can come up with code solutions that are interesting or unique and we’ll post them there. Sometimes it’s just code [inaudible] going, “Ah, the Apple template’s not that great. Here’s a better version. Here’s what we use instead of the Apple version” and things like that. We’ll answer complex problems there that are beyond the simple problems that can be answered on Stack Overflow, or in some of the Apple developer forums. Those are the two best ways. I can also be reached on twitter: first initial, last name, just @mzarra. I tend to respond to Twitter fairly quickly, so if you have a question on Stack Overflow or somewhere else that you want me to take a look at, ping me on Twitter and I generally respond there. Those are usually the best ways. You can send me email; I’ll look at it, but my track record of replying to email is not nearly as good as replying to things on Twitter. CHUCK: Well, thanks again. I guess we’ll wrap the show; we’ll catch you all next week! [Hosting and bandwidth provided by the Blue Box Group. Check them out at BlueBox.net.] [Bandwidth for this segment is provided by CacheFly, the world’s fastest CDN. Deliver your content fast with CacheFly. Visit cachefly.com to learn more]